From d51840d4331ef09e4b6266774b65242cfefd3218 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 14:31:14 +0000 Subject: [PATCH 01/20] docs: rewrite Lakebase Search as an architecture-first hub page Leads with the agent-native use case and adds an architecture diagram. Adds a what-you-get section and demotes search types to a reference section. Keeps pgvector first-class and frames lakebase_vector as a compatible superset. Pending before un-drafting: compression-ratio wording and benchmark publication clearance. --- content/docs/ai/lakebase-search.md | 97 +++++++++++++++++++++--------- 1 file changed, 68 insertions(+), 29 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index 40290123d7..32ecd3f472 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -1,57 +1,96 @@ --- title: Lakebase Search -subtitle: Scalable vector and full-text search for Postgres +subtitle: A search system built for agents, running inside your Postgres database summary: >- - Lakebase Search adds vector, keyword, and hybrid search to Neon through the - lakebase_vector and lakebase_text Postgres extensions, with optional - configurable tokenization from lakebase_tokenizer. Use this page to understand - the search types, how the extensions work, the scale-to-zero architecture - advantages, and where to get started. + Lakebase Search is agent-native vector, keyword, and hybrid search built into + Lakebase Postgres on Neon. It scales vector search in Postgres past the limits + of pgvector by separating storage from compute, so a single index handles 100 + million vectors and beyond with serverless scale-to-zero. The lakebase_vector + extension adds the lakebase_ann index (a pgvector-compatible superset), + lakebase_text adds the lakebase_bm25 index for BM25 keyword search, and hybrid + search runs as plain SQL. enableTableOfContents: true redirectFrom: - /docs/extensions/pg_search - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-10-23T11:22:59.906Z' +updatedOn: '2026-11-17T14:31:14.840Z' --- - -Lakebase Search is developed by Databricks. These extensions are part of the shared technology foundation between Neon and the Databricks Lakebase platform. +Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. It's built for retrieval at agent scale: there's no separate vector database to run, no pipeline to keep in sync, and nothing to rebuild when your data changes. + + + + +Lakebase Search is developed by Databricks. The same extensions power search on Neon and on the Databricks Lakebase platform. -Lakebase Search adds vector, keyword, and hybrid search to your Neon project. Install `lakebase_vector` for vector search, `lakebase_text` for BM25 keyword search, and `lakebase_tokenizer` for configurable tokenization. +## Scaling search past pgvector - +Most teams start vector search in Postgres with `pgvector`, and it works well until the data grows. The usual `pgvector` index, HNSW (Hierarchical Navigable Small World), has to fit in memory, and that memory grows with the number of vectors, not with how much you actually query. Past roughly tens of millions of vectors, you're paying for a large, always-on instance just to hold the index. These are the limits you run into: -## Vector, keyword, and hybrid search +- **Memory scales with data, not usage**: an HNSW index keeps its whole graph in RAM and traverses it with random reads at query time, so cost tracks how much data you store rather than how much you search. +- **Index builds are slow and disruptive**: building or rebuilding an HNSW index is CPU-heavy and can tie up the database for a long time on large tables. +- **Recall versus latency is a constant tradeoff**: tuning HNSW means trading recall against query speed, and the balance shifts as the dataset grows. +- **A separate vector database adds a sync tax**: moving vectors out to a dedicated vector store means running a second system and a pipeline to feed it. Your search results lag your writes, and you own the plumbing that keeps the two in step. -Lakebase Search gives you two complementary ways to search. Use either on its own, or combine them: +That last point is the one teams feel most. When search lives in a different system from your data, your results are only as fresh as the last sync. Lakebase Search keeps search in the same database as your data, so a row is searchable as soon as it's written. No pipeline, no second system, no sync lag. -- **Vector (semantic) search** finds rows whose meaning is closest to your query, even when they share no words. You search with an embedding (a numeric vector from a model), and the `lakebase_ann` index returns the nearest vectors by distance. Use it for natural-language questions, recommendations, and retrieval-augmented generation (RAG). -- **Keyword (full-text) search** ranks rows by how well they match the exact terms in your query, using BM25 relevance scoring from the `lakebase_bm25` index. Use it for names, codes, and exact-term lookups where wording matters. -- **Hybrid search** combines vector and keyword results with fusion or reranking, so you get semantic and exact-term matches together. Use it when queries mix intent with specific terms, which covers most real-world search. The [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to combine results with Jev through Neon AI Gateway or RRF. +None of this means leaving `pgvector` behind. `lakebase_vector` is a strict superset of `pgvector`: the same `vector` types, the same distance operators, and the same SQL. You keep what you already use and add an index tuned to run faster and scale further, without the HNSW memory wall. -![Keyword search matches only documents that contain the typed words and misses synonyms like "quick automobile" for "fast sports car." Vector search places the query and documents in the same embedding space and returns the nearest neighbors by meaning, including those synonyms.](/docs/ai/lakebase-search-keyword-vs-vector.png) +## How Lakebase Search works + +Lakebase Search scales vector search in Postgres past `pgvector` by separating storage from compute, the same architecture that powers Neon. Your indexes live in storage rather than in compute memory, so compute stays stateless and can scale to zero when the database is idle. Lakebase Search is built on two search extensions, `lakebase_vector` and `lakebase_text`, plus `lakebase_tokenizer` for configurable tokenization. + +```mermaid +flowchart TD + A["App or agent"] -->|"one SQL query"| B["Lakebase compute
stateless, scales to zero"] + B --> C["lakebase_vector
lakebase_ann index: IVF + RaBitQ"] + B --> D["lakebase_text
lakebase_bm25 index: BM25 + top-K"] + C --> F["Hybrid result"] + D --> F + F --> A + G[("Separated storage
indexes and data")] --- C + G --- D +``` -## How it works +The pieces behind that flow: -Lakebase Search is built on three Postgres extensions: +- **Separated storage and compute**: indexes live in storage, not in compute RAM, so the size of your index doesn't force you onto a large, always-on instance. +- **IVF clustering**: `lakebase_ann` partitions the vector space into lists (Inverted File, or IVF) and scans only the relevant ones at query time, using sequential I/O that suits storage-backed systems rather than random pointer-chasing. +- **RaBitQ compression**: vectors are compressed with RaBitQ quantization, which keeps index size and build time down while preserving search quality. +- **Stateless, scale-to-zero compute**: you pay for compute only while the database is serving queries, so an idle dev or staging database doesn't cost you compute while it waits. +- **Native BM25 keyword search**: `lakebase_text` adds the `lakebase_bm25` index with Block-Max WAND top-K pushdown, returning the most relevant results directly instead of scoring the whole match set the way a GIN index does. +- **Hybrid as plain SQL**: vector and keyword search are both just SQL, so you can combine their results in one query and then fuse or rerank them. -- **[`lakebase_vector`](/docs/extensions/lakebase-vector)**: adds the `lakebase_ann` index type for vector similarity search. No migration from `pgvector` required. The same `vector` types, distance operators, and query syntax work unchanged. Scales to over 1 billion vectors on a single index. -- **[`lakebase_text`](/docs/extensions/lakebase-text)**: adds the `lakebase_bm25` index type for BM25 keyword search. No migration from Postgres full-text search required. Standard `tsvector` types and query operators work unchanged. Adds BM25 ranking and top-K pushdown that native GIN lacks. -- **[`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer)**: adds configurable whole-word tokenization and Chinese, Japanese, and Korean (CJK) morphological analysis through Postgres's text-search dictionary interface. It produces standard `tsvector` values that work with GIN and `lakebase_bm25` indexes. +## What you get -## How `lakebase_ann` scales +If you need the `pgvector`-compatible path to scale past tens of millions of vectors, Lakebase Search gives you: -`lakebase_ann` is designed for the separated compute/storage architecture that powers Neon. Because the index lives in storage rather than in compute memory, it works naturally with Neon's scale-to-zero model: your vector index is available immediately after a cold start, with no warmup required. You only pay for compute when your database is actively serving requests. This holds for production workloads and for development or staging environments where the database sits idle most of the time. +- **Lower cost at scale**: because indexes live in storage and compute scales to zero, you avoid paying for a large, always-on instance sized to hold an in-memory HNSW index. +- **Scale to 100 million vectors and beyond**: a single `lakebase_ann` index grows with your data without resharding or rebuilding. +- **Serverless with scale-to-zero**: indexes survive a cold start with no rebuild or re-index. The first query after a cold start pays object-storage latency while the cache warms. You can call `lakebase_ann_prewarm` to warm an index ahead of time. +- **Hybrid search with BM25**: semantic matching and exact-term matching in a single query, so queries that mix intent with specific terms work well. +- **Data searchable immediately**: a row is searchable as soon as it's written, with no second system to keep in sync. -It uses IVF (Inverted File) partitioning to divide the vector space into lists and searches only the relevant ones at query time, using sequential I/O that suits storage-backed systems. RaBitQ quantization compresses vectors 4–8x, keeping index size and build time down. The result: +Because branches on Neon copy on write, your search indexes are available on every branch without re-indexing, which keeps preview and development branches cheap. -- **Scale to 1 billion+ vectors**: a single `lakebase_ann` index grows with your data without resharding or rebuilding -- **Faster index builds**: compression reduces index size significantly, making builds 50–100x faster than HNSW -- **Branching without index rebuilds**: Neon branches copy-on-write; your search indexes are available instantly on every branch without re-indexing -- **Scale-to-zero compatible**: indexes survive cold starts without warmup; you don't pay for idle compute +## Benchmarks + +How fast is it? Databricks has published benchmark results for Lakebase Search on large vector workloads, including query throughput and recall at 100-million-vector scale. For the methodology and the measured figures, see the Databricks engineering blog, [Lakebase Search: state-of-the-art full-text and vector search in Postgres](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres). Those figures were measured by Databricks on Databricks Lakebase. + +## Search types + +Lakebase Search supports three kinds of search. Hybrid search on Neon combines the first two in a single SQL query: + +- **Vector (semantic) search** finds rows whose meaning is closest to your query, even when they share no words. You search with an embedding, and the `lakebase_ann` index from [`lakebase_vector`](/docs/extensions/lakebase-vector) returns the nearest vectors by distance. Use it for natural-language questions, recommendations, and retrieval-augmented generation (RAG). +- **Keyword (full-text) search** ranks rows by how well they match the exact terms in your query, using BM25 relevance scoring from the `lakebase_bm25` index in [`lakebase_text`](/docs/extensions/lakebase-text). Use it for names, codes, and exact-term lookups where wording matters. +- **Hybrid search** combines vector and keyword results with fusion or reranking, so you get semantic and exact-term matches together. The [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to combine results with Jev, a reranking model on Neon AI Gateway, or with Reciprocal Rank Fusion (RRF). + +For configurable whole-word tokenization and Chinese, Japanese, and Korean (CJK) analysis, [`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer) produces standard `tsvector` values that work with both GIN and `lakebase_bm25` indexes. + +![Keyword search matches only documents that contain the typed words and misses synonyms like "quick automobile" for "fast sports car." Vector search places the query and documents in the same embedding space and returns the nearest neighbors by meaning, including those synonyms.](/docs/ai/lakebase-search-keyword-vs-vector.png) ## Get started From cd4755aad3a710eb4a5e56c39b97c97b78bcc060 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 15:32:48 +0000 Subject: [PATCH 02/20] docs: tighten Lakebase Search voice and integrate search types Rewrites the pgvector-limits and how-it-works sections as prose instead of stacked bullets. Cuts editorializing and marketing phrasing for a direct, developer-first tone. Folds the vector, keyword, and hybrid reference into the page flow instead of an appendix. --- content/docs/ai/lakebase-search.md | 58 ++++++++---------------------- 1 file changed, 15 insertions(+), 43 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index 32ecd3f472..b196777093 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -2,20 +2,18 @@ title: Lakebase Search subtitle: A search system built for agents, running inside your Postgres database summary: >- - Lakebase Search is agent-native vector, keyword, and hybrid search built into - Lakebase Postgres on Neon. It scales vector search in Postgres past the limits - of pgvector by separating storage from compute, so a single index handles 100 - million vectors and beyond with serverless scale-to-zero. The lakebase_vector - extension adds the lakebase_ann index (a pgvector-compatible superset), - lakebase_text adds the lakebase_bm25 index for BM25 keyword search, and hybrid - search runs as plain SQL. + Lakebase Search adds vector, keyword, and hybrid search to Lakebase Postgres + on Neon. It scales vector search past pgvector's HNSW memory limits by keeping + indexes in storage, so a single lakebase_ann index handles 100 million vectors + and beyond with serverless scale-to-zero. lakebase_text adds BM25 keyword + search, and hybrid search runs as plain SQL. enableTableOfContents: true redirectFrom: - /docs/extensions/pg_search - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T14:31:14.840Z' +updatedOn: '2026-11-17T15:32:48.504Z' --- Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. It's built for retrieval at agent scale: there's no separate vector database to run, no pipeline to keep in sync, and nothing to rebuild when your data changes. @@ -28,20 +26,15 @@ Lakebase Search is developed by Databricks. The same extensions power search on ## Scaling search past pgvector -Most teams start vector search in Postgres with `pgvector`, and it works well until the data grows. The usual `pgvector` index, HNSW (Hierarchical Navigable Small World), has to fit in memory, and that memory grows with the number of vectors, not with how much you actually query. Past roughly tens of millions of vectors, you're paying for a large, always-on instance just to hold the index. These are the limits you run into: +Most teams start vector search in Postgres with `pgvector`, and it holds up until the data grows. The usual `pgvector` index, HNSW (Hierarchical Navigable Small World), has to stay in memory, and that memory grows with the number of vectors, not with how often you query. Past roughly tens of millions of vectors you're paying for a large, always-on instance just to hold the index, index builds get slow enough to disrupt the database, and every tuning pass trades recall against latency. -- **Memory scales with data, not usage**: an HNSW index keeps its whole graph in RAM and traverses it with random reads at query time, so cost tracks how much data you store rather than how much you search. -- **Index builds are slow and disruptive**: building or rebuilding an HNSW index is CPU-heavy and can tie up the database for a long time on large tables. -- **Recall versus latency is a constant tradeoff**: tuning HNSW means trading recall against query speed, and the balance shifts as the dataset grows. -- **A separate vector database adds a sync tax**: moving vectors out to a dedicated vector store means running a second system and a pipeline to feed it. Your search results lag your writes, and you own the plumbing that keeps the two in step. +The other common fix, moving vectors into a dedicated vector database, swaps those problems for a second system to run and a pipeline to keep in sync. Your search then lags your writes, and you own the plumbing in between. Lakebase Search keeps search in the same database as your data, so a row is searchable the moment it's written: no pipeline, no second system, no sync lag. -That last point is the one teams feel most. When search lives in a different system from your data, your results are only as fresh as the last sync. Lakebase Search keeps search in the same database as your data, so a row is searchable as soon as it's written. No pipeline, no second system, no sync lag. - -None of this means leaving `pgvector` behind. `lakebase_vector` is a strict superset of `pgvector`: the same `vector` types, the same distance operators, and the same SQL. You keep what you already use and add an index tuned to run faster and scale further, without the HNSW memory wall. +This doesn't mean leaving `pgvector` behind. `lakebase_vector` is a strict superset of `pgvector`, with the same types, operators, and SQL. You keep what you already use and swap in an index built to scale further, without the HNSW memory wall. ## How Lakebase Search works -Lakebase Search scales vector search in Postgres past `pgvector` by separating storage from compute, the same architecture that powers Neon. Your indexes live in storage rather than in compute memory, so compute stays stateless and can scale to zero when the database is idle. Lakebase Search is built on two search extensions, `lakebase_vector` and `lakebase_text`, plus `lakebase_tokenizer` for configurable tokenization. +Lakebase Search separates storage from compute, the architecture Neon already uses for Postgres. Your indexes live in storage instead of compute memory, so compute stays stateless and scales to zero when the database is idle. It adds two search engines to Postgres, `lakebase_vector` for vector search and `lakebase_text` for keyword search, plus `lakebase_tokenizer` for tokenization. ```mermaid flowchart TD @@ -55,40 +48,19 @@ flowchart TD G --- D ``` -The pieces behind that flow: - -- **Separated storage and compute**: indexes live in storage, not in compute RAM, so the size of your index doesn't force you onto a large, always-on instance. -- **IVF clustering**: `lakebase_ann` partitions the vector space into lists (Inverted File, or IVF) and scans only the relevant ones at query time, using sequential I/O that suits storage-backed systems rather than random pointer-chasing. -- **RaBitQ compression**: vectors are compressed with RaBitQ quantization, which keeps index size and build time down while preserving search quality. -- **Stateless, scale-to-zero compute**: you pay for compute only while the database is serving queries, so an idle dev or staging database doesn't cost you compute while it waits. -- **Native BM25 keyword search**: `lakebase_text` adds the `lakebase_bm25` index with Block-Max WAND top-K pushdown, returning the most relevant results directly instead of scoring the whole match set the way a GIN index does. -- **Hybrid as plain SQL**: vector and keyword search are both just SQL, so you can combine their results in one query and then fuse or rerank them. +`lakebase_vector` builds the `lakebase_ann` index, which splits vectors into IVF (Inverted File) lists and scans only the relevant ones, reading sequentially instead of chasing pointers through an in-memory graph. It compresses the vectors with RaBitQ quantization to keep the index small and the build fast. `lakebase_text` builds the `lakebase_bm25` index, which adds BM25 ranking with Block-Max WAND top-K pushdown, so it returns the top matches directly instead of scoring every row the way a GIN index does. Because both are ordinary Postgres indexes, hybrid search is just SQL: run both in one query and fuse or rerank the results. ## What you get -If you need the `pgvector`-compatible path to scale past tens of millions of vectors, Lakebase Search gives you: - -- **Lower cost at scale**: because indexes live in storage and compute scales to zero, you avoid paying for a large, always-on instance sized to hold an in-memory HNSW index. -- **Scale to 100 million vectors and beyond**: a single `lakebase_ann` index grows with your data without resharding or rebuilding. -- **Serverless with scale-to-zero**: indexes survive a cold start with no rebuild or re-index. The first query after a cold start pays object-storage latency while the cache warms. You can call `lakebase_ann_prewarm` to warm an index ahead of time. -- **Hybrid search with BM25**: semantic matching and exact-term matching in a single query, so queries that mix intent with specific terms work well. -- **Data searchable immediately**: a row is searchable as soon as it's written, with no second system to keep in sync. - -Because branches on Neon copy on write, your search indexes are available on every branch without re-indexing, which keeps preview and development branches cheap. +That architecture is what makes Lakebase Search a `pgvector`-compatible path past tens of millions of vectors. A single `lakebase_ann` index scales to 100 million vectors and beyond without resharding, and because the index lives in storage you don't pay for a large instance sized to hold it in memory. Compute scales to zero, so an idle dev or staging database doesn't cost you compute while it waits. After a cold start there's no rebuild or re-index; the first query pays object-storage latency while the cache warms, which you can avoid by calling `lakebase_ann_prewarm` ahead of time. ## Benchmarks -How fast is it? Databricks has published benchmark results for Lakebase Search on large vector workloads, including query throughput and recall at 100-million-vector scale. For the methodology and the measured figures, see the Databricks engineering blog, [Lakebase Search: state-of-the-art full-text and vector search in Postgres](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres). Those figures were measured by Databricks on Databricks Lakebase. - -## Search types - -Lakebase Search supports three kinds of search. Hybrid search on Neon combines the first two in a single SQL query: +For throughput and recall figures, Databricks has published benchmarks for Lakebase Search at 100-million-vector scale. See the Databricks engineering blog, [Lakebase Search: state-of-the-art full-text and vector search in Postgres](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres). Those results were measured by Databricks on Databricks Lakebase, not on Neon. -- **Vector (semantic) search** finds rows whose meaning is closest to your query, even when they share no words. You search with an embedding, and the `lakebase_ann` index from [`lakebase_vector`](/docs/extensions/lakebase-vector) returns the nearest vectors by distance. Use it for natural-language questions, recommendations, and retrieval-augmented generation (RAG). -- **Keyword (full-text) search** ranks rows by how well they match the exact terms in your query, using BM25 relevance scoring from the `lakebase_bm25` index in [`lakebase_text`](/docs/extensions/lakebase-text). Use it for names, codes, and exact-term lookups where wording matters. -- **Hybrid search** combines vector and keyword results with fusion or reranking, so you get semantic and exact-term matches together. The [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to combine results with Jev, a reranking model on Neon AI Gateway, or with Reciprocal Rank Fusion (RRF). +## Vector, keyword, and hybrid search -For configurable whole-word tokenization and Chinese, Japanese, and Korean (CJK) analysis, [`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer) produces standard `tsvector` values that work with both GIN and `lakebase_bm25` indexes. +Vector, keyword, and hybrid are the three ways you query, and each maps to an extension you can go deep on. [`lakebase_vector`](/docs/extensions/lakebase-vector) handles vector (semantic) search through the `lakebase_ann` index: you query with an embedding and get the nearest rows by meaning, which is what you want for natural-language questions, recommendations, and retrieval-augmented generation (RAG). [`lakebase_text`](/docs/extensions/lakebase-text) handles keyword (full-text) search through the `lakebase_bm25` index, ranking rows by BM25 for names, codes, and exact terms. Hybrid search runs both and merges the results; the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to fuse them with Reciprocal Rank Fusion (RRF) or rerank them with Jev, a reranking model on Neon AI Gateway. For custom tokenization, including Chinese, Japanese, and Korean (CJK) text, [`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer) emits standard `tsvector` values. ![Keyword search matches only documents that contain the typed words and misses synonyms like "quick automobile" for "fast sports car." Vector search places the query and documents in the same embedding space and returns the nearest neighbors by meaning, including those synonyms.](/docs/ai/lakebase-search-keyword-vs-vector.png) From b676688ab4ee47e38edf6956a7b659b4c16cc41d Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 15:40:41 +0000 Subject: [PATCH 03/20] docs: cut redundant Lakebase Search section and trim editorializing Removes the what-you-get section, folding its cold-start detail into how-it-works. Simplifies the lede and drops benefit-speak so each fact appears once. --- content/docs/ai/lakebase-search.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index b196777093..587b4c874f 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,10 +13,10 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T15:32:48.504Z' +updatedOn: '2026-11-17T15:40:41.656Z' --- -Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. It's built for retrieval at agent scale: there's no separate vector database to run, no pipeline to keep in sync, and nothing to rebuild when your data changes. +Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -48,11 +48,9 @@ flowchart TD G --- D ``` -`lakebase_vector` builds the `lakebase_ann` index, which splits vectors into IVF (Inverted File) lists and scans only the relevant ones, reading sequentially instead of chasing pointers through an in-memory graph. It compresses the vectors with RaBitQ quantization to keep the index small and the build fast. `lakebase_text` builds the `lakebase_bm25` index, which adds BM25 ranking with Block-Max WAND top-K pushdown, so it returns the top matches directly instead of scoring every row the way a GIN index does. Because both are ordinary Postgres indexes, hybrid search is just SQL: run both in one query and fuse or rerank the results. +`lakebase_vector` builds the `lakebase_ann` index, which splits vectors into IVF (Inverted File) lists and scans only the relevant ones, reading sequentially instead of chasing pointers through an in-memory graph. It compresses the vectors with RaBitQ quantization to keep the index small and the build fast. Because the index lives in storage, a cold start needs no rebuild or re-index; the first query pays object-storage latency while the cache warms, which you can avoid by calling `lakebase_ann_prewarm` ahead of time. -## What you get - -That architecture is what makes Lakebase Search a `pgvector`-compatible path past tens of millions of vectors. A single `lakebase_ann` index scales to 100 million vectors and beyond without resharding, and because the index lives in storage you don't pay for a large instance sized to hold it in memory. Compute scales to zero, so an idle dev or staging database doesn't cost you compute while it waits. After a cold start there's no rebuild or re-index; the first query pays object-storage latency while the cache warms, which you can avoid by calling `lakebase_ann_prewarm` ahead of time. +`lakebase_text` builds the `lakebase_bm25` index, which adds BM25 ranking with Block-Max WAND top-K pushdown, so it returns the top matches directly instead of scoring every row the way a GIN index does. Because both are ordinary Postgres indexes, hybrid search is just SQL: run both in one query and fuse or rerank the results. ## Benchmarks @@ -60,7 +58,7 @@ For throughput and recall figures, Databricks has published benchmarks for Lakeb ## Vector, keyword, and hybrid search -Vector, keyword, and hybrid are the three ways you query, and each maps to an extension you can go deep on. [`lakebase_vector`](/docs/extensions/lakebase-vector) handles vector (semantic) search through the `lakebase_ann` index: you query with an embedding and get the nearest rows by meaning, which is what you want for natural-language questions, recommendations, and retrieval-augmented generation (RAG). [`lakebase_text`](/docs/extensions/lakebase-text) handles keyword (full-text) search through the `lakebase_bm25` index, ranking rows by BM25 for names, codes, and exact terms. Hybrid search runs both and merges the results; the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to fuse them with Reciprocal Rank Fusion (RRF) or rerank them with Jev, a reranking model on Neon AI Gateway. For custom tokenization, including Chinese, Japanese, and Korean (CJK) text, [`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer) emits standard `tsvector` values. +Those two engines back three kinds of query. [`lakebase_vector`](/docs/extensions/lakebase-vector) handles vector (semantic) search through the `lakebase_ann` index: you query with an embedding and get the nearest rows by meaning, for natural-language questions, recommendations, and retrieval-augmented generation (RAG). [`lakebase_text`](/docs/extensions/lakebase-text) handles keyword (full-text) search through the `lakebase_bm25` index, ranking rows by BM25 for names, codes, and exact terms. Hybrid search runs both and merges the results; the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to fuse them with Reciprocal Rank Fusion (RRF) or rerank them with Jev, a reranking model on Neon AI Gateway. For custom tokenization, including Chinese, Japanese, and Korean (CJK) text, [`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer) emits standard `tsvector` values. ![Keyword search matches only documents that contain the typed words and misses synonyms like "quick automobile" for "fast sports car." Vector search places the query and documents in the same embedding space and returns the nearest neighbors by meaning, including those synonyms.](/docs/ai/lakebase-search-keyword-vs-vector.png) From b4bada476fc036b65442b547efc8edc215f6d1af Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 16:35:36 +0000 Subject: [PATCH 04/20] docs: swap Lakebase Search diagram for themed SVG Replace the Mermaid architecture diagram with a self-contained themed SVG embedded via InlineSvg so it follows the site's class-based dark mode, and drop the redundant keyword-vs-vector explainer image. --- content/docs/ai/lakebase-search.md | 16 +----- .../docs/ai/lakebase-search-architecture.svg | 53 +++++++++++++++++++ 2 files changed, 55 insertions(+), 14 deletions(-) create mode 100644 public/docs/ai/lakebase-search-architecture.svg diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index 587b4c874f..b33aa66df1 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T15:40:41.656Z' +updatedOn: '2026-11-17T16:35:36.759Z' --- Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -36,17 +36,7 @@ This doesn't mean leaving `pgvector` behind. `lakebase_vector` is a strict super Lakebase Search separates storage from compute, the architecture Neon already uses for Postgres. Your indexes live in storage instead of compute memory, so compute stays stateless and scales to zero when the database is idle. It adds two search engines to Postgres, `lakebase_vector` for vector search and `lakebase_text` for keyword search, plus `lakebase_tokenizer` for tokenization. -```mermaid -flowchart TD - A["App or agent"] -->|"one SQL query"| B["Lakebase compute
stateless, scales to zero"] - B --> C["lakebase_vector
lakebase_ann index: IVF + RaBitQ"] - B --> D["lakebase_text
lakebase_bm25 index: BM25 + top-K"] - C --> F["Hybrid result"] - D --> F - F --> A - G[("Separated storage
indexes and data")] --- C - G --- D -``` + `lakebase_vector` builds the `lakebase_ann` index, which splits vectors into IVF (Inverted File) lists and scans only the relevant ones, reading sequentially instead of chasing pointers through an in-memory graph. It compresses the vectors with RaBitQ quantization to keep the index small and the build fast. Because the index lives in storage, a cold start needs no rebuild or re-index; the first query pays object-storage latency while the cache warms, which you can avoid by calling `lakebase_ann_prewarm` ahead of time. @@ -60,8 +50,6 @@ For throughput and recall figures, Databricks has published benchmarks for Lakeb Those two engines back three kinds of query. [`lakebase_vector`](/docs/extensions/lakebase-vector) handles vector (semantic) search through the `lakebase_ann` index: you query with an embedding and get the nearest rows by meaning, for natural-language questions, recommendations, and retrieval-augmented generation (RAG). [`lakebase_text`](/docs/extensions/lakebase-text) handles keyword (full-text) search through the `lakebase_bm25` index, ranking rows by BM25 for names, codes, and exact terms. Hybrid search runs both and merges the results; the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to fuse them with Reciprocal Rank Fusion (RRF) or rerank them with Jev, a reranking model on Neon AI Gateway. For custom tokenization, including Chinese, Japanese, and Korean (CJK) text, [`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer) emits standard `tsvector` values. -![Keyword search matches only documents that contain the typed words and misses synonyms like "quick automobile" for "fast sports car." Vector search places the query and documents in the same embedding space and returns the nearest neighbors by meaning, including those synonyms.](/docs/ai/lakebase-search-keyword-vs-vector.png) - ## Get started diff --git a/public/docs/ai/lakebase-search-architecture.svg b/public/docs/ai/lakebase-search-architecture.svg new file mode 100644 index 0000000000..8b42d80be7 --- /dev/null +++ b/public/docs/ai/lakebase-search-architecture.svg @@ -0,0 +1,53 @@ + + + + + +your app + + +one SQL query + + +ranked results + +Postgres +search on your tables + +vector search +by meaning + +text search +by keywords + + + +one ranked result + From 7996ff7a39f92e2ef7c94325a674b76d35e42147 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 16:52:33 +0000 Subject: [PATCH 05/20] docs: restructure Lakebase Search hub for architecture-first flow Move the architecture section ahead of the pgvector problem section, drop the Databricks callout box (attribution stays as the inline benchmark link), cut rhetorical lines, and use subheadings and a query-type list for scannability. --- content/docs/ai/lakebase-search.md | 38 +++++++++++++++++++----------- 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index b33aa66df1..23789eb076 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,34 +13,38 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T16:35:36.759Z' +updatedOn: '2026-11-17T16:52:33.861Z' --- Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. - -Lakebase Search is developed by Databricks. The same extensions power search on Neon and on the Databricks Lakebase platform. - +## How Lakebase Search works -## Scaling search past pgvector +Lakebase Search separates storage from compute, the same architecture Neon uses for Postgres. Your indexes live in storage, not in compute memory, so compute stays stateless and scales to zero when the database is idle. -Most teams start vector search in Postgres with `pgvector`, and it holds up until the data grows. The usual `pgvector` index, HNSW (Hierarchical Navigable Small World), has to stay in memory, and that memory grows with the number of vectors, not with how often you query. Past roughly tens of millions of vectors you're paying for a large, always-on instance just to hold the index, index builds get slow enough to disrupt the database, and every tuning pass trades recall against latency. +It adds two search engines to Postgres: `lakebase_vector` for vector search and `lakebase_text` for keyword search, plus `lakebase_tokenizer` for tokenization. Both build ordinary Postgres indexes, so you query them, and combine them, with plain SQL. -The other common fix, moving vectors into a dedicated vector database, swaps those problems for a second system to run and a pipeline to keep in sync. Your search then lags your writes, and you own the plumbing in between. Lakebase Search keeps search in the same database as your data, so a row is searchable the moment it's written: no pipeline, no second system, no sync lag. + -This doesn't mean leaving `pgvector` behind. `lakebase_vector` is a strict superset of `pgvector`, with the same types, operators, and SQL. You keep what you already use and swap in an index built to scale further, without the HNSW memory wall. +### How the vector index scales -## How Lakebase Search works +`lakebase_vector` builds the `lakebase_ann` index. It splits vectors into IVF (Inverted File) lists and scans only the relevant ones, reading sequentially instead of chasing pointers through an in-memory graph. It compresses the vectors with RaBitQ quantization to keep the index small and the build fast. -Lakebase Search separates storage from compute, the architecture Neon already uses for Postgres. Your indexes live in storage instead of compute memory, so compute stays stateless and scales to zero when the database is idle. It adds two search engines to Postgres, `lakebase_vector` for vector search and `lakebase_text` for keyword search, plus `lakebase_tokenizer` for tokenization. +Because the index lives in storage, a cold start needs no rebuild or re-index. The first query pays object-storage latency while the cache warms; call `lakebase_ann_prewarm` ahead of time to skip that. - +### How the text index ranks -`lakebase_vector` builds the `lakebase_ann` index, which splits vectors into IVF (Inverted File) lists and scans only the relevant ones, reading sequentially instead of chasing pointers through an in-memory graph. It compresses the vectors with RaBitQ quantization to keep the index small and the build fast. Because the index lives in storage, a cold start needs no rebuild or re-index; the first query pays object-storage latency while the cache warms, which you can avoid by calling `lakebase_ann_prewarm` ahead of time. +`lakebase_text` builds the `lakebase_bm25` index. It ranks with BM25 and uses Block-Max WAND top-K pushdown to return the top matches directly, instead of scoring every row the way a GIN index does. -`lakebase_text` builds the `lakebase_bm25` index, which adds BM25 ranking with Block-Max WAND top-K pushdown, so it returns the top matches directly instead of scoring every row the way a GIN index does. Because both are ordinary Postgres indexes, hybrid search is just SQL: run both in one query and fuse or rerank the results. +## Scaling search past pgvector + +Most teams start vector search in Postgres with `pgvector`, and the limits show up as the data grows. Its usual index, HNSW (Hierarchical Navigable Small World), has to stay in memory, and that memory grows with the number of vectors, not with how often you query. Past roughly tens of millions of vectors, you pay for a large, always-on instance just to hold the index, index builds get slow enough to disrupt the database, and every tuning pass trades recall against latency. Keeping the index in storage, as `lakebase_vector` does, means the index size no longer sets the instance size. + +A dedicated vector database trades the memory problem for a sync problem: a second system to run and a pipeline to keep in step, so your search lags your writes. Lakebase Search keeps search in the same database as your data, so a row is searchable the moment it's written. No pipeline, no second system, no sync lag. + +`lakebase_vector` is a strict superset of `pgvector`: the same types, operators, and SQL. You keep your existing queries and add the `lakebase_ann` index, which lives in storage and scales past the HNSW memory wall. ## Benchmarks @@ -48,7 +52,13 @@ For throughput and recall figures, Databricks has published benchmarks for Lakeb ## Vector, keyword, and hybrid search -Those two engines back three kinds of query. [`lakebase_vector`](/docs/extensions/lakebase-vector) handles vector (semantic) search through the `lakebase_ann` index: you query with an embedding and get the nearest rows by meaning, for natural-language questions, recommendations, and retrieval-augmented generation (RAG). [`lakebase_text`](/docs/extensions/lakebase-text) handles keyword (full-text) search through the `lakebase_bm25` index, ranking rows by BM25 for names, codes, and exact terms. Hybrid search runs both and merges the results; the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to fuse them with Reciprocal Rank Fusion (RRF) or rerank them with Jev, a reranking model on Neon AI Gateway. For custom tokenization, including Chinese, Japanese, and Korean (CJK) text, [`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer) emits standard `tsvector` values. +Lakebase Search answers three kinds of query, each mapped to an extension you can read about in depth: + +- **Vector (semantic) search** with [`lakebase_vector`](/docs/extensions/lakebase-vector): query the `lakebase_ann` index with an embedding to get the nearest rows by meaning, for natural-language questions, recommendations, and retrieval-augmented generation (RAG). +- **Keyword (full-text) search** with [`lakebase_text`](/docs/extensions/lakebase-text): rank rows by BM25 through the `lakebase_bm25` index, for names, codes, and exact terms. +- **Hybrid search**: run both in one query and merge the results. The [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to fuse them with Reciprocal Rank Fusion (RRF) or rerank them with Jev, a reranking model on Neon AI Gateway. + +For custom tokenization, including Chinese, Japanese, and Korean (CJK) text, [`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer) emits standard `tsvector` values. ## Get started From 3422d6b858dba8ec5796bcba53d0c0de7c83da17 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 16:58:29 +0000 Subject: [PATCH 06/20] docs: plainer wording for pgvector limits on Lakebase Search hub Replace the memory-wall metaphor with a literal storage-vs-RAM statement, turn the sync slogan into a plain sentence, and break the pgvector-limits paragraph into a lead sentence and a scannable list. --- content/docs/ai/lakebase-search.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index 23789eb076..3afbc12160 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T16:52:33.861Z' +updatedOn: '2026-11-17T16:58:30.424Z' --- Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -40,11 +40,19 @@ Because the index lives in storage, a cold start needs no rebuild or re-index. T ## Scaling search past pgvector -Most teams start vector search in Postgres with `pgvector`, and the limits show up as the data grows. Its usual index, HNSW (Hierarchical Navigable Small World), has to stay in memory, and that memory grows with the number of vectors, not with how often you query. Past roughly tens of millions of vectors, you pay for a large, always-on instance just to hold the index, index builds get slow enough to disrupt the database, and every tuning pass trades recall against latency. Keeping the index in storage, as `lakebase_vector` does, means the index size no longer sets the instance size. +Most teams start vector search in Postgres with `pgvector`, and the limits show up as the data grows. Its usual index, HNSW (Hierarchical Navigable Small World), keeps the whole vector index in RAM, so its memory grows with the number of vectors, not with how often you query. -A dedicated vector database trades the memory problem for a sync problem: a second system to run and a pipeline to keep in step, so your search lags your writes. Lakebase Search keeps search in the same database as your data, so a row is searchable the moment it's written. No pipeline, no second system, no sync lag. +Past roughly tens of millions of vectors, you run into several limits: -`lakebase_vector` is a strict superset of `pgvector`: the same types, operators, and SQL. You keep your existing queries and add the `lakebase_ann` index, which lives in storage and scales past the HNSW memory wall. +- a large, always-on instance just to hold the index in memory +- index builds slow enough to disrupt the database +- a recall-versus-latency tradeoff on every tuning pass + +The `lakebase_ann` index lives in storage instead of RAM, so the index size no longer sets the instance size. + +A dedicated vector database trades the memory problem for a sync problem: a second system to run and a pipeline to keep in step, so your search lags your writes. Lakebase Search keeps search in the same database as your data, so a row is searchable the moment it's written. There's no separate search system to keep in sync, and no lag between a write and when its row is searchable. + +`lakebase_vector` is a strict superset of `pgvector`: the same types, operators, and SQL. You keep your existing queries and add the `lakebase_ann` index. ## Benchmarks From 34a71a5013ff0e893ccbea67209be530e5327be4 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 17:30:27 +0000 Subject: [PATCH 07/20] docs: add a hybrid-search SQL example to the Lakebase Search hub Show vector plus keyword search fused with RRF in one statement, right after the architecture diagram, and point to the get-started guide for the runnable version. --- content/docs/ai/lakebase-search.md | 42 +++++++++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index 3afbc12160..f676ccf464 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T16:58:30.424Z' +updatedOn: '2026-11-17T17:30:27.940Z' --- Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -28,6 +28,46 @@ It adds two search engines to Postgres: `lakebase_vector` for vector search and +Both searches and the fusion are one statement. This query takes candidates from the `lakebase_ann` vector index and the `lakebase_bm25` keyword index, then merges them with Reciprocal Rank Fusion (RRF): + +```sql +WITH vector_ranked AS ( + SELECT id, RANK() OVER (ORDER BY dist) AS rank + FROM ( + SELECT id, embedding <=> $1::vector AS dist + FROM documents + WHERE embedding IS NOT NULL + ORDER BY dist + FETCH FIRST 40 ROWS WITH TIES + ) v +), +keyword_ranked AS ( + SELECT id, RANK() OVER (ORDER BY score) AS rank + FROM ( + SELECT id, score + FROM ( + SELECT id, body_tsv <@> to_bm25query( + to_tsvector('english', $2), 'documents_bm25' + ) AS score + FROM documents + ) AS scored + WHERE score < 0 + ORDER BY score, id + LIMIT 40 + ) k +) +SELECT d.id, d.title, + COALESCE(1.0 / (60 + v.rank), 0) + COALESCE(1.0 / (60 + k.rank), 0) AS rrf_score +FROM documents d +LEFT JOIN vector_ranked v ON d.id = v.id +LEFT JOIN keyword_ranked k ON d.id = k.id +WHERE v.id IS NOT NULL OR k.id IS NOT NULL +ORDER BY rrf_score DESC, d.id +LIMIT 5; +``` + +For the full runnable version, including Jev reranking, see the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search). + ### How the vector index scales `lakebase_vector` builds the `lakebase_ann` index. It splits vectors into IVF (Inverted File) lists and scans only the relevant ones, reading sequentially instead of chasing pointers through an in-memory graph. It compresses the vectors with RaBitQ quantization to keep the index small and the build fast. From 2e17f1d59243adfefca8f16a925ca3dd616aa4ca Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 17:42:05 +0000 Subject: [PATCH 08/20] docs: use a short vector-search example on the Lakebase Search hub Swap the long hybrid RRF query for a minimal vector-search snippet after the diagram, and link to the get-started guide for the full hybrid query. --- content/docs/ai/lakebase-search.md | 41 +++++------------------------- 1 file changed, 6 insertions(+), 35 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index f676ccf464..c52d946847 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T17:30:27.940Z' +updatedOn: '2026-11-17T17:42:06.037Z' --- Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -28,45 +28,16 @@ It adds two search engines to Postgres: `lakebase_vector` for vector search and -Both searches and the fusion are one statement. This query takes candidates from the `lakebase_ann` vector index and the `lakebase_bm25` keyword index, then merges them with Reciprocal Rank Fusion (RRF): +Search by meaning is a single SQL query: order rows by the distance between their embedding and your query vector, then take the top matches. ```sql -WITH vector_ranked AS ( - SELECT id, RANK() OVER (ORDER BY dist) AS rank - FROM ( - SELECT id, embedding <=> $1::vector AS dist - FROM documents - WHERE embedding IS NOT NULL - ORDER BY dist - FETCH FIRST 40 ROWS WITH TIES - ) v -), -keyword_ranked AS ( - SELECT id, RANK() OVER (ORDER BY score) AS rank - FROM ( - SELECT id, score - FROM ( - SELECT id, body_tsv <@> to_bm25query( - to_tsvector('english', $2), 'documents_bm25' - ) AS score - FROM documents - ) AS scored - WHERE score < 0 - ORDER BY score, id - LIMIT 40 - ) k -) -SELECT d.id, d.title, - COALESCE(1.0 / (60 + v.rank), 0) + COALESCE(1.0 / (60 + k.rank), 0) AS rrf_score -FROM documents d -LEFT JOIN vector_ranked v ON d.id = v.id -LEFT JOIN keyword_ranked k ON d.id = k.id -WHERE v.id IS NOT NULL OR k.id IS NOT NULL -ORDER BY rrf_score DESC, d.id +SELECT id, title +FROM documents +ORDER BY embedding <=> $1::vector LIMIT 5; ``` -For the full runnable version, including Jev reranking, see the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search). +Keyword search and hybrid search (vector and keyword fused with Reciprocal Rank Fusion (RRF), with optional Jev reranking) run the same way, as SQL. See the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) for the full runnable hybrid query. ### How the vector index scales From 4d12de55eeaa26caf65753cb33cebf7c55baf21e Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 19:41:59 +0000 Subject: [PATCH 09/20] docs: annotate vector example and add cleared benchmark claims Add a lead comment to the vector-search snippet, and replace the benchmarks stub with the cleared comparative result (fastest and most cost-efficient, ~4x over cloud pgvector) attributed to the Databricks blog, with no raw figures. --- content/docs/ai/lakebase-search.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index c52d946847..0630017289 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T17:42:06.037Z' +updatedOn: '2026-11-17T19:42:00.158Z' --- Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -31,6 +31,8 @@ It adds two search engines to Postgres: `lakebase_vector` for vector search and Search by meaning is a single SQL query: order rows by the distance between their embedding and your query vector, then take the top matches. ```sql +-- Semantic search is one SQL query: order rows by the distance +-- between their embedding and the query vector. No separate vector store. SELECT id, title FROM documents ORDER BY embedding <=> $1::vector @@ -67,7 +69,7 @@ A dedicated vector database trades the memory problem for a sync problem: a seco ## Benchmarks -For throughput and recall figures, Databricks has published benchmarks for Lakebase Search at 100-million-vector scale. See the Databricks engineering blog, [Lakebase Search: state-of-the-art full-text and vector search in Postgres](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres). Those results were measured by Databricks on Databricks Lakebase, not on Neon. +On VectorDBBench at 100-million-vector scale, Lakebase Search was the fastest and most cost-efficient system measured, and it ran roughly 4x more cost-effectively than a standard cloud `pgvector` setup. Those results come from the [Databricks Lakebase Search blog](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres), measured by Databricks on Databricks Lakebase, not on Neon. See the blog for the detailed chart. ## Vector, keyword, and hybrid search From f041936c0000c5b1208bb666615821a5a5ea6b93 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 20:48:48 +0000 Subject: [PATCH 10/20] docs: explain IVF, RaBitQ, and Block-Max WAND on the Lakebase Search hub Give IVF and RaBitQ a what-and-why gloss, drop the pointer-chasing metaphor, make the GIN comparison accurate (GIN scores all matching rows), gloss RRF, Jev, and recall on first use. --- content/docs/ai/lakebase-search.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index 0630017289..e6acd8aebd 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T19:42:00.158Z' +updatedOn: '2026-11-17T20:48:49.063Z' --- Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -39,17 +39,17 @@ ORDER BY embedding <=> $1::vector LIMIT 5; ``` -Keyword search and hybrid search (vector and keyword fused with Reciprocal Rank Fusion (RRF), with optional Jev reranking) run the same way, as SQL. See the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) for the full runnable hybrid query. +Keyword search and hybrid search run the same way, as SQL. Hybrid search fuses the vector and keyword result lists with Reciprocal Rank Fusion (RRF), which merges two ranked lists into one, and can optionally rerank them with Jev, a reranking model on Neon AI Gateway. See the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) for the full runnable hybrid query. ### How the vector index scales -`lakebase_vector` builds the `lakebase_ann` index. It splits vectors into IVF (Inverted File) lists and scans only the relevant ones, reading sequentially instead of chasing pointers through an in-memory graph. It compresses the vectors with RaBitQ quantization to keep the index small and the build fast. +`lakebase_vector` builds the `lakebase_ann` index, which combines two techniques. IVF (Inverted File) partitioning groups the vectors into lists and scans only the lists closest to your query, so a search reads a small part of the index with sequential reads instead of random reads across the whole index. RaBitQ quantization compresses each vector into a smaller form, which keeps the index small and makes index builds fast. Because the index lives in storage, a cold start needs no rebuild or re-index. The first query pays object-storage latency while the cache warms; call `lakebase_ann_prewarm` ahead of time to skip that. ### How the text index ranks -`lakebase_text` builds the `lakebase_bm25` index. It ranks with BM25 and uses Block-Max WAND top-K pushdown to return the top matches directly, instead of scoring every row the way a GIN index does. +`lakebase_text` builds the `lakebase_bm25` index. It ranks results with BM25, and it uses Block-Max WAND, an algorithm that finds the top K matches without fully scoring every matching row, so text search stays fast as the index grows. A standard GIN full-text index scores all matching rows before returning the top ones; `lakebase_bm25` returns the top K directly. ## Scaling search past pgvector @@ -59,7 +59,7 @@ Past roughly tens of millions of vectors, you run into several limits: - a large, always-on instance just to hold the index in memory - index builds slow enough to disrupt the database -- a recall-versus-latency tradeoff on every tuning pass +- a recall-versus-latency tradeoff on every tuning pass (recall is the share of true matches returned) The `lakebase_ann` index lives in storage instead of RAM, so the index size no longer sets the instance size. From 55f1d342a55f4a39a0119e319292503df5644123 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 21:03:40 +0000 Subject: [PATCH 11/20] docs: plainer search glosses and a which-search-to-use section Swap the IVF, RaBitQ, Block-Max WAND, RRF, and recall glosses for plainer wording, and reframe the search-types section into a short decision aid without the duplicated extension links. --- content/docs/ai/lakebase-search.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index e6acd8aebd..a3fc37cdc2 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T20:48:49.063Z' +updatedOn: '2026-11-17T21:03:40.986Z' --- Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -39,17 +39,17 @@ ORDER BY embedding <=> $1::vector LIMIT 5; ``` -Keyword search and hybrid search run the same way, as SQL. Hybrid search fuses the vector and keyword result lists with Reciprocal Rank Fusion (RRF), which merges two ranked lists into one, and can optionally rerank them with Jev, a reranking model on Neon AI Gateway. See the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) for the full runnable hybrid query. +Keyword search and hybrid search run the same way, as SQL. Hybrid search combines the vector and keyword results into one ranked list (using Reciprocal Rank Fusion, or RRF), and can optionally reorder them with Jev, a reranking model on Neon AI Gateway. See the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) for the full runnable hybrid query. ### How the vector index scales -`lakebase_vector` builds the `lakebase_ann` index, which combines two techniques. IVF (Inverted File) partitioning groups the vectors into lists and scans only the lists closest to your query, so a search reads a small part of the index with sequential reads instead of random reads across the whole index. RaBitQ quantization compresses each vector into a smaller form, which keeps the index small and makes index builds fast. +`lakebase_vector` builds the `lakebase_ann` index. It sorts vectors into groups and only searches the groups closest to your query, so it reads a small part of the index instead of scanning all of it (this grouping is called IVF, short for Inverted File). It also shrinks each vector into a smaller form, which keeps the index small and quick to build (this compression is called RaBitQ). Because the index lives in storage, a cold start needs no rebuild or re-index. The first query pays object-storage latency while the cache warms; call `lakebase_ann_prewarm` ahead of time to skip that. ### How the text index ranks -`lakebase_text` builds the `lakebase_bm25` index. It ranks results with BM25, and it uses Block-Max WAND, an algorithm that finds the top K matches without fully scoring every matching row, so text search stays fast as the index grows. A standard GIN full-text index scores all matching rows before returning the top ones; `lakebase_bm25` returns the top K directly. +`lakebase_text` builds the `lakebase_bm25` index. It ranks results with BM25 and finds the best matches without scoring every matching row, so text search stays fast as your data grows (the technique for this is called Block-Max WAND). A standard Postgres full-text index (GIN) scores all the matching rows first and then picks the top ones; `lakebase_bm25` goes straight to the best matches. ## Scaling search past pgvector @@ -59,7 +59,7 @@ Past roughly tens of millions of vectors, you run into several limits: - a large, always-on instance just to hold the index in memory - index builds slow enough to disrupt the database -- a recall-versus-latency tradeoff on every tuning pass (recall is the share of true matches returned) +- a recall-versus-latency tradeoff on every tuning pass (recall means how many of the true matches you actually get back) The `lakebase_ann` index lives in storage instead of RAM, so the index size no longer sets the instance size. @@ -71,15 +71,15 @@ A dedicated vector database trades the memory problem for a sync problem: a seco On VectorDBBench at 100-million-vector scale, Lakebase Search was the fastest and most cost-efficient system measured, and it ran roughly 4x more cost-effectively than a standard cloud `pgvector` setup. Those results come from the [Databricks Lakebase Search blog](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres), measured by Databricks on Databricks Lakebase, not on Neon. See the blog for the detailed chart. -## Vector, keyword, and hybrid search +## Which search to use: vector, keyword, or hybrid -Lakebase Search answers three kinds of query, each mapped to an extension you can read about in depth: +Pick the search that fits the query: -- **Vector (semantic) search** with [`lakebase_vector`](/docs/extensions/lakebase-vector): query the `lakebase_ann` index with an embedding to get the nearest rows by meaning, for natural-language questions, recommendations, and retrieval-augmented generation (RAG). -- **Keyword (full-text) search** with [`lakebase_text`](/docs/extensions/lakebase-text): rank rows by BM25 through the `lakebase_bm25` index, for names, codes, and exact terms. -- **Hybrid search**: run both in one query and merge the results. The [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to fuse them with Reciprocal Rank Fusion (RRF) or rerank them with Jev, a reranking model on Neon AI Gateway. +- **Vector (semantic) search** when you care about meaning: natural-language questions, recommendations, and retrieval-augmented generation (RAG). +- **Keyword (full-text) search** when the exact words matter: names, codes, and other precise terms. +- **Hybrid search** when you want both. It runs the vector and keyword searches and merges them into one ranked list. -For custom tokenization, including Chinese, Japanese, and Korean (CJK) text, [`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer) emits standard `tsvector` values. +For Chinese, Japanese, and Korean (CJK) text or other custom tokenization, add `lakebase_tokenizer`. ## Get started From d66c00ccac8e9c4283da71c624e70f4e51cfedcb Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 21:31:09 +0000 Subject: [PATCH 12/20] docs: group Lakebase Search docs under one sidebar section Give the Lakebase Search product entry an Extension reference section with the three extension pages, move those pages out of the alphabetical Postgres extensions list, drop the duplicate nested entry, and remove the Platform cross-link so the Overview breadcrumb resolves cleanly. --- content/docs/navigation.yaml | 24 ++++++++---------------- 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/content/docs/navigation.yaml b/content/docs/navigation.yaml index e5da17a796..a42b14c10b 100644 --- a/content/docs/navigation.yaml +++ b/content/docs/navigation.yaml @@ -346,12 +346,6 @@ slug: extensions/hstore - title: intarray slug: extensions/intarray - - title: lakebase_text - slug: extensions/lakebase-text - - title: lakebase_tokenizer - slug: extensions/lakebase-tokenizer - - title: lakebase_vector - slug: extensions/lakebase-vector - title: ltree slug: extensions/ltree - title: neon @@ -449,14 +443,6 @@ slug: ai/ai-scale-with-neon - title: Google Colab slug: ai/ai-google-colab - - title: Lakebase Search - icon: sparks - slug: ai/lakebase-search - items: - - title: Overview - slug: ai/lakebase-search - - title: Get started - slug: ai/lakebase-search-get-started - title: Postgres guides slug: postgresql/introduction icon: book @@ -1048,6 +1034,14 @@ slug: ai/lakebase-search - title: Get started slug: ai/lakebase-search-get-started + - section: Extension reference + items: + - title: lakebase_vector + slug: extensions/lakebase-vector + - title: lakebase_text + slug: extensions/lakebase-text + - title: lakebase_tokenizer + slug: extensions/lakebase-tokenizer - nav: Platform title: Platform @@ -1323,8 +1317,6 @@ slug: ai/ai-database-versioning - title: Neon API Reference slug: reference/api - - title: Lakebase Search - slug: ai/lakebase-search - section: Agent frameworks icon: frameworks items: From b18ad6316cf08bd1ccbfab0ed41513facba0b7fd Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 21:48:30 +0000 Subject: [PATCH 13/20] docs: tighten the Lakebase Search benchmark line Use present tense and link the Databricks benchmark for the full results. --- content/docs/ai/lakebase-search.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index a3fc37cdc2..648b24723f 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T21:03:40.986Z' +updatedOn: '2026-11-17T21:48:30.966Z' --- Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -69,7 +69,7 @@ A dedicated vector database trades the memory problem for a sync problem: a seco ## Benchmarks -On VectorDBBench at 100-million-vector scale, Lakebase Search was the fastest and most cost-efficient system measured, and it ran roughly 4x more cost-effectively than a standard cloud `pgvector` setup. Those results come from the [Databricks Lakebase Search blog](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres), measured by Databricks on Databricks Lakebase, not on Neon. See the blog for the detailed chart. +On VectorDBBench at 100-million-vector scale, Lakebase Search is the fastest and most cost-efficient system measured, running roughly 4x more cost-effectively than a standard cloud `pgvector` setup. See the [Databricks benchmark](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres) for the full results. ## Which search to use: vector, keyword, or hybrid From d8d5faf4c4ea7f634acd9607b2759b94092076eb Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 22:25:40 +0000 Subject: [PATCH 14/20] docs: retire the vector-search guide and redirect to Lakebase Search Remove the stale pgvector plus HNSW tutorial, 308-redirect /guides/vector-search to the Lakebase Search get-started page, and repoint its inbound links. --- .../ten-most-popular-postgres-extensions.md | 2 +- content/docs/extensions/pgvector.md | 4 +- content/guides/vector-search.md | 186 ------------------ next.config.js | 5 + 4 files changed, 8 insertions(+), 189 deletions(-) delete mode 100644 content/guides/vector-search.md diff --git a/content/blog/posts/ten-most-popular-postgres-extensions.md b/content/blog/posts/ten-most-popular-postgres-extensions.md index 88d1858a0e..611736ae7c 100644 --- a/content/blog/posts/ten-most-popular-postgres-extensions.md +++ b/content/blog/posts/ten-most-popular-postgres-extensions.md @@ -121,7 +121,7 @@ ORDER BY distance LIMIT 5; ``` -[Check out this guide](https://neon.com/guides/vector-search) to get started with pgvector. +[Check out this guide](https://neon.com/docs/ai/lakebase-search-get-started) to get started with vector search. ### pgcrypto diff --git a/content/docs/extensions/pgvector.md b/content/docs/extensions/pgvector.md index 3813a26139..9efef6bd85 100644 --- a/content/docs/extensions/pgvector.md +++ b/content/docs/extensions/pgvector.md @@ -13,7 +13,7 @@ summary: >- enableTableOfContents: true redirectFrom: - /docs/extensions/pg_embedding -updatedOn: '2026-11-06T11:14:41.532Z' +updatedOn: '2026-11-17T22:25:43.741Z' --- @@ -684,7 +684,7 @@ LINE 1: SELECT '[4e38,1]'::vector; ## Resources - `pgvector` source code: [https://github.com/pgvector/pgvector](https://github.com/pgvector/pgvector) -- [Vector search in Postgres](/guides/vector-search) +- [Get started with Lakebase Search](/docs/ai/lakebase-search-get-started) - [Building semantic search with AI embeddings, Neon, and pgvector](/guides/ai-embeddings-postgres-search) diff --git a/content/guides/vector-search.md b/content/guides/vector-search.md deleted file mode 100644 index b7849654d7..0000000000 --- a/content/guides/vector-search.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -title: Vector search in Postgres -subtitle: A step-by-step guide describing how to use pgvector for vector search in Postgres -author: vkarpov15 -enableTableOfContents: true -createdAt: '2025-02-04T13:24:36.612Z' -updatedOn: '2026-09-30T00:39:07.369Z' ---- - - - -## Build vector search with pgvector - -Vector search lets you run similarity searches on vectors stored in Postgres. -With the [pgvector](https://github.com/pgvector/pgvector) extension, you can store and query vector embeddings in Postgres for applications like retrieval-augmented generation (RAG) and semantic search. - -## Steps - -- Install and enable pgvector -- Create a table with a vector column -- Insert and retrieve vector data -- Perform nearest neighbor searches -- Index using HNSW indexes -- Insert and retrieve embeddings - -## Install and enable pgvector - -Before using vector search, you need to install the [`pgvector` extension](https://github.com/pgvector/pgvector). -The `pgvector` extension adds a `vector` data type, operators for similarity search (`<->`, `<#>`, `<=>`), and support for ANN indexes. -On Neon, `pgvector` is already installed. You only need to enable it with the following command. - -```sql -CREATE EXTENSION IF NOT EXISTS vector; -``` - -## Create a table with a vector column - -To store embeddings, you need a table with a column defined as the `vector` data type. You must specify the dimensions of the vector (also known as _dimensionality_), which is determined by the embedding model you plan to use (e.g., OpenAI's `text-embedding-3-small` uses 1536 dimensions). - -For example, the following command creates a table `embeddings` with a 3-dimensional vector column `data`. - -```sql -CREATE TABLE embeddings ( - id SERIAL PRIMARY KEY, - data VECTOR(3) -- 3-dimensional vector example -); -``` - -## Insert and retrieve vector data - -You can insert vectors as arrays using the following command. -Under the hood, vectors are just fixed-length arrays of floats. - -```sql -INSERT INTO embeddings (data) -VALUES ('[0.1, 0.2, 0.3]'), - ('[0.5, 0.1, 0.8]'); -``` - -You can retrieve all stored vectors using the following command. - -```sql -SELECT * FROM embeddings; -``` - -## Perform nearest neighbor searches - -Vector search typically means finding the closest vectors in the database to a given vector. -There are different distance metrics to calculate which vector is closest, like Euclidean distance (`<->`), cosine distance (`<=>`), and inner product (`<#>`). - -- `<->`: Euclidean distance (L2). Measures straight-line distance between two vectors. Good for general similarity tasks when magnitude matters. -- `<=>`: Cosine distance. Compares the angle between two vectors. Common for text embeddings, where direction matters more than magnitude. -- `<#>`: Negative inner product. Used for maximizing similarity. Often used in ranking or recommendation systems. - -For example, the following command runs nearest neighbor search to find the most similar vector to `[0.2, 0.1, 0.3]` using Euclidean distance, which is `[0.1, 0.2, 0.3]`. - -```sql -SELECT * FROM embeddings -ORDER BY data <-> '[0.2, 0.1, 0.3]' -LIMIT 1; -``` - -You should see the following output. - -``` - id | data -----+-------------- - 1 | [0.1,0.2,0.3] -(1 row) -``` - -## Index using HNSW indexes - -By default, the query above performs a sequential scan of the `embeddings` table, which can be slow for large datasets. To speed up nearest neighbor searches, you can create an approximate nearest neighbor (ANN) index. - -`pgvector` supports two different indexes for nearest neighbor search: HNSW and IVFFlat. - -The following command creates a HNSW index. - -```sql -CREATE INDEX ON embeddings USING hnsw (data); -``` - -## Insert and retrieve embeddings - -Vector databases are typically used to store _embeddings_. -An embedding is a numerical representation of data in a high-dimensional space that captures semantic relationships and similarities between entities. -First, run the following command to recreate the `embeddings` table to store vectors with dimensionality 512. - -```sql -DROP TABLE embeddings; - -CREATE TABLE embeddings ( - id SERIAL PRIMARY KEY, - content TEXT, - data VECTOR(512) -); -``` - -### How to generate embeddings - -In most cases, embeddings are created using an embeddings API such as the [Neon AI Gateway](/docs/ai-gateway/embeddings), [OpenAI](https://platform.openai.com/docs/guides/embeddings), or [Gemini](https://ai.google.dev/gemini-api/docs/embeddings). Once generated, they can be stored in Postgres for vector search with `pgvector`. The dimensionality of the `vector` column must match the model you use (for example, 512, 768, or 1536 dimensions), and your query vectors must use the same model. - -The Neon AI Gateway serves embedding models on an OpenAI-compatible endpoint, using the same Neon credential as the rest of your project. Point the `openai` SDK at your branch's gateway URL: - -```javascript shouldWrap -import OpenAI from 'openai'; - -const client = new OpenAI({ - apiKey: process.env.NEON_AI_GATEWAY_TOKEN, - baseURL: `${process.env.NEON_AI_GATEWAY_BASE_URL}/v1`, -}); - -const response = await client.embeddings.create({ - model: 'qwen3-embedding-0-6b', // returns 1024-dimensional vectors - input: 'i like to eat tacos', - encoding_format: 'float', // required on the openai SDK v6; harmless on v7+ -}); - -const embedding = response.data[0].embedding; // store this in a VECTOR(1024) column -``` - -Size the `vector` column to match the model (1024 for the gateway models above). See [Embeddings](/docs/ai-gateway/embeddings) for the full model list. - -For this example, we will use embeddings generated from the [Nomic API](https://docs.nomic.ai/reference/api/embed-text-v-1-embedding-text-post). - -### Insert embeddings - -You can insert embeddings into the `embeddings` table using the following SQL command. - -```sql -INSERT INTO embeddings (content, data) -VALUES - ('i like to eat tacos', '[0.017120361,0.09112549,-0.24157715,0.0045776367,-0.024642944,0.0062828064,-0.06707764,0.022094727,-0.022232056,-0.019546509,0.010147095,0.05722046,0.027832031,0.07006836,-0.0051574707,-0.041259766,0.0008292198,-0.08605957,0.014213562,0.10180664,-0.045318604,-0.046447754,-0.002002716,-0.04144287,0.11590576,0.0093688965,-0.019638062,0.08929443,-0.057739258,0.031173706,-0.030471802,-0.07293701,0.019317627,0.100097656,0.017288208,-0.053222656,0.082092285,0.018234253,0.024536133,-0.0541687,-0.027191162,0.038635254,0.05657959,-0.050445557,0.06378174,-0.015579224,0.0736084,0.059173584,0.029037476,-0.03451538,-0.030151367,-0.027633667,-0.038604736,-0.06750488,-0.0038433075,-0.06210327,0.055664062,-0.06677246,-0.01828003,0.025848389,0.10809326,0.021942139,0.016067505,0.08532715,0.02708435,0.031311035,-0.046691895,0.078125,-0.07287598,-0.021347046,0.07159424,-0.0037384033,0.03878784,0.014350891,-0.02381897,-0.04309082,-0.031463623,-0.00541687,-0.03274536,-0.015464783,-0.0046539307,-0.017654419,0.08538818,-0.025238037,0.035949707,-0.012565613,0.0625,-0.057647705,-0.0418396,0.052825928,0.024276733,0.002412796,0.051452637,0.01663208,-0.029724121,0.035247803,-0.025817871,0.046081543,-0.007888794,-0.05114746,-0.036346436,0.017074585,0.009651184,-0.010925293,0.103759766,0.022567749,0.056121826,-0.0058555603,0.0362854,0.0031356812,0.03062439,0.042755127,-0.026870728,-0.05215454,-0.006095886,0.00006586313,0.010673523,-0.09136963,0.033721924,0.040740967,-0.01991272,-0.01953125,0.00033140182,0.05831909,0.015686035,0.024383545,-0.005264282,0.022613525,-0.048858643,-0.028945923,-0.002817154,0.03781128,0.014976501,-0.014030457,0.011795044,0.06008911,0.03262329,-0.066101074,0.015686035,0.008361816,0.005657196,0.06335449,0.051635742,-0.015274048,0.02571106,-0.044281006,0.0140686035,-0.09503174,0.011451721,-0.039886475,-0.02571106,0.0073432922,-0.0067329407,0.042541504,-0.022781372,-0.061798096,0.025634766,-0.05718994,-0.0023117065,0.015312195,0.04937744,-0.029815674,-0.009246826,0.05505371,0.014663696,-0.049468994,-0.0051002502,0.06573486,0.030593872,0.07922363,-0.026885986,-0.019348145,-0.051452637,-0.06427002,-0.04324341,-0.076171875,-0.09637451,-0.03753662,0.04888916,-0.017456055,0.02520752,-0.070129395,0.0022792816,0.08203125,-0.038635254,-0.044769287,-0.0020809174,0.025283813,-0.06549072,-0.028427124,0.011878967,0.010292053,-0.07965088,-0.05239868,-0.03062439,0.025115967,0.033081055,0.0035209656,0.014038086,-0.038909912,-0.023147583,-0.03616333,-0.10192871,0.027648926,-0.054382324,0.030395508,-0.05493164,-0.0048446655,-0.03756714,0.022705078,0.06274414,-0.030807495,-0.023605347,-0.02330017,-0.026519775,-0.034210205,-0.004245758,-0.014305115,-0.014213562,0.03845215,0.045684814,-0.014465332,0.009208679,-0.032562256,0.022567749,-0.027557373,-0.0033683777,-0.038085938,-0.04937744,-0.022033691,-0.014198303,-0.07611084,0.14099121,0.003921509,0.034576416,0.05404663,0.066345215,0.0847168,-0.0026435852,-0.051452637,-0.013175964,0.01701355,0.034820557,-0.039642334,-0.05734253,0.039093018,-0.004928589,-0.052215576,-0.027740479,0.050689697,0.049041748,-0.016693115,0.015731812,-0.01158905,0.024597168,-0.01878357,-0.012107849,0.040100098,0.031158447,-0.06994629,0.045135498,-0.10028076,0.033843994,-0.08734131,-0.021850586,-0.009010315,-0.03894043,0.052642822,-0.015525818,-0.07067871,0.023330688,0.011230469,-0.00018012524,0.046447754,-0.06591797,0.019104004,0.02494812,-0.0345459,-0.03277588,-0.0038433075,0.031051636,-0.03744507,-0.011779785,0.031234741,0.0041542053,0.070373535,0.023498535,0.0054016113,-0.011703491,-0.0067710876,0.04724121,0.06185913,-0.025558472,0.040130615,0.03439331,0.013008118,0.08886719,-0.032836914,-0.032958984,-0.043029785,-0.009384155,0.04269409,0.037475586,0.022415161,-0.038513184,0.035064697,0.07702637,-0.057861328,0.06274414,-0.028869629,0.0027332306,-0.024215698,-0.0067977905,0.07885742,-0.047668457,0.03137207,-0.020477295,0.0036449432,0.053375244,-0.002811432,0.03074646,-0.051513672,-0.0021152496,-0.05166626,-0.03869629,0.012924194,0.03878784,0.05831909,0.014884949,-0.07141113,0.001496315,0.01776123,0.03353882,-0.030471802,-0.028747559,0.028167725,0.068725586,0.025894165,-0.030807495,0.05807495,-0.007843018,-0.028762817,0.018737793,-0.04714966,-0.03149414,-0.007259369,-0.057128906,0.014770508,0.095458984,0.016723633,-0.039123535,0.02015686,-0.022628784,0.04852295,-0.0047912598,-0.026687622,0.055267334,-0.048736572,0.014633179,-0.005859375,0.02470398,-0.026916504,0.01083374,-0.010940552,-0.007030487,0.027557373,0.027526855,-0.015853882,0.013328552,0.030960083,-0.048919678,-0.051086426,-0.017242432,0.04147339,-0.004863739,0.017288208,-0.13586426,-0.035247803,0.057891846,-0.037750244,-0.0022220612,0.01576233,-0.057861328,0.039489746,0.055114746,0.037200928,0.04522705,0.0023956299,-0.030136108,-0.004131317,-0.006034851,-0.02619934,-0.07397461,-0.008293152,0.027572632,-0.061828613,0.07537842,-0.038635254,0.031341553,-0.002708435,-0.022384644,-0.057861328,0.00024557114,-0.024810791,-0.047729492,0.06677246,-0.030838013,-0.052520752,0.0579834,-0.03805542,-0.010284424,0.06323242,0.04699707,-0.030380249,-0.0010614395,0.0057678223,0.04824829,-0.014038086,0.016036987,-0.026031494,0.02708435,0.05987549,-0.0025463104,0.030838013,-0.046691895,0.041381836,-0.008102417,0.08227539,0.006324768,-0.07458496,-0.058410645,-0.0014505386,0.04196167,-0.014968872,-0.04714966,-0.03579712,-0.085876465,0.013183594,-0.005340576,0.06896973,0.012649536,-0.029388428,-0.11816406,-0.044128418,0.052124023,0.074645996,0.06384277,-0.023330688,0.019119263,-0.0146865845,0.02279663,0.015640259,0.05090332,0.021072388,0.09814453,-0.066711426,0.02671814,-0.027130127,0.038757324,-0.019317627,0.03741455,0.02746582,-0.03463745,0.00001001358,0.01638794,-0.0362854,0.02861023,0.0057754517,0.045562744,0.013206482,-0.010543823,0.03213501,0.044952393,-0.00018644333,-0.0040397644,-0.027236938,-0.026794434,0.004016876,0.016860962,0.0949707,-0.0129852295,0.024124146,-0.06185913,-0.08068848,-0.005054474,0.037353516,0.028411865,0.008850098,0.04940796,0.018356323,0.008979797,0.016098022,0.013702393,-0.03942871,0.03463745,0.006729126,-0.042541504,0.02607727,0.06451416,-0.029632568,-0.029647827,-0.014167786,-0.01675415,-0.0017442703,0.07269287,0.00013709068,-0.044708252,-0.059417725,-0.097839355,0.013648987,-0.0041923523,0.0025196075]'), - ('An embedding is a numerical representation of data in a high-dimensional space that captures semantic relationships and similarities between entities.', '[0.043701172,0.09063721,-0.24499512,-0.1385498,0.025177002,-0.020385742,0.0074653625,-0.016143799,-0.08294678,-0.03427124,-0.04864502,0.003490448,0.1060791,0.035461426,-0.023712158,0.04220581,-0.016342163,-0.039001465,-0.06008911,0.034362793,-0.048858643,0.023666382,0.012779236,0.012001038,0.057250977,0.038970947,0.08312988,-0.046936035,-0.02229309,0.00674057,0.04751587,-0.015289307,0.0027866364,-0.028762817,-0.043548584,-0.037200928,-0.0045547485,0.09539795,0.026290894,0.018051147,0.0000140070915,-0.002450943,-0.04751587,0.013076782,-0.031982422,-0.0035572052,0.044952393,-0.04220581,0.08660889,-0.037109375,-0.010673523,-0.013221741,-0.015609741,-0.028411865,0.11138916,0.02218628,0.008255005,0.015991211,0.043395996,-0.044189453,0.09460449,0.1005249,-0.06817627,0.09283447,0.0625,0.026916504,-0.097961426,0.05682373,-0.011253357,-0.085510254,0.10241699,-0.010391235,0.03656006,0.028320312,-0.025604248,-0.0149383545,-0.00881958,0.0362854,-0.002401352,0.052734375,0.04220581,0.03640747,0.09686279,-0.040527344,0.09460449,0.045043945,-0.010475159,0.00006771088,-0.06567383,0.060913086,0.016830444,0.009277344,0.02458191,0.05444336,-0.024734497,0.006401062,-0.00166893,0.028289795,-0.033447266,-0.03704834,-0.055389404,-0.01486969,-0.021697998,0.01322937,-0.005695343,0.053649902,-0.000044941902,0.026565552,-0.06561279,0.022399902,-0.022094727,0.015525818,-0.06402588,-0.06585693,-0.0055732727,-0.018295288,0.09020996,-0.07720947,-0.014472961,0.057434082,0.01537323,-0.041870117,0.042419434,0.05392456,0.007080078,0.011199951,-0.020095825,0.007774353,-0.044433594,-0.04031372,-0.016448975,-0.060394287,-0.009780884,0.010131836,0.005207062,0.038879395,-0.048675537,-0.024917603,-0.0069351196,0.08514404,-0.0041885376,-0.015586853,-0.0029888153,-0.0546875,0.008361816,-0.09490967,0.035705566,-0.02935791,0.009742737,-0.015213013,-0.00970459,0.08270264,-0.03753662,-0.045074463,0.01612854,-0.0030441284,0.024749756,0.0041542053,0.064697266,-0.007019043,0.038970947,0.04284668,-0.030029297,0.04623413,0.019699097,-0.074523926,-0.0024147034,0.019836426,0.011489868,0.009597778,-0.04751587,-0.03125,-0.023025513,-0.0064201355,0.0007266998,-0.007888794,0.036834717,-0.068359375,0.056671143,0.006175995,0.021530151,-0.04324341,0.07232666,-0.004169464,-0.025619507,-0.019226074,-0.007259369,-0.01902771,-0.060760498,-0.03161621,-0.055877686,0.012390137,-0.031280518,-0.00705719,-0.019470215,-0.00061893463,0.06774902,-0.034301758,-0.003293991,-0.023925781,-0.007820129,0.011604309,-0.024002075,0.05206299,-0.012214661,0.043304443,-0.04232788,-0.0005931854,-0.050872803,0.04647827,0.06555176,-0.017486572,0.001001358,0.010131836,0.04776001,-0.0076560974,0.0063323975,-0.04147339,-0.02243042,-0.00008791685,0.028717041,-0.01927185,0.039794922,-0.0769043,0.03289795,-0.019439697,-0.03137207,0.047088623,-0.045532227,0.0011854172,-0.03768921,-0.04663086,0.044525146,-0.031173706,0.011817932,0.06109619,-0.01701355,0.06524658,0.006614685,0.037841797,0.018707275,0.053833008,-0.02468872,-0.03387451,-0.02897644,0.05923462,0.024429321,-0.060516357,-0.0435791,0.07159424,-0.04446411,-0.036712646,0.012107849,0.007286072,0.07183838,-0.031829834,-0.047790527,-0.07092285,0.014518738,-0.008964539,0.05621338,-0.017486572,-0.0129470825,-0.036499023,-0.06890869,-0.021835327,0.027175903,-0.007709503,0.02960205,-0.02003479,0.01058197,0.017303467,0.018112183,0.019622803,0.011024475,-0.013412476,0.02229309,-0.012329102,-0.05053711,0.01197052,-0.05316162,-0.018341064,-0.07086182,0.0146865845,-0.018798828,0.021240234,0.036895752,0.020812988,0.025863647,0.031097412,0.037475586,-0.042053223,-0.03768921,0.04321289,-0.00054073334,0.045806885,0.06732178,-0.001572609,0.049682617,-0.064086914,-0.010314941,0.049835205,0.099975586,0.011741638,-0.0135269165,-0.033843994,0.040924072,-0.056121826,0.020202637,-0.04135132,-0.06286621,-0.0056762695,-0.054840088,0.0025749207,-0.0647583,0.051208496,0.027694702,-0.00026249886,0.03201294,-0.07409668,-0.005104065,-0.12463379,0.036010742,-0.031173706,0.0036354065,0.07354736,-0.050201416,0.013839722,-0.01612854,0.021835327,-0.039001465,0.012069702,0.055603027,-0.005886078,-0.034606934,0.017791748,-0.02961731,-0.033721924,0.011795044,0.0029697418,0.08337402,-0.008636475,0.02470398,-0.09301758,0.026794434,0.03869629,-0.061767578,-0.004070282,0.04171753,0.021850586,-0.03186035,0.00680542,0.009895325,-0.032104492,0.022888184,-0.0076675415,0.0440979,-0.00548172,-0.006793976,-0.0138168335,0.060913086,-0.0035152435,-0.02609253,-0.053619385,-0.0090789795,0.012084961,0.03604126,0.040924072,0.020462036,0.031585693,0.0057411194,-0.0006456375,-0.060272217,0.042297363,0.04827881,-0.0340271,-0.087646484,-0.06738281,0.005554199,-0.014373779,-0.017181396,0.03753662,0.015686035,0.005493164,0.037750244,-0.0031909943,0.035125732,0.00712204,0.017791748,0.007865906,0.004673004,-0.015129089,-0.052978516,0.01751709,0.026031494,-0.06939697,-0.018112183,0.010276794,0.03741455,-0.010620117,-0.014030457,-0.066223145,0.0015687943,-0.023376465,-0.0043296814,-0.029556274,-0.008255005,-0.07354736,0.044281006,-0.031341553,-0.0026378632,0.049835205,0.03503418,-0.10040283,0.0003578663,0.039642334,0.037841797,0.040100098,-0.017211914,0.014572144,0.019897461,0.101989746,-0.03503418,0.025268555,0.040802002,-0.015068054,0.006248474,0.0960083,0.016464233,-0.050231934,-0.015098572,0.041625977,0.062927246,0.0340271,-0.034210205,-0.026412964,-0.045013428,-0.0032138824,-0.0058021545,0.07849121,0.009056091,-0.06359863,-0.019699097,-0.016143799,0.016113281,0.12384033,-0.0044937134,-0.01789856,-0.08276367,0.069885254,0.07110596,0.018173218,-0.0017271042,0.033447266,0.12609863,-0.036712646,-0.012878418,-0.042633057,-0.00087690353,-0.00091171265,0.002943039,0.04800415,-0.08984375,0.035003662,-0.004058838,-0.058410645,-0.007270813,-0.07141113,0.068237305,0.10491943,-0.012290955,0.02571106,0.020767212,-0.03253174,0.04916382,0.05633545,-0.008430481,-0.052886963,0.026992798,0.016601562,0.014930725,0.0026130676,-0.07116699,-0.031280518,0.0006713867,0.049682617,0.012771606,0.00046944618,0.034973145,0.02885437,0.020858765,-0.050842285,0.04437256,0.015289307,-0.027572632,-0.11541748,-0.008483887,0.005844116,0.037109375,-0.0057868958,0.03164673,0.06451416,0.000603199,0.004924774,0.053344727,-0.027374268,0.08270264,-0.04724121,-0.11883545,-0.010147095,0.008865356,0.044281006]'); -``` - -The above SQL command inserts two rows into the `embeddings` table: - -- Row 1: The text "i like to eat tacos" and its corresponding embedding vector. -- Row 2: A longer text about embeddings itself: "An embedding is a numerical representation of data in a high-dimensional space that captures semantic relationships and similarities between entities." and its corresponding embedding vector. - -The embedding vectors are generated using the Nomic API's `nomic-embed-text-v1.5` model with 512 dimensions. - -You can then query for which embeddings are closest to a new vector. -For example, the following query finds the closest vector to the embedding for "burgers are tasty" using the cosine distance operator `<=>`. - -```sql -SELECT content, data <=> '[0.001080513,0.08959961,-0.29296875,0.0014181137,-0.019119263,0.021392822,-0.015617371,0.0345459,-0.0690918,0.009246826,-0.018981934,0.091796875,0.0041160583,0.02947998,-0.021835327,-0.03503418,-0.07702637,-0.07989502,-0.021102905,0.09667969,0.024597168,0.0124053955,0.027420044,-0.039001465,0.10235596,0.008583069,-0.06512451,0.08111572,-0.031982422,0.013595581,0.009635925,-0.036315918,-0.08148193,-0.014015198,0.0082092285,-0.0793457,0.0597229,0.024673462,-0.032440186,-0.047332764,0.0021572113,0.037597656,0.009010315,-0.019104004,0.03967285,0.011817932,0.02178955,0.06695557,0.091308594,-0.020004272,-0.022216797,-0.051361084,-0.031402588,-0.076416016,0.050109863,-0.00223732,0.07714844,0.0385437,0.032440186,0.016860962,0.08496094,0.039123535,-0.026733398,0.044921875,0.034698486,-0.025970459,-0.046142578,0.09326172,-0.030349731,0.022888184,0.06933594,-0.04663086,0.049987793,-0.011802673,-0.015655518,0.013885498,-0.04559326,0.00554657,-0.032836914,-0.007724762,0.013305664,0.01574707,0.0519104,0.024887085,0.044128418,0.010795593,0.055603027,-0.054901123,-0.045837402,0.036468506,0.047302246,0.024810791,0.05645752,0.062805176,-0.07165527,0.027282715,-0.011566162,0.040283203,-0.019454956,-0.04058838,-0.032196045,0.024856567,0.0023899078,0.016921997,0.062316895,0.054748535,0.04498291,0.012550354,-0.0043678284,-0.01309967,0.010169983,0.011619568,-0.022766113,-0.032806396,0.016647339,0.049713135,0.03414917,-0.07495117,0.027526855,0.07147217,0.037597656,-0.015792847,-0.0010890961,0.024108887,0.015419006,0.041870117,-0.035705566,0.015434265,-0.038238525,-0.03668213,-0.0011501312,-0.0234375,0.013130188,-0.054748535,0.010276794,0.074523926,0.015075684,0.01109314,0.047698975,0.0103302,0.03717041,0.030288696,0.0032520294,-0.025756836,-0.019424438,-0.05316162,0.04171753,-0.061798096,-0.014389038,0.039855957,0.003578186,0.018844604,0.032562256,0.04837036,-0.0023269653,-0.08227539,0.0446167,-0.042022705,0.016723633,0.023666382,0.03869629,-0.053985596,0.0029525757,0.052124023,0.024337769,-0.03427124,0.0262146,0.057861328,-0.020095825,0.10736084,-0.08642578,-0.07116699,-0.06298828,-0.039398193,0.0034751892,-0.0927124,-0.03677368,-0.026733398,0.044769287,-0.020767212,-0.02268982,-0.020584106,0.015777588,0.1083374,0.015617371,-0.02407837,-0.009567261,0.03842163,-0.0597229,-0.044799805,-0.013885498,0.045318604,-0.070373535,-0.06274414,-0.02027893,0.015327454,0.060546875,-0.0058174133,0.056396484,-0.04336548,-0.036346436,-0.044036865,-0.08642578,-0.017105103,-0.060516357,0.06915283,-0.027557373,0.068359375,0.010856628,-0.007587433,0.043945312,-0.03378296,0.031951904,-0.044433594,0.009857178,-0.013473511,0.00003117323,-0.004611969,-0.006034851,0.021774292,0.018936157,0.036132812,-0.03555298,-0.014785767,0.057800293,-0.036895752,-0.050323486,-0.020721436,-0.016647339,0.030593872,-0.0025672913,-0.05908203,0.06549072,-0.0019426346,0.009986877,0.022094727,0.053375244,0.09222412,0.025772095,0.033935547,-0.0058135986,0.031402588,-0.00065660477,-0.019119263,-0.05419922,0.029510498,0.022735596,-0.04534912,0.018798828,0.04397583,0.03881836,0.0067596436,0.035339355,0.03378296,0.07281494,-0.037597656,-0.035064697,-0.006439209,0.015007019,-0.07867432,-0.015945435,-0.10491943,0.019180298,-0.04498291,0.0010375977,-0.020828247,-0.045959473,0.0075912476,0.007785797,-0.056365967,0.012489319,-0.01574707,-0.014175415,0.07446289,-0.0058784485,-0.015113831,0.0082473755,0.010566711,-0.026016235,-0.007232666,-0.0340271,-0.04611206,-0.05783081,0.027282715,0.00010895729,0.07336426,0.06262207,0.00085544586,0.011199951,0.018447876,0.023620605,0.016677856,0.0335083,0.057250977,0.0013751984,0.0012817383,0.07122803,-0.030960083,-0.0058670044,-0.079589844,0.031188965,-0.023361206,0.054901123,0.03768921,-0.049957275,0.03604126,0.038604736,-0.022659302,0.085510254,-0.03479004,-0.031951904,0.015419006,-0.024887085,0.025009155,-0.06262207,0.01574707,0.015945435,0.031982422,0.09954834,-0.032714844,0.00178051,-0.046905518,-0.011581421,-0.039367676,-0.013847351,0.028335571,0.012199402,0.024276733,0.010948181,-0.03036499,0.0079956055,0.06933594,0.0062446594,-0.055786133,-0.031677246,0.030548096,0.029815674,0.0021915436,-0.032196045,0.07525635,0.022064209,-0.014625549,0.03717041,-0.02999878,-0.072631836,0.0010557175,-0.07318115,-0.05340576,0.050201416,0.016723633,-0.06274414,0.042419434,-0.004119873,0.014839172,-0.042175293,-0.041503906,0.040527344,-0.03591919,0.036987305,-0.0005173683,0.072265625,-0.021835327,-0.01689148,-0.005054474,-0.03012085,-0.002128601,-0.0046463013,-0.011474609,0.041534424,0.026062012,-0.03286743,-0.024169922,-0.005432129,0.014198303,0.024719238,-0.034698486,-0.13830566,-0.037750244,0.0053977966,-0.015106201,-0.012840271,-0.01802063,-0.004131317,0.0043945312,0.00042247772,0.050933838,-0.015274048,-0.0026550293,-0.012771606,0.024887085,-0.012039185,0.014595032,-0.06744385,0.039398193,0.035858154,-0.049957275,0.084106445,-0.014602661,0.009880066,0.04208374,-0.0637207,-0.07525635,-0.015319824,-0.052612305,-0.0031909943,0.038116455,-0.019561768,-0.054351807,0.020751953,-0.024337769,-0.0069236755,0.04043579,0.061676025,-0.052978516,-0.0061302185,0.022094727,0.03894043,-0.03213501,-0.026260376,-0.03050232,0.019104004,0.06640625,0.0076293945,0.040527344,-0.0357666,0.007484436,0.008728027,0.09289551,-0.02178955,-0.07244873,-0.009490967,0.014511108,-0.0084991455,0.0057640076,-0.02168274,0.008926392,-0.084350586,-0.008476257,0.009986877,0.09112549,0.05078125,-0.08001709,-0.046722412,-0.050933838,0.04296875,0.09881592,0.072631836,-0.09197998,0.0047340393,-0.025939941,0.048919678,0.017501831,-0.0037174225,0.046661377,0.13549805,-0.08215332,0.0181427,-0.015930176,0.022705078,0.004009247,-0.030014038,0.014945984,-0.04776001,-0.008041382,-0.054473877,-0.042633057,0.037994385,-0.012489319,0.051116943,0.06933594,0.01612854,0.07757568,0.047698975,-0.019378662,-0.019744873,0.0015363693,-0.021011353,0.0021209717,0.07318115,0.061920166,0.026687622,-0.024475098,-0.027694702,-0.07867432,-0.02645874,0.02218628,0.007041931,-0.018508911,0.05105591,-0.01612854,-0.0007419586,0.018554688,0.025772095,0.0012435913,-0.010955811,-0.024887085,-0.088012695,0.0077705383,0.056915283,-0.011207581,0.0073928833,-0.011749268,-0.0021152496,-0.008453369,-0.019515991,0.020965576,0.042022705,-0.08166504,-0.07745361,0.009712219,-0.03048706,0.026046753]' as distance -FROM embeddings -ORDER BY distance ASC -``` - -You should see the following output: - -| content | distance | -| ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------- | -| i like to eat tacos | 0.25041233062525814 | -| An embedding is a numerical representation of data in a high-dimensional space that captures semantic relationships and similarities between entities. | 0.4526167402866883 | - -Unsurprisingly, Postgres finds that the embedding for "i like to eat tacos" is the closest to the embedding for "burgers are tasty". - -## Resources - -- [pgvector GitHub repository](https://github.com/pgvector/pgvector) -- [Nomic API documentation](https://docs.nomic.ai/) diff --git a/next.config.js b/next.config.js index 2586973fee..50dbedf3bd 100644 --- a/next.config.js +++ b/next.config.js @@ -311,6 +311,11 @@ const defaultConfig = { destination: '/docs/reference/sdk', permanent: true, }, + { + source: '/guides/vector-search', + destination: '/docs/ai/lakebase-search-get-started', + permanent: true, + }, { source: '/blog/neon-serverless-driver-on-jsr', destination: '/docs/serverless/serverless-driver', From 305c918476c59cd2a9f0f5322ef60a4299d28939 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 22:34:05 +0000 Subject: [PATCH 15/20] docs: point the pgvector blog link at the pgvector docs The surrounding text is about pgvector, so link the pgvector extension page. --- content/blog/posts/ten-most-popular-postgres-extensions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/posts/ten-most-popular-postgres-extensions.md b/content/blog/posts/ten-most-popular-postgres-extensions.md index 611736ae7c..43b2f52ae2 100644 --- a/content/blog/posts/ten-most-popular-postgres-extensions.md +++ b/content/blog/posts/ten-most-popular-postgres-extensions.md @@ -121,7 +121,7 @@ ORDER BY distance LIMIT 5; ``` -[Check out this guide](https://neon.com/docs/ai/lakebase-search-get-started) to get started with vector search. +Check out the [pgvector docs](https://neon.com/docs/extensions/pgvector) to get started with pgvector. ### pgcrypto From 0f89446e61428f89cda84fe5199385d9e9ac0773 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Tue, 17 Nov 2026 23:36:26 +0000 Subject: [PATCH 16/20] docs: add VectorDBBench results to the Lakebase Search benchmarks Report the Databricks VectorDBBench figures with needs-verification markers on the unreproduced numbers, keep the cleared 4x cost claim, and lead the page with the agent-native value instead of the three search types. --- content/docs/ai/lakebase-search.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index 648b24723f..0e4bf36d17 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,10 +13,10 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T21:48:30.966Z' +updatedOn: '2026-11-17T23:36:26.891Z' --- -Lakebase Search gives your agents and apps semantic, keyword, and hybrid search over your own data, running as plain SQL inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. +Lakebase Search is search built for agents: retrieval over your own data as plain SQL, inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -69,7 +69,16 @@ A dedicated vector database trades the memory problem for a sync problem: a seco ## Benchmarks -On VectorDBBench at 100-million-vector scale, Lakebase Search is the fastest and most cost-efficient system measured, running roughly 4x more cost-effectively than a standard cloud `pgvector` setup. See the [Databricks benchmark](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres) for the full results. +Databricks benchmarked Lakebase Search on VectorDBBench at 100-million-vector scale. These results were measured by Databricks on Databricks Lakebase, so treat them as a reference point, not numbers a Neon project is guaranteed to hit. + +On VectorDBBench, as reported by Databricks: + +- Roughly 4x more cost-effective than a standard cloud `pgvector` setup. +- About 2x the throughput of the next-best system measured [needs verification]. +- 97% recall at 71 ms P99 latency [needs verification]. +- 100 million vectors served on a single compute unit (CU) [needs verification]. + +See the [Databricks benchmark](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres) for the full methodology and results. ## Which search to use: vector, keyword, or hybrid From 5b74155a4f3d245d2b94f9b366696dd8f35b1a4b Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Wed, 18 Nov 2026 14:21:48 +0000 Subject: [PATCH 17/20] docs: new architecture and contrast diagrams for Lakebase Search Replace the architecture diagram with a themed SVG showing stateless compute, separated storage, and both engines, and add a contrast diagram for the index-in-RAM-versus-storage point. Both SVGs theme via the site dark class. Rewrite the surrounding prose to carry scale-to-zero, cold-start, and the storage-not-RAM payoff instead of restating the pictures. --- content/docs/ai/lakebase-search.md | 16 +-- .../docs/ai/lakebase-search-architecture.svg | 115 +++++++++++------- public/docs/ai/lakebase-search-contrast.svg | 78 ++++++++++++ 3 files changed, 155 insertions(+), 54 deletions(-) create mode 100644 public/docs/ai/lakebase-search-contrast.svg diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index 0e4bf36d17..f78a676144 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-17T23:36:26.891Z' +updatedOn: '2026-11-18T14:21:49.129Z' --- Lakebase Search is search built for agents: retrieval over your own data as plain SQL, inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -22,11 +22,11 @@ Lakebase Search is search built for agents: retrieval over your own data as plai ## How Lakebase Search works -Lakebase Search separates storage from compute, the same architecture Neon uses for Postgres. Your indexes live in storage, not in compute memory, so compute stays stateless and scales to zero when the database is idle. +Lakebase Search adds two engines to Postgres: `lakebase_vector` for vector search and `lakebase_text` for keyword search, plus `lakebase_tokenizer` for tokenization. Both build ordinary Postgres indexes, so a single SQL query can run vector search, keyword search, or both, and come back as one ranked result. -It adds two search engines to Postgres: `lakebase_vector` for vector search and `lakebase_text` for keyword search, plus `lakebase_tokenizer` for tokenization. Both build ordinary Postgres indexes, so you query them, and combine them, with plain SQL. + - +The engines run on stateless compute, and their indexes persist in separate storage, the same architecture Neon uses for Postgres. Compute scales to zero when the database is idle. Because the indexes stay in storage, waking the database needs no rebuild or re-index: the first query after a wake pays object-storage latency while the cache warms, and you can call `lakebase_ann_prewarm` ahead of time to avoid it. Search by meaning is a single SQL query: order rows by the distance between their embedding and your query vector, then take the top matches. @@ -45,15 +45,13 @@ Keyword search and hybrid search run the same way, as SQL. Hybrid search combine `lakebase_vector` builds the `lakebase_ann` index. It sorts vectors into groups and only searches the groups closest to your query, so it reads a small part of the index instead of scanning all of it (this grouping is called IVF, short for Inverted File). It also shrinks each vector into a smaller form, which keeps the index small and quick to build (this compression is called RaBitQ). -Because the index lives in storage, a cold start needs no rebuild or re-index. The first query pays object-storage latency while the cache warms; call `lakebase_ann_prewarm` ahead of time to skip that. - ### How the text index ranks `lakebase_text` builds the `lakebase_bm25` index. It ranks results with BM25 and finds the best matches without scoring every matching row, so text search stays fast as your data grows (the technique for this is called Block-Max WAND). A standard Postgres full-text index (GIN) scores all the matching rows first and then picks the top ones; `lakebase_bm25` goes straight to the best matches. ## Scaling search past pgvector -Most teams start vector search in Postgres with `pgvector`, and the limits show up as the data grows. Its usual index, HNSW (Hierarchical Navigable Small World), keeps the whole vector index in RAM, so its memory grows with the number of vectors, not with how often you query. +Most teams start vector search in Postgres with `pgvector`, and the limits show up as the data grows. On traditional Postgres, a search index has to stay resident in memory to answer quickly, so `pgvector`'s usual HNSW (Hierarchical Navigable Small World) index grows its memory with the number of vectors, not with how often you query. Past roughly tens of millions of vectors, you run into several limits: @@ -61,7 +59,9 @@ Past roughly tens of millions of vectors, you run into several limits: - index builds slow enough to disrupt the database - a recall-versus-latency tradeoff on every tuning pass (recall means how many of the true matches you actually get back) -The `lakebase_ann` index lives in storage instead of RAM, so the index size no longer sets the instance size. + + +Lakebase Search keeps the `lakebase_ann` index in storage and reads it from stateless compute on demand, so the index size no longer sets the instance size. You no longer need an always-on instance sized to hold the whole index in memory, and a single index scales to 100 million vectors and beyond. A dedicated vector database trades the memory problem for a sync problem: a second system to run and a pipeline to keep in step, so your search lags your writes. Lakebase Search keeps search in the same database as your data, so a row is searchable the moment it's written. There's no separate search system to keep in sync, and no lag between a write and when its row is searchable. diff --git a/public/docs/ai/lakebase-search-architecture.svg b/public/docs/ai/lakebase-search-architecture.svg index 8b42d80be7..19b8e1e1de 100644 --- a/public/docs/ai/lakebase-search-architecture.svg +++ b/public/docs/ai/lakebase-search-architecture.svg @@ -1,53 +1,76 @@ - - - + + - -your app - - -one SQL query - - -ranked results - -Postgres -search on your tables - -vector search -by meaning - -text search -by keywords - - - -one ranked result - + + + + + agent / app + + + + + + one SQL query + + + ranked results + + + + + Stateless compute + + lakebase_vector + lakebase_ann index + IVF grouping + RaBitQ compression + + lakebase_text + lakebase_bm25 index + BM25 ranking + + + + + storage / compute separation + + + + + + + Storage + + lakebase_ann index + + lakebase_bm25 index + + + \ No newline at end of file diff --git a/public/docs/ai/lakebase-search-contrast.svg b/public/docs/ai/lakebase-search-contrast.svg new file mode 100644 index 0000000000..a8e3c61e3a --- /dev/null +++ b/public/docs/ai/lakebase-search-contrast.svg @@ -0,0 +1,78 @@ + + + + + + + traditional Postgres + + compute + always on + + RAM + + + + + + + search index + + storage + tables + + + + + Lakebase Search + + compute + stateless + + RAM + + + reads + + storage + tables + + + + + + + search index + + + \ No newline at end of file From f61d0446e6c2917eff0b1d545ffd981bd7e90466 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Wed, 18 Nov 2026 14:23:45 +0000 Subject: [PATCH 18/20] docs: use updated Lakebase Search contrast diagram Draw the search index as a single labeled block instead of a bar stack that was misread as table rows. Metadata-stripped and dark-mode themed as before. --- public/docs/ai/lakebase-search-contrast.svg | 24 ++++++--------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/public/docs/ai/lakebase-search-contrast.svg b/public/docs/ai/lakebase-search-contrast.svg index a8e3c61e3a..5fa16017f3 100644 --- a/public/docs/ai/lakebase-search-contrast.svg +++ b/public/docs/ai/lakebase-search-contrast.svg @@ -40,13 +40,8 @@ always on RAM - - - - - - - search index + + search index storage tables @@ -60,19 +55,14 @@ stateless RAM - - - reads + + + reads storage tables - - - - - - - search index + + search index \ No newline at end of file From 8dc64365da831b24d9555e529d92eda0db8a8b99 Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Wed, 18 Nov 2026 15:03:02 +0000 Subject: [PATCH 19/20] docs: copy edits across the Lakebase Search hub Tighten the cold-start, vector-search, keyword/hybrid, pgvector, and separate- vector-database wording, trim the acronym hand-holding, and simplify the benchmark list lead-in. --- content/docs/ai/lakebase-search.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index f78a676144..3b6c053227 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-18T14:21:49.129Z' +updatedOn: '2026-11-18T15:03:03.596Z' --- Lakebase Search is search built for agents: retrieval over your own data as plain SQL, inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -26,32 +26,31 @@ Lakebase Search adds two engines to Postgres: `lakebase_vector` for vector searc -The engines run on stateless compute, and their indexes persist in separate storage, the same architecture Neon uses for Postgres. Compute scales to zero when the database is idle. Because the indexes stay in storage, waking the database needs no rebuild or re-index: the first query after a wake pays object-storage latency while the cache warms, and you can call `lakebase_ann_prewarm` ahead of time to avoid it. +The engines run on stateless compute, and their indexes persist in separate storage, the same architecture Neon uses for Postgres. Compute scales to zero when the database is idle. Because the indexes stay in storage, waking the database doesn't rebuild or re-index anything. The first query after a wake is slower while the data loads into memory, and you can run `lakebase_ann_prewarm` ahead of time to avoid it. -Search by meaning is a single SQL query: order rows by the distance between their embedding and your query vector, then take the top matches. +Vector search is a single SQL query: order rows by the distance between their embedding and your query vector, then return the top matches. ```sql --- Semantic search is one SQL query: order rows by the distance --- between their embedding and the query vector. No separate vector store. +-- Vector search SELECT id, title FROM documents ORDER BY embedding <=> $1::vector LIMIT 5; ``` -Keyword search and hybrid search run the same way, as SQL. Hybrid search combines the vector and keyword results into one ranked list (using Reciprocal Rank Fusion, or RRF), and can optionally reorder them with Jev, a reranking model on Neon AI Gateway. See the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) for the full runnable hybrid query. +Keyword search and hybrid search also run as SQL. Hybrid search combines the vector and keyword results into one ranked list (using Reciprocal Rank Fusion, or RRF). You can also reorder the results with Jev, a reranking model on the Neon AI Gateway. See the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) for the full runnable hybrid query. ### How the vector index scales -`lakebase_vector` builds the `lakebase_ann` index. It sorts vectors into groups and only searches the groups closest to your query, so it reads a small part of the index instead of scanning all of it (this grouping is called IVF, short for Inverted File). It also shrinks each vector into a smaller form, which keeps the index small and quick to build (this compression is called RaBitQ). +`lakebase_vector` builds the `lakebase_ann` index. It sorts vectors into groups and only searches the groups closest to your query, so it reads a small part of the index instead of scanning all of it (IVF, for Inverted File). It also shrinks each vector into a smaller form, which keeps the index small and quick to build (RaBitQ). ### How the text index ranks -`lakebase_text` builds the `lakebase_bm25` index. It ranks results with BM25 and finds the best matches without scoring every matching row, so text search stays fast as your data grows (the technique for this is called Block-Max WAND). A standard Postgres full-text index (GIN) scores all the matching rows first and then picks the top ones; `lakebase_bm25` goes straight to the best matches. +`lakebase_text` builds the `lakebase_bm25` index. It ranks results with BM25 and finds the best matches without scoring every matching row, so text search stays fast as your data grows (Block-Max WAND). A standard Postgres full-text index (GIN) scores all the matching rows first and then picks the top ones; `lakebase_bm25` goes straight to the best matches. ## Scaling search past pgvector -Most teams start vector search in Postgres with `pgvector`, and the limits show up as the data grows. On traditional Postgres, a search index has to stay resident in memory to answer quickly, so `pgvector`'s usual HNSW (Hierarchical Navigable Small World) index grows its memory with the number of vectors, not with how often you query. +While `pgvector` is the standard extension for vector search in Postgres, its limits show up as the data grows. On traditional Postgres, a search index has to stay resident in memory to answer quickly, so `pgvector`'s usual HNSW (Hierarchical Navigable Small World) index grows its memory with the number of vectors, not with how often you query. Past roughly tens of millions of vectors, you run into several limits: @@ -63,7 +62,7 @@ Past roughly tens of millions of vectors, you run into several limits: Lakebase Search keeps the `lakebase_ann` index in storage and reads it from stateless compute on demand, so the index size no longer sets the instance size. You no longer need an always-on instance sized to hold the whole index in memory, and a single index scales to 100 million vectors and beyond. -A dedicated vector database trades the memory problem for a sync problem: a second system to run and a pipeline to keep in step, so your search lags your writes. Lakebase Search keeps search in the same database as your data, so a row is searchable the moment it's written. There's no separate search system to keep in sync, and no lag between a write and when its row is searchable. +A separate vector database avoids that memory problem, but then you have to keep it in sync with your Postgres data: a second system to run and a pipeline to keep in step, so your search lags your writes. Lakebase Search keeps search in the same database as your data, so a row is searchable the moment it's written. There's no separate search system to keep in sync, and no lag between a write and when its row is searchable. `lakebase_vector` is a strict superset of `pgvector`: the same types, operators, and SQL. You keep your existing queries and add the `lakebase_ann` index. @@ -71,7 +70,7 @@ A dedicated vector database trades the memory problem for a sync problem: a seco Databricks benchmarked Lakebase Search on VectorDBBench at 100-million-vector scale. These results were measured by Databricks on Databricks Lakebase, so treat them as a reference point, not numbers a Neon project is guaranteed to hit. -On VectorDBBench, as reported by Databricks: +On VectorDBBench: - Roughly 4x more cost-effective than a standard cloud `pgvector` setup. - About 2x the throughput of the next-best system measured [needs verification]. From d82e31724489f3d14e85657c2a9b8df14bd239ae Mon Sep 17 00:00:00 2001 From: Barry Grenon Date: Wed, 18 Nov 2026 15:34:52 +0000 Subject: [PATCH 20/20] docs: condense the Lakebase Search benchmarks section Keep the cleared 4x cost comparison and the Databricks benchmark link, and drop the unverified throughput, recall, latency, and single-CU numbers. --- content/docs/ai/lakebase-search.md | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/content/docs/ai/lakebase-search.md b/content/docs/ai/lakebase-search.md index 3b6c053227..46bd9bb6ef 100644 --- a/content/docs/ai/lakebase-search.md +++ b/content/docs/ai/lakebase-search.md @@ -13,7 +13,7 @@ redirectFrom: - /docs/extensions/migrate-pg-search-to-lakebase-text - /guides/pg-search - /guides/pg-search-vs-tsvector -updatedOn: '2026-11-18T15:03:03.596Z' +updatedOn: '2026-11-18T15:34:53.666Z' --- Lakebase Search is search built for agents: retrieval over your own data as plain SQL, inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon. @@ -68,16 +68,7 @@ A separate vector database avoids that memory problem, but then you have to keep ## Benchmarks -Databricks benchmarked Lakebase Search on VectorDBBench at 100-million-vector scale. These results were measured by Databricks on Databricks Lakebase, so treat them as a reference point, not numbers a Neon project is guaranteed to hit. - -On VectorDBBench: - -- Roughly 4x more cost-effective than a standard cloud `pgvector` setup. -- About 2x the throughput of the next-best system measured [needs verification]. -- 97% recall at 71 ms P99 latency [needs verification]. -- 100 million vectors served on a single compute unit (CU) [needs verification]. - -See the [Databricks benchmark](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres) for the full methodology and results. +Lakebase Search is built to stay fast and cost-efficient at large scale, around 4x more cost-effective than a standard cloud `pgvector` setup. Databricks benchmarked it with VectorDBBench on a 100-million-vector dataset; see the [Databricks benchmark](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres) for the results. They were measured by Databricks on Databricks Lakebase, so treat them as a reference point. ## Which search to use: vector, keyword, or hybrid