Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
d51840d
docs: rewrite Lakebase Search as an architecture-first hub page
bgrenon Nov 17, 2026
cd4755a
docs: tighten Lakebase Search voice and integrate search types
bgrenon Nov 17, 2026
b676688
docs: cut redundant Lakebase Search section and trim editorializing
bgrenon Nov 17, 2026
b4bada4
docs: swap Lakebase Search diagram for themed SVG
bgrenon Nov 17, 2026
7996ff7
docs: restructure Lakebase Search hub for architecture-first flow
bgrenon Nov 17, 2026
3422d6b
docs: plainer wording for pgvector limits on Lakebase Search hub
bgrenon Nov 17, 2026
34a71a5
docs: add a hybrid-search SQL example to the Lakebase Search hub
bgrenon Nov 17, 2026
2e17f1d
docs: use a short vector-search example on the Lakebase Search hub
bgrenon Nov 17, 2026
4d12de5
docs: annotate vector example and add cleared benchmark claims
bgrenon Nov 17, 2026
f041936
docs: explain IVF, RaBitQ, and Block-Max WAND on the Lakebase Search hub
bgrenon Nov 17, 2026
55f1d34
docs: plainer search glosses and a which-search-to-use section
bgrenon Nov 17, 2026
d66c00c
docs: group Lakebase Search docs under one sidebar section
bgrenon Nov 17, 2026
b18ad63
docs: tighten the Lakebase Search benchmark line
bgrenon Nov 17, 2026
d8d5faf
docs: retire the vector-search guide and redirect to Lakebase Search
bgrenon Nov 17, 2026
305c918
docs: point the pgvector blog link at the pgvector docs
bgrenon Nov 17, 2026
0f89446
docs: add VectorDBBench results to the Lakebase Search benchmarks
bgrenon Nov 17, 2026
5b74155
docs: new architecture and contrast diagrams for Lakebase Search
bgrenon Nov 18, 2026
f61d044
docs: use updated Lakebase Search contrast diagram
bgrenon Nov 18, 2026
8dc6436
docs: copy edits across the Lakebase Search hub
bgrenon Nov 18, 2026
d82e317
docs: condense the Lakebase Search benchmarks section
bgrenon Nov 18, 2026
30be585
Merge origin/main into docs/lakebase-search-hub
bgrenon Nov 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ ORDER BY distance
LIMIT 5;
```

[Check out this guide](https://neon.com/guides/vector-search) to get started with pgvector.
Check out the [pgvector docs](https://neon.com/docs/extensions/pgvector) to get started with pgvector.

### pgcrypto

Expand Down
87 changes: 57 additions & 30 deletions content/docs/ai/lakebase-search.md
Original file line number Diff line number Diff line change
@@ -1,57 +1,84 @@
---
title: Lakebase Search
subtitle: Scalable vector and full-text search for Postgres
subtitle: A search system built for agents, running inside your Postgres database
summary: >-
Lakebase Search adds vector, keyword, and hybrid search to Neon through the
lakebase_vector and lakebase_text Postgres extensions, with optional
configurable tokenization from lakebase_tokenizer. Use this page to understand
the search types, how the extensions work, the scale-to-zero architecture
advantages, and where to get started.
Lakebase Search adds vector, keyword, and hybrid search to Lakebase Postgres
on Neon. It scales vector search past pgvector's HNSW memory limits by keeping
indexes in storage, so a single lakebase_ann index handles 100 million vectors
and beyond with serverless scale-to-zero. lakebase_text adds BM25 keyword
search, and hybrid search runs as plain SQL.
enableTableOfContents: true
redirectFrom:
- /docs/extensions/pg_search
- /docs/extensions/migrate-pg-search-to-lakebase-text
- /guides/pg-search
- /guides/pg-search-vs-tsvector
updatedOn: '2026-10-23T11:22:59.906Z'
updatedOn: '2026-11-18T15:34:53.666Z'
---

<Callout title="About Lakebase">
Lakebase Search is developed by Databricks. These extensions are part of the shared technology foundation between Neon and the Databricks Lakebase platform.
</Callout>

Lakebase Search adds vector, keyword, and hybrid search to your Neon project. Install `lakebase_vector` for vector search, `lakebase_text` for BM25 keyword search, and `lakebase_tokenizer` for configurable tokenization.
Lakebase Search is search built for agents: retrieval over your own data as plain SQL, inside your [Lakebase Postgres](/docs/postgres/overview) database on Neon.

<AgentPrompt title="Set up with your agent" src="/prompts/lakebase-search-landing.md" buttonText="Copy prompt" />

## Vector, keyword, and hybrid search
## How Lakebase Search works

Lakebase Search adds two engines to Postgres: `lakebase_vector` for vector search and `lakebase_text` for keyword search, plus `lakebase_tokenizer` for tokenization. Both build ordinary Postgres indexes, so a single SQL query can run vector search, keyword search, or both, and come back as one ranked result.

<InlineSvg src="/docs/ai/lakebase-search-architecture.svg" title="An agent sends one SQL query to stateless compute that runs the lakebase_vector and lakebase_text engines. Their lakebase_ann and lakebase_bm25 indexes persist in separate storage, and ranked results return to the agent." />

The engines run on stateless compute, and their indexes persist in separate storage, the same architecture Neon uses for Postgres. Compute scales to zero when the database is idle. Because the indexes stay in storage, waking the database doesn't rebuild or re-index anything. The first query after a wake is slower while the data loads into memory, and you can run `lakebase_ann_prewarm` ahead of time to avoid it.

Vector search is a single SQL query: order rows by the distance between their embedding and your query vector, then return the top matches.

```sql
-- Vector search
SELECT id, title
FROM documents
ORDER BY embedding <=> $1::vector
LIMIT 5;
```

Keyword search and hybrid search also run as SQL. Hybrid search combines the vector and keyword results into one ranked list (using Reciprocal Rank Fusion, or RRF). You can also reorder the results with Jev, a reranking model on the Neon AI Gateway. See the [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) for the full runnable hybrid query.

### How the vector index scales

`lakebase_vector` builds the `lakebase_ann` index. It sorts vectors into groups and only searches the groups closest to your query, so it reads a small part of the index instead of scanning all of it (IVF, for Inverted File). It also shrinks each vector into a smaller form, which keeps the index small and quick to build (RaBitQ).

### How the text index ranks

`lakebase_text` builds the `lakebase_bm25` index. It ranks results with BM25 and finds the best matches without scoring every matching row, so text search stays fast as your data grows (Block-Max WAND). A standard Postgres full-text index (GIN) scores all the matching rows first and then picks the top ones; `lakebase_bm25` goes straight to the best matches.

## Scaling search past pgvector

While `pgvector` is the standard extension for vector search in Postgres, its limits show up as the data grows. On traditional Postgres, a search index has to stay resident in memory to answer quickly, so `pgvector`'s usual HNSW (Hierarchical Navigable Small World) index grows its memory with the number of vectors, not with how often you query.

Past roughly tens of millions of vectors, you run into several limits:

- a large, always-on instance just to hold the index in memory
- index builds slow enough to disrupt the database
- a recall-versus-latency tradeoff on every tuning pass (recall means how many of the true matches you actually get back)

Lakebase Search gives you two complementary ways to search. Use either on its own, or combine them:
<InlineSvg src="/docs/ai/lakebase-search-contrast.svg" title="In traditional Postgres the search index sits in RAM on always-on compute. In Lakebase Search the index lives in storage, and stateless compute reads it on demand." />

- **Vector (semantic) search** finds rows whose meaning is closest to your query, even when they share no words. You search with an embedding (a numeric vector from a model), and the `lakebase_ann` index returns the nearest vectors by distance. Use it for natural-language questions, recommendations, and retrieval-augmented generation (RAG).
- **Keyword (full-text) search** ranks rows by how well they match the exact terms in your query, using BM25 relevance scoring from the `lakebase_bm25` index. Use it for names, codes, and exact-term lookups where wording matters.
- **Hybrid search** combines vector and keyword results with fusion or reranking, so you get semantic and exact-term matches together. Use it when queries mix intent with specific terms, which covers most real-world search. The [Get started guide](/docs/ai/lakebase-search-get-started#combine-results-with-hybrid-search) shows how to combine results with Jev through Neon AI Gateway or RRF.
Lakebase Search keeps the `lakebase_ann` index in storage and reads it from stateless compute on demand, so the index size no longer sets the instance size. You no longer need an always-on instance sized to hold the whole index in memory, and a single index scales to 100 million vectors and beyond.

![Keyword search matches only documents that contain the typed words and misses synonyms like "quick automobile" for "fast sports car." Vector search places the query and documents in the same embedding space and returns the nearest neighbors by meaning, including those synonyms.](/docs/ai/lakebase-search-keyword-vs-vector.png)
A separate vector database avoids that memory problem, but then you have to keep it in sync with your Postgres data: a second system to run and a pipeline to keep in step, so your search lags your writes. Lakebase Search keeps search in the same database as your data, so a row is searchable the moment it's written. There's no separate search system to keep in sync, and no lag between a write and when its row is searchable.

## How it works
`lakebase_vector` is a strict superset of `pgvector`: the same types, operators, and SQL. You keep your existing queries and add the `lakebase_ann` index.

Lakebase Search is built on three Postgres extensions:
## Benchmarks

- **[`lakebase_vector`](/docs/extensions/lakebase-vector)**: adds the `lakebase_ann` index type for vector similarity search. No migration from `pgvector` required. The same `vector` types, distance operators, and query syntax work unchanged. Scales to over 1 billion vectors on a single index.
- **[`lakebase_text`](/docs/extensions/lakebase-text)**: adds the `lakebase_bm25` index type for BM25 keyword search. No migration from Postgres full-text search required. Standard `tsvector` types and query operators work unchanged. Adds BM25 ranking and top-K pushdown that native GIN lacks.
- **[`lakebase_tokenizer`](/docs/extensions/lakebase-tokenizer)**: adds configurable whole-word tokenization and Chinese, Japanese, and Korean (CJK) morphological analysis through Postgres's text-search dictionary interface. It produces standard `tsvector` values that work with GIN and `lakebase_bm25` indexes.
Lakebase Search is built to stay fast and cost-efficient at large scale, around 4x more cost-effective than a standard cloud `pgvector` setup. Databricks benchmarked it with VectorDBBench on a 100-million-vector dataset; see the [Databricks benchmark](https://www.databricks.com/blog/lakebase-search-state-art-full-text-and-vector-search-postgres) for the results. They were measured by Databricks on Databricks Lakebase, so treat them as a reference point.

## How `lakebase_ann` scales
## Which search to use: vector, keyword, or hybrid

`lakebase_ann` is designed for the separated compute/storage architecture that powers Neon. Because the index lives in storage rather than in compute memory, it works naturally with Neon's scale-to-zero model: your vector index is available immediately after a cold start, with no warmup required. You only pay for compute when your database is actively serving requests. This holds for production workloads and for development or staging environments where the database sits idle most of the time.
Pick the search that fits the query:

It uses IVF (Inverted File) partitioning to divide the vector space into lists and searches only the relevant ones at query time, using sequential I/O that suits storage-backed systems. RaBitQ quantization compresses vectors 4–8x, keeping index size and build time down. The result:
- **Vector (semantic) search** when you care about meaning: natural-language questions, recommendations, and retrieval-augmented generation (RAG).
- **Keyword (full-text) search** when the exact words matter: names, codes, and other precise terms.
- **Hybrid search** when you want both. It runs the vector and keyword searches and merges them into one ranked list.

- **Scale to 1 billion+ vectors**: a single `lakebase_ann` index grows with your data without resharding or rebuilding
- **Faster index builds**: compression reduces index size significantly, making builds 50–100x faster than HNSW
- **Branching without index rebuilds**: Neon branches copy-on-write; your search indexes are available instantly on every branch without re-indexing
- **Scale-to-zero compatible**: indexes survive cold starts without warmup; you don't pay for idle compute
For Chinese, Japanese, and Korean (CJK) text or other custom tokenization, add `lakebase_tokenizer`.

## Get started

Expand Down
4 changes: 2 additions & 2 deletions content/docs/extensions/pgvector.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ summary: >-
enableTableOfContents: true
redirectFrom:
- /docs/extensions/pg_embedding
updatedOn: '2026-11-13T20:59:05.000Z'
updatedOn: '2026-11-17T22:25:43.741Z'
---

<LakebaseVectorTip />
Expand Down Expand Up @@ -684,7 +684,7 @@ LINE 1: SELECT '[4e38,1]'::vector;
## Resources

- `pgvector` source code: [https://github.com/pgvector/pgvector](https://github.com/pgvector/pgvector)
- [Vector search in Postgres](/guides/vector-search)
- [Get started with Lakebase Search](/docs/ai/lakebase-search-get-started)
- [Building semantic search with AI embeddings, Neon, and pgvector](/guides/ai-embeddings-postgres-search)

<NeedHelp/>
5 changes: 3 additions & 2 deletions content/docs/guides/grafana-cloud.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
---
title: Grafana Cloud integration
subtitle: Send metrics and logs from Lakebase Postgres on Neon to Grafana Cloud
Expand Down Expand Up @@ -566,7 +566,7 @@
},
{
"name": "DS_LOKI",
"label": "Loki Datasource",
"label": "Loki Datasource",
"type": "datasource",
"query": "loki",
"hide": 0,
Expand Down Expand Up @@ -660,7 +660,7 @@
{
"name": "Low Cache Hit Rate",
"datasource": {
"type": "prometheus",
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "neon_lfc_hits / (neon_lfc_hits + neon_lfc_misses) * 100 < 90",
Expand All @@ -686,6 +686,7 @@
]
}
```

</details>

## Available metrics
Expand Down
3 changes: 1 addition & 2 deletions content/docs/guides/opentelemetry.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
---
title: OpenTelemetry
subtitle: Send Neon metrics and Postgres logs to any OTEL-compatible observability
Expand Down Expand Up @@ -118,8 +118,7 @@
1. Click on your user menu in the bottom left corner.
2. Select **API Keys** from the menu.

![New Relic profile menu showing API Keys option](/docs/guides/new_relic_api_keys.png)
3. Click **Create a key** and choose **Ingest - License**. Copy the key immediately (you can't view it again later).
![New Relic profile menu showing API Keys option](/docs/guides/new_relic_api_keys.png) 3. Click **Create a key** and choose **Ingest - License**. Copy the key immediately (you can't view it again later).

![New Relic API Keys page showing license key types](/docs/guides/new_relic_copy_key.png)

Expand Down
20 changes: 10 additions & 10 deletions content/docs/guides/schema-diff.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
---
title: Schema diff
subtitle: Learn how to use Neon's Schema Diff tool to compare branches of your database
Expand Down Expand Up @@ -118,17 +118,17 @@

The `compare_schema` endpoint supports the following parameters:

| Parameter | Description | Required | Example |
| ------------------ | ----------------------------------------------------------------------------- | -------- | -------------------------- |
| `<project_id>` | The ID of your Neon project. | Yes | `wispy-butterfly-25042691` |
| `<branch_id>` | The ID of the target branch to compare (the branch with the modified schema). | Yes | `br-rough-boat-a54bs9yb` |
| Parameter | Description | Required | Example |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------ | -------- | -------------------------- |
| `<project_id>` | The ID of your Neon project. | Yes | `wispy-butterfly-25042691` |
| `<branch_id>` | The ID of the target branch to compare (the branch with the modified schema). | Yes | `br-rough-boat-a54bs9yb` |
| `<base_branch_id>` | The ID of the base branch for comparison. If you omit it, the endpoint compares against the parent of the target branch. | No | `br-royal-star-a54kykl2` |
| `<db_name>` | The name of the database in the target branch. | Yes | `neondb` |
| `lsn` | The LSN on the target branch for which the schema is retrieved. | No | `0/1EC5378` |
| `timestamp` | The point in time on the target branch for which the schema is retrieved. | No | `2022-11-30T20:09:48Z` |
| `base_lsn` | The LSN for the base branch schema. | No | `0/2FC6321` |
| `base_timestamp` | The point in time for the base branch schema. | No | `2022-11-30T20:09:48Z` |
| `Authorization` | Bearer token for API access (your [Neon API key](/docs/manage/api-keys)) | Yes | `$NEON_API_KEY` |
| `<db_name>` | The name of the database in the target branch. | Yes | `neondb` |
| `lsn` | The LSN on the target branch for which the schema is retrieved. | No | `0/1EC5378` |
| `timestamp` | The point in time on the target branch for which the schema is retrieved. | No | `2022-11-30T20:09:48Z` |
| `base_lsn` | The LSN for the base branch schema. | No | `0/2FC6321` |
| `base_timestamp` | The point in time for the base branch schema. | No | `2022-11-30T20:09:48Z` |
| `Authorization` | Bearer token for API access (your [Neon API key](/docs/manage/api-keys)) | Yes | `$NEON_API_KEY` |

<Admonition type="note" title="notes">
- The optional `jq -r '.diff'` command appended to the example above extracts the diff field from the JSON response and outputs it as plain text to make it easier to read. This command is not necessary when using the endpoint programmatically.
Expand Down
10 changes: 5 additions & 5 deletions content/docs/guides/vercel-connection-methods.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
---
title: Connecting to Neon from Vercel
subtitle: Learn how Vercel Fluid compute optimizes database connections and why standard
Expand Down Expand Up @@ -81,11 +81,11 @@

This table breaks down the trade-offs, which are all about setup cost versus query speed.

| Connection Method | Protocol | Setup Cost (Roundtrips) | Best For... |
| :----------------- | :------------ | :---------------------- | :---------------------------------------------------------------------------------------------------- |
| **Postgres (TCP)** | `postgres://` | High (~8) | **Fluid compute / Long-running servers.** (Render, Railway). Once established, queries skip connection setup. |
| **HTTP** | `https://` | Lowest (~3) | **Classic Serverless.** Fastest for a _single query_ where you can't pool connections. |
| **WebSocket** | `wss://` | Low (~4) | **Classic Serverless.** A good alternative to HTTP, especially in environments that don't support it. |
| Connection Method | Protocol | Setup Cost (Roundtrips) | Best For... |
| :----------------- | :------------ | :---------------------- | :------------------------------------------------------------------------------------------------------------ |
| **Postgres (TCP)** | `postgres://` | High (~8) | **Fluid compute / Long-running servers.** (Render, Railway). Once established, queries skip connection setup. |
| **HTTP** | `https://` | Lowest (~3) | **Classic Serverless.** Fastest for a _single query_ where you can't pool connections. |
| **WebSocket** | `wss://` | Low (~4) | **Classic Serverless.** A good alternative to HTTP, especially in environments that don't support it. |

_Note: Roundtrip counts are estimates and vary based on authentication and configuration._

Expand Down
1 change: 0 additions & 1 deletion content/docs/import/import-data-assistant.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
---
title: Import Data Assistant
subtitle: Import a database under 10 GB to Neon using our automated import tool
Expand Down Expand Up @@ -73,7 +73,6 @@

Click the **Create new Neon project** button, specify your project settings including the **Postgres version** and **region**, and click **Create**. Choose the same Postgres version as your source database or a newer one. You can't import into an older version.


## Start import process

Back in the assistant, click the **Start import process** button.
Expand Down
Loading
Loading