Databricks' Lakebase Search moves Postgres vector indexes from RAM into object storage
Databricks made Lakebase Search generally available, pairing BM25 with a Postgres vector index it says runs 4 times cheaper than pgvector at 100M vectors. The vector index lives in object storage behind a cache, so large agent corpora no longer need RAM sized to the whole index.
The Engineer · Build desk
What happened
- Lakebase Search ships as two Postgres extensions, lakebase_vector for approximate nearest-neighbor search and lakebase_text for BM25 full-text search, on AWS and Azure.
- Existing pgvector users switch by adding a lakebase_ann index while keeping the same vector types, distance operators and query syntax, with no data migration.
- lakebase_text adds a lakebase_bm25 index that keeps standard Postgres tsvector types and operators while adding corpus-wide BM25 ranking and top-K pushdown.
- Databricks says customer Conexiom runs BM25 hybrid search on over 100 million rows using half the compute of its previous pgvector setup.
- Neon pairs the extensions with TypeScript Functions for ingestion, an OpenAI-compatible embeddings endpoint in AI Gateway, and a trigger that indexes files uploaded to Object Storage.
Why it matters
- Who has to decide decision Trying lakebase_vector on an existing pgvector table costs an index build and a recall check on real queries; the schema and application code stay as they are.
- What it costs cost Agent databases that idle between bursts no longer have to keep always-on compute sized to the whole vector index, since the index persists in object storage through suspends.
- What it limits constraint The fourfold cost claim is a 100-million-vector result against an unnamed pgvector vendor, so teams whose corpora fit in RAM have little evidence it applies to them.
- What's now possible capability An agent's retrieval tool can pull semantic and exact-term matches from the same Postgres that holds operational data, without an ETL pipeline into a separate search engine.
Next story · Build · 2d ago
Filtered queries on Tiger Data's alpha PRISM index can return fewer rows than LIMITConfirmed2 publishers
Databricks starts its case with pgvector's memory model. By its count, a 768-dimension float32 vector needs about 3.3 KB once HNSW graph links and Postgres overhead are included, so 100 million rows need roughly 330 GB of RAM to keep the index resident [11]. When the index spills to disk, Databricks says, queries slow by 10x to 50x [24]. HNSW has no global rebalancing, so restoring search quality takes a full REINDEX that locks the table and blocks writes [12]. Each query also runs in one backend process, and the index scan is never parallelized [13]. Databricks is grading an index it now competes with, though pgvector is also the most installed extension on its own platform [15].
The design follows from where Lakebase keeps data. Durable data sits in object storage, with RAM and local NVMe acting as caches in front of it. On that layout, Databricks writes, HNSW search means a series of random object store reads [5]. lakebase_vector uses hierarchical IVF to narrow each search to a small set of clusters, and RaBitQ quantization to compress vectors so more of the index fits in the cache layers [3]. The durable index stays in object storage [2]. I think IVF is the right choice for a database built this way. A cache in front of object storage handles a few cluster reads better than a chain of random hops through a graph.
Scale to zero is the piece of engineering I would credit first. When compute suspends, the index stays in object storage, and returning traffic reconnects to the same index instead of rebuilding it [4]. By Databricks' own figure, a pgvector index build that spills to disk takes almost 50 hours on a standard cloud instance [25].
The headline benchmark is a VectorDBBench run that Neon says used LAION-100M [20]. Databricks reports twice the throughput of the next best system, and sets the fourfold cost saving against "a cloud Postgres vendor using pgvector", before any autoscaling savings [19]. It reports P99 latency of 71 milliseconds at 97% recall [21]. That leaves about 3 in every 100 true nearest neighbors unreturned [17].
For the result to transfer, a corpus has to be large enough that an HNSW index would spill out of RAM. The recall target has to sit near 97%. Queries also have to keep hitting a working set small enough to stay cached. Databricks' per-vector figure puts a 10-million-row index near 33 GB [18], and at that size I'd expect pgvector's spill penalty to matter much less.
Hybrid search is plain SQL. An application combines vector and keyword results in one query [10]. Neon's example wraps that query in a Function that the agent calls as one tool, and the call returns ranked passages with their sources [16]. The Neon post does not say how the two rankings are merged into a single order.
Jordan Voves, an AI/ML architect at Conexiom, said in the Databricks post: "Lakebase Search gives us a whole new level of scalability over pgvector, and unlocks BM25 in the same serverless database. We use Lakebase to connect data to our agents at scale." [23]
What to watch
- An independent VectorDBBench run that names the pgvector vendor and instance sizes behind Databricks' 4x cost figure.
- Documentation of how lakebase_vector and lakebase_text scores are fused into one ranking in hybrid queries.
- Recall and latency figures for lakebase_vector at corpus sizes well under 100 million vectors, where pgvector's index fits in memory.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence40
- Adoption25
- Hype gap+35
- Incentives90
- Confidence50
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Lakebase Search brings search to Lakebase Postgres via two extensions, lakebase_vector (scalable approximate nearest neighbor search) and lakebase_text (BM25 full-text search), both generally available on AWS and Azure.
- [2]
lakebase_vector's durable index lives in object storage, while RAM and local storage cache frequently accessed data.
- [3]
lakebase_vector combines hierarchical IVF with RaBitQ quantization; IVF narrows the search to a small set of relevant clusters, while RaBitQ compresses vectors so more index data fits in the faster cache layers.
- [4]
When compute suspends, the index stays in object storage; when traffic returns, compute reconnects to the same index instead of rebuilding it.
- [5]
Lakebase Postgres keeps durable data in cloud object storage with RAM and local NVMe as caches; with this architecture, HNSW search means a series of random object store reads.
- [6]
Because the durable index lives in object storage and compute scales independently, users do not have to size always-on compute around the full index.
- [7]
Lakebase Search gives agents vector and BM25 search inside Lakebase Postgres without moving data to a separate search system.
- [8]
Switching from pgvector to lakebase_vector does not require migrating data or changing queries; users add a lakebase_ann index while keeping the same vector types, distance operators, and query syntax.
- [9]
lakebase_text adds the lakebase_bm25 index type, preserving standard Postgres tsvector types and query operators while adding corpus-wide BM25 ranking and top-K pushdown.
- [10]
Applications can combine vector and keyword results in one SQL query.
- [11]
A 768-dimensional float32 vector requires roughly 3.3 KB of memory after accounting for HNSW graph links and Postgres overhead; at 100 million rows, about 330 GB of RAM is needed to keep a pgvector index resident.
- [12]
Because HNSW lacks global rebalancing, restoring search quality requires a full REINDEX, which locks the table and blocks production writes.
- [13]
Each pgvector query runs on a single Postgres backend process, so the HNSW index scan is never parallelized.
- [14]
A TypeScript Neon Function can parse, chunk, embed and index data; AI Gateway provides an OpenAI-compatible embeddings endpoint; a storage_object_created trigger invokes the ingestion Function when files are uploaded to Object Storage.
- [15]
pgvector is the most installed extension in Lakebase Postgres.
- [16]
A Neon Function can wrap the hybrid query and expose it to an agent as one tool call that runs vector and keyword search in Postgres and returns ranked passages with their sources.
- [17]
At 97% recall, roughly 3 of every 100 true nearest neighbors are not returned.
- [18]
Using Databricks' figure of about 3.3 KB per 768-dimension vector, a 10-million-row pgvector index needs roughly 33 GB of RAM.
- [19]
On the VectorDBBench 100M benchmark, lakebase_vector delivers twice the throughput of the next best system and is 4 times cheaper than a cloud Postgres vendor using pgvector, before accounting for additional savings from autoscaling.
- [20]
Neon says its VectorDBBench tests ran on LAION-100M and that Lakebase Search led the tested systems on price-performance.
- [21]
In Databricks' tests, lakebase_vector delivered a P99 latency of 71 milliseconds at 97% recall.
- [22]
Customer Conexiom runs hybrid search with BM25 on over 100 million rows with half the compute footprint of its previous pgvector setup.
- [23]
Lakebase Search gives us a whole new level of scalability over pgvector, and unlocks BM25 in the same serverless database. We use Lakebase to connect data to our agents at scale.
- [24]
When a pgvector HNSW index spills to disk, queries turn into chains of random reads and performance drops by 10x to 50x.
- [25]
When a pgvector index build spills to disk, it takes almost 50 hours to build on a standard cloud instance.
Sources
2 independent publishers whose own reporting we read for this story.
- databricks.comLakebase Search: State-of-the-art full text and vector search for Postgres
1 article · September 28, 2026
- Lakebase Search: the retrieval primitive for agents on Neon
neon.tech
1 article · September 30, 2026
Topics and entities
Topics
- Hybrid Lexical and Dense Search
- Postgres extensions
- Serverless databases
- Vector Similarity Search
Entities
- Lakebase Postgres
- pgvector
- HNSW
- Databricks
- Neon
- BM25