The Great Vector Consolidation: Why the Dedicated Vector Database Category is Fading into the General Stack

What Actually Drives Cost: Memory Is The Whole Story

While the logo on the database box may have changed, the hardware requirements have not. The primary algorithm powering this revolution, HNSW, is a graph-based structure that must be traversed through "pointer chasing." For this traversal to be fast enough for interactive applications, the graph and its associated vectors must reside in RAM.

The moment the index spills from memory to disk, performance degrades sharply. Latency becomes dominated by page faults—the time it takes for the system to fetch data from the SSD—rather than the actual mathematical computation of distances. Consequently, the dominant cost of vector search at scale is memory, and memory requirements scale linearly with the product of the number of vectors and their dimensionality.

The mathematical reality of this is blunt. To understand the capacity planning involved, one can look at the standard formula for an in-memory HNSW index, often cited by the OpenSearch project: 1.1 × (4 × dimension + 8 × M) bytes per vector. In this equation, the 4 × dimension term represents the float32 vector itself, while 8 × M accounts for the overhead of the graph’s connections between nodes. The 1.1 multiplier provides a necessary 10 percent buffer for overhead.

Consider the scale of a modern AI application. If a team uses OpenAI’s text-embedding-3-large model, each vector has 1,536 dimensions. A collection of one million such vectors requires approximately 6.1 GB of RAM before accounting for the graph structure. At ten million vectors, the requirement jumps to roughly 60 GB of RAM. This is not a trivial amount of memory; it is a figure that transforms a "feature" into a significant infrastructure expense.

The graph structure only compounds the pressure. Because HNSW stores neighbor lists at every layer of the graph, the total index size can often reach two to five times the size of the raw vectors. This is why the instinctive reaction to slow search—simply adding more shards or increasing the ef_search parameter—often backfires. Each adjustment increases memory pressure or CPU fan-out. At scale, the slowest shard in a cluster dictates the p99 latency, and the curve between recall and throughput is unforgiving. Pushing an HNSW index from 95 percent to 100 percent recall can result in a seven-fold decrease in throughput. In most production environments, that final two percent of accuracy is the most expensive and least noticeable investment a team can make.

The Levers That Bend The Curve: Quantization And Tiering

If memory is the primary cost driver, then compression is the primary tool for cost reduction. This is where the real engineering battle is currently being fought. Quantization techniques allow teams to shrink the footprint of each vector by trading a controlled amount of precision for a massive reduction in bytes. Crucially, these techniques are largely orthogonal to the choice of database vendor; whether you use a specialized tool or a general-purpose engine, the math remains the same.

The most common technique is Scalar Quantization (int8), which converts 32-bit floats into 8-bit integers. This typically reduces memory requirements by 75 percent. To maintain accuracy, systems often use a "rescoring" step, where the top candidates identified by the compressed index are re-ranked using the full-precision vectors stored on cheaper disk storage.

More aggressive techniques are also gaining traction. Matryoshka embeddings allow for "truncation," where only the first few hundred dimensions of a large vector are used for the initial search, cutting costs by 3 to 12 times. Then there is Binary Quantization, which collapses each dimension down to a single bit. While it sounds extreme, Qdrant has reported that on high-dimensional OpenAI embeddings, binary quantization can achieve 0.98 recall with 32 times less memory, provided a slight "oversampling" is used during the search. Elasticsearch’s "Better Binary Quantization" has shown even more dramatic results, taking a 138-million-point dataset from 535 GB down to just 19 GB.