Number Class
← Back to Explore
Computer Science Databases & Information Systems Information Retrieval

Vector Databases Were Never Built to Remember Correctly

The same engineer who invented the approximation algorithm now profits from a market betting billions that 'close enough' counts as memory

S.J. Nam 8 min read
Vector Databases Were Never Built to Remember Correctly

The Fine Print on "Nearest"

Ask a vector database to find the ten documents closest to your query, and it will hand you ten documents with total confidence. What it won't tell you, unless you go digging into the engineering docs, is that "closest" is a promise it doesn't fully keep.
In an ANN search, the k-returned vectors aren't the true top k-nearest neighbors because the ANN search calculates approximate distances and might not look at all the vectors in the dataset.
Google's own Spanner documentation calls this "recall loss" and treats it as a routine cost of doing business:
how much recall loss is acceptable to you depends on the use case, but in most cases, losing a bit of recall in return for improved database performance is an acceptable tradeoff.

That sentence is the whole industry in miniature. Vector databases are routinely described, in marketing copy and explainer articles alike, as the "memory" behind modern AI — the thing that lets a chatbot recall your company's return policy or a coding assistant find the right function across a million-line repo. But memory, in the everyday sense, implies fidelity: you either remember where you left your keys or you don't. Vector databases were built, deliberately and explicitly, to sometimes not remember correctly, in exchange for speed. That's not a bug the field is embarrassed about. It's the founding design decision, and the more interesting story is what happens once you take that decision seriously.

The Man Who Built the Vault, Then Invested in One

In 2016, researchers Yury Malkov and Dmitry Yashunin published the algorithm that would become the default machinery inside nearly every commercial vector database: Hierarchical Navigable Small World graphs, or HNSW.
Introduced by Yury Malkov and Dmitry Yashunin in 2016, HNSW addresses the critical limitation of existing similarity search methods that struggled to maintain both speed and accuracy as dataset sizes grew to millions or billions of vectors.
The trick is architectural:
the algorithm achieves logarithmic search complexity through a sophisticated multi-layer graph structure that separates connections by characteristic distance scales, enabling efficient navigation from coarse to fine granularity during search operations.
Instead of comparing a query against every vector in a dataset — the only way to guarantee a true nearest neighbor — HNSW hops through a graph, layer by layer, betting that a search which almost always finds the right answer, almost instantly, is worth more than a search that always finds it, slowly.

Here's the detail that rarely makes it into the explainers: Malkov didn't just publish the algorithm and walk away. According to a business breakdown by Contrary Research, Pinecone — one of the highest-profile vector database companies — counts among its backers
industry veterans such as Bob Muglia (former CEO of Snowflake) and Yury Malkov (inventor of the Hierarchical Navigable Small Worlds algorithm)
. The person who engineered the approximation now has equity riding on the world accepting it as good enough. That's not a scandal; it's just a useful reminder that "approximate" was never a temporary compromise on the way to something more exact. It's the business model.

Pinecone itself traces back to founder Edo Liberty's time at AWS, where he concluded
machine learning development was being held back by the lack of access to vector databases, a type of database that powers machine learning models by storing data as high-dimensional vectors
, technology that
only large technology companies with ample resources, such as Google and Facebook, had been able to take advantage of.
The company he built to democratize that access went from founding in 2019 to a
$100 million Series B at a $750 million valuation
in April 2023, led by Andreessen Horowitz, bringing its total raised to $138 million.

Amnesia by Design

The other half of this story is why anyone needed approximate memory in the first place, and it has less to do with databases than with a limitation baked into large language models themselves. An LLM's knowledge is frozen at training time and boxed into a fixed context window; ask it about something that happened after its cutoff, or something too specific to have made it into training data, and it will either say so or, worse, guess. As Oracle's explainer puts it, a sports league's chatbot built on a generic model
wouldn't be able to discuss last night's game or provide current information about a particular athlete's injury because the LLM wouldn't have that information—and given that an LLM takes significant computing horsepower to retrain, it isn't feasible to keep the model current.

The fix, proposed in a 2020 paper by
Patrick Lewis and colleagues, with coauthors from the former Facebook AI Research (now Meta AI), University College London and New York University
, was retrieval-augmented generation — RAG — described in the paper as
"a general-purpose fine-tuning recipe" because it can be used by nearly any LLM to connect with practically any external resource.
Lewis later admitted, a little sheepishly, that the acronym stuck before anyone thought hard about branding:
"We always planned to have a nicer sounding name, but when it came time to write the paper, no one had a better idea."
Vector databases became the storage layer for that recipe — the place where a company's documents, chat logs, or product catalog get converted into embeddings and parked, waiting to be fetched at query time. Calling this "memory" is a metaphor of convenience. What's actually happening is that a fundamentally amnesiac model is being handed a filing cabinet on every single request and asked to skim it fast enough that the user doesn't notice the handoff.

The Industry's Bet, and the Numbers Behind It

Investors have priced this filing-cabinet trick as a serious business. Estimates of the vector database market disagree with each other by a wide enough margin to be worth noting rather than smoothing over: Grand View Research put the market at
USD 1.66 billion in 2023, projected to reach USD 7.34 billion by 2030
, Market Research Future has it at
USD 1.73 billion in 2024, growing to 7.86 billion USD by 2034
, and GMI Insights instead starts its clock at
USD 2.55 billion in 2025
. Three research shops, three different base years, three different trajectories — a sign that this is still a market too young and too fast-moving for anyone to agree on where it even stands today, let alone where it's going. What the estimates do agree on is direction: by 2024,
MongoDB, Redis, DataStax, KX, Qdrant, Pinecone, and Zilliz collectively accounted for around 45% of the global market share
, a moderately concentrated field racing to be the default place AI systems keep their "memory."

The Old Technology Sneaking Back Through the Service Entrance

Here is the part that undercuts the memory framing most directly. If vector search were truly a superior substitute for exact retrieval, the industry wouldn't be quietly re-hiring the technology it was supposed to make obsolete. Yet across the RAG literature, one finding keeps surfacing:
hybrid retrieval (combining vector search + keyword search) consistently outperforms pure vector search for factual accuracy.
IBM's own framing concedes the same point, noting that
hybrid retrieval approaches used in RAG systems combine semantic retrieval with traditional search methods to improve both recall and precision, particularly in enterprise environments where data is heterogeneous and complex.

Put plainly: the semantic, meaning-based search that made vector databases exciting in the first place isn't trusted, on its own, to get facts right — even by the companies building on top of it. Keyword matching, the deterministic, unglamorous technology vector search was meant to retire, is being wired back in as a safety net, because
generative models are probabilistic, meaning they generate plausible responses, not verified facts,
and an approximate memory feeding a probabilistic generator compounds uncertainty rather than resolving it.

None of this means vector databases were a bad bet — the funding rounds, the market growth, the sheer usefulness of semantic search for fuzzy queries all argue otherwise. But the framing of "memory" flatters the technology in a way that obscures its actual trade, which is: exactness for speed, certainty for scale. The more honest description is that AI got a filing system that's fast, cheap, and occasionally, quietly, wrong about what's actually in the file — and the industry's response, tellingly, hasn't been to fix that so much as to hire an old-fashioned librarian to double-check the robot's work.

References