Number Class
← Back to Explore
Computer Science Databases & Information Systems Information Retrieval

Hybrid Search Doesn't End the Tuning Problem, It Just Hides It

OpenSearch's own benchmarks show the default fusion method, Reciprocal Rank Fusion, scoring 3.86% lower than score-based alternatives on NDCG@10

S.J. Nam 8 min read
Hybrid Search Doesn't End the Tuning Problem, It Just Hides It

A Dutch customer typed a simple question into an automotive company's AI assistant: "kenteken AB-123-CD apk verlopen?" — has the inspection on license plate AB-123-CD expired? The dense-vector retrieval system, built on state-of-the-art sentence transformers, did exactly what it was trained to do.
It returned semantically related documents about APK inspections, vehicle registration, and general automotive services. Technically correct. Semantically relevant. But critically wrong.
The one document that mattered — the record tied to that exact plate — never surfaced, because nothing in an embedding space tells a model that "AB-123-CD" is a proper noun rather than a string of noise. The team's fix was not more parameters or a bigger model. It was reintroducing keyword search alongside the vectors, which took retrieval accuracy from 62 percent to 91 percent,
a 48% improvement that transformed the RAG system from "good enough" to genuinely reliable
.

That story has become the industry's favorite parable, and it's usually told with a tidy moral: vectors miss exact matches, so bolt on BM25, fuse the two lists, ship it. The moral is true as far as it goes. But it stops one step short of the more uncomfortable finding buried in the same body of evidence: the fusion technique everyone reaches for by default — the one marketed as requiring "no tuning" — has been shown, in the search vendors' own benchmarks, to score measurably worse than the alternative it replaced. Hybrid search doesn't eliminate the tuning problem that made pure vector search unreliable. It relocates it, one layer downstream, where fewer people think to look.

The Blind Spot Was Never About Meaning

Vector embeddings are extraordinary at capturing what a piece of text is about. They are structurally indifferent to which specific token appears, which is precisely the property that makes them fail on identifiers. BM25, the ranking function underneath Elasticsearch, OpenSearch, and most lexical engines, works on the opposite principle.
It relies on probabilistic information retrieval theory with three built-in mechanisms that directly address the exact-match problem. Inverse document frequency (IDF) measures how rare a term is across the corpus.

Common words like "service" or "deployment" receive low weight, while rare distinguishing tokens like "v3.2", "ERR_PAYMENT_GATEWAY_TIMEOUT" or "payment_v2_enforce" receive high weight.
A part number, an error code, a license plate — these are exactly the strings BM25 was built to reward and embeddings were never built to notice.

What makes this genuinely counterintuitive is the age of the algorithm doing the rescuing.
BM25 was published in 1994. The math is simple enough to fit on a whiteboard.
Yet in production benchmarks run in 2025, that three-decade-old formula
still outperforms multi-billion-parameter dense embedding models on a meaningful slice of real-world queries
. The newest thing in modern search infrastructure is being propped up by one of the oldest.

Combining Two Systems Doesn't Combine Their Simplicity

The natural next step — run both searches and merge the results — sounds like a free lunch, and on aggregate metrics it often looks like one. On the WANDS e-commerce dataset,
baseline BM25 and pure KNN are statistically indistinguishable at NDCG 0.6983 vs 0.6953, while a well-tuned hybrid reaches 0.7497 — a 7.4% NDCG lift over either alone
. That's a real gain. But "well-tuned" is doing quiet, heavy lifting in that sentence, and the tuning is rarely as simple as flipping a switch.

Consider the account from engineer Ranjan Kumar, who documented his own hybrid build in granular detail rather than presenting a clean success story.
We started with alpha=0.5 (equal weighting). Precision was 0.68. After analyzing failures, we found BM25 was missing on too many paraphrase queries. Increased alpha to 0.6 (more dense weight). Precision dropped to 0.65
, so the team eventually settled on 0.4 after testing five different values — and then, in his own words,
the corpus changed
, and the calibration work started over. That is not the story of a system that "just works" once you add embeddings to your keyword index. It's the story of a new hyperparameter — how much to trust semantics versus tokens — that has to be re-earned every time the underlying data shifts.

There's a resource cost too, one that's easy to omit from architecture diagrams. On a benchmark cluster running five million product documents,
pure BM25 can often deliver results in 5–10ms; hybrid adds 20–30ms
, and for teams with strict low-latency requirements, that gap is not trivial. Hybrid search is a trade, not a strict upgrade.

The Fusion Algorithm Nobody Interrogates

Here is where the standard explainer usually ends: run BM25 and vector search in parallel, then merge the ranked lists with Reciprocal Rank Fusion, the method Elasticsearch, OpenSearch, and virtually every modern hybrid pipeline now ship as the default. RRF has an appealing simplicity — it originates in a 2009 SIGIR paper by Cormack, Clarke, and Büttcher, and its formula only needs the position of a document in each ranked list, not the raw relevance score.
Reciprocal rank fusion ignores raw scores entirely and focuses on how high a document appears in each list. Documents ranked near the top of any list are rewarded strongly, and documents appearing in multiple lists receive additive boosts.
Because it sidesteps the awkward problem of BM25 and cosine similarity living on incompatible scales, RRF has been marketed, accurately, as needing almost no configuration.

That simplicity is bought with information loss, and the people who built RRF's most prominent implementation have quietly admitted as much. In introducing RRF support to OpenSearch's Neural Search plugin, the engineering team ran their own comparison across six datasets and reported the result plainly:
RRF scores 3.86% lower than traditional score-based methods
on NDCG@10. The same post frames this as an acceptable trade for
significant latency improvements, making it particularly suitable for high-throughput search applications
— which is a legitimate engineering argument, but it is a different argument than "RRF gives you the best of both worlds." It gives you most of both worlds, faster, with a documented accuracy discount that almost none of the breezier hybrid-search tutorials mention. Elastic itself later shipped a "weighted RRF" and a separate linear retriever specifically because, as their search-labs team put it,
Weighted RRF expands RRF to production use cases that require more control than standard RRF could provide
— an implicit concession that the original one-size-fits-all formula wasn't fitting every size.

Where This Leaves the Practitioner

None of this is an argument against hybrid search — the license plate story and the WANDS numbers settle that question. It's an argument against treating "hybrid" as a finished answer rather than a family of decisions, each with its own failure mode. Vector-only search fails silently on identifiers because embeddings encode meaning, not identity. BM25-only search fails on paraphrase and vocabulary mismatch for the mirror-image reason. And a hybrid pipeline fails more subtly than either, when the fusion layer chosen for its convenience quietly discounts accuracy the team never measured in the first place. The engineers at KX put the remedy plainly:
Establishing a vector search without a proper evaluation framework can lead to inconsistent query performance and a lack of understanding of the underlying issues. Create a small, reliable eval set: Even 50–100 labeled queries are enough to reveal issues
.

The uncomfortable takeaway is that there is no configuration of vector search, keyword search, or their fusion that removes the need for that evaluation set. Every layer in the stack — embeddings, BM25 weighting, alpha blending, RRF's rank constant — is a place where a system can look correct in a demo and fail on the one query that actually mattered to a customer. The teams that get hybrid search right aren't the ones who picked the fashionable architecture. They're the ones still watching the logs the Tuesday morning something like AB-123-CD comes through.

References