Hybrid Retrieval: BM25 and Dense RRF

The short version: dense-only retrieval misses model numbers and IDs. Hybrid retrieval runs BM25 (keyword) and dense vector retrieval in parallel and fuses them with weighted RRF (reciprocal rank fusion), weighted roughly 1.6 : 1.0 in favor of BM25.

1. Why vectors alone are not enough

Dense retrieval excels at semantics: ask about revenue performance and it surfaces paragraphs about business results, even in entirely different words. But enterprise documents carry another class of queries—stock codes, security numbers, metric terms, clause numbers. These live or die by literal matching, and embedding models tend to map semantically similar, lexically different words to nearby vectors, which erases exactly the difference that matters. Hitting ten out of ten questions on 000002 takes more than semantics; it takes precise matching on the code as a literal token.

BM25 is the mirror image: extremely strong on literal hits—codes and terms land at once—but blind to paraphrase, so a reworded query loses it. The weaknesses are each other's strengths, so the answer is not either-or: retrieve with both in parallel and fuse the rankings, giving every query type a suitable channel.

2. Two upgrades BM25 needs

Running BM25 on the raw query is too crude. Two fixes come first. One is colloquial-to-field mapping: rewrite casual phrasing like how much revenue into report field terms such as operating revenue, so everyday language can hit professional terminology. The other is CJK bigram tokenization: split Chinese text with a two-character sliding window, which avoids the noise of single-character splits while easing the missed recalls of whole-word segmentation.

Both fixes are plain engineering, yet they decide how strong the BM25 leg is. Bigram granularity and the upkeep of the mapping table are long-term work, not a one-time tokenizer install. The trade-offs deserve their own article—see Chinese Retrieval with CJK Bigrams.

3. RRF fusion

Each retriever returns a ranking; RRF scores by reciprocal rank—higher ranks contribute more—and needs no score normalization, so mismatched scales between the two channels do not matter. UGLYPEAR AI uses weighted RRF with BM25 and Dense at roughly 1.6 : 1.0, giving keyword signal slightly more weight. The tilt comes from measurement: the cost of semantic drift swallowing an exact hit is higher than the cost of missing one semantic neighbor.

The weight is neither arbitrary nor permanent. When corpus composition shifts—say, a batch of ID-dense documents arrives—rescan the weights and let the HitRate@K and NDCG@K curves decide. Treat the fusion ratio as a parameter that needs continuous validation, not a one-time setting.

4. Rerank after fusion

After RRF produces the initial ranking, a Cross-Encoder reranks—scoring each query-document pair together—which lifts ordering quality another step; the latency trade-offs are covered in Cross-Encoder Reranking. On top of reranking sit intent classification, time filtering, institution anchoring, and query rewrite, followed by layered generation. Time filtering is striking in practice: 18,805 candidates compressed to 827, screening out about 95.5% of stale documents and leaving far cleaner context for generation.

One fallback is worth knowing: when the embedding cache is incomplete, the system silently degrades to pure BM25—retrieval keeps running, just temporarily without the semantic leg. This stack follows chunking and enrichment; each step paves the way for the next, and the best single component does not make the best pipeline.

Hybrid retrieval is one link in a chain. Enrichment after chunking in Chunk Enrichment; Chinese tokenization choices in CJK Bigrams; latency trade-offs after fusion in Cross-Encoder Reranking.

Real-World Scenario: Where a 95.5% Time Filter Comes From

Every component of hybrid retrieval earns its place in production. In ashare-web retrieval smoke tests, time filtering compressed the candidate set from 17,590 to 829 and from 18,805 to 827 — about 95.5% filtered out — which is what keeps a query scoped to one reporting year from mixing in answers from other years. BM25 and dense vectors fuse at a 1.6 : 1.0 weighting (weighted RRF), with the keyword side slightly ahead because research queries are dense with precise tokens: company short names, metric names, reporting periods. Combined with intent classification, institution anchoring, and query rewriting (colloquial mapping plus CJK bigrams), vague phrasings like the report mentioning revenue become searchable queries.

One fault-tolerance detail: when the embedding cache is incomplete, the system silently degrades to pure BM25 — retrieval stays up with reduced precision, then recovers once the cache is rebuilt. That fits production requirements better than hard failure. For exact-token recall gaps, see CJK Bigram Retrieval; for the final ranking stage, see Cross-Encoder Reranking.

FAQ

Q1: What is hybrid retrieval?

BM25 keyword retrieval and dense vector retrieval run in parallel, then weighted RRF fuses the two rankings by reciprocal rank, weighted roughly 1.6 : 1.0 toward BM25. Exact IDs and semantic queries each get a retrieval channel that truly serves them well every time.

Q2: Why does dense-only retrieval miss model numbers and IDs?

Embedding models map semantically similar, lexically different words to nearby vectors, which erases the surface differences that stock codes and security IDs depend on for exact recall. BM25's literal matching covers precisely that gap, which is why hybrid retrieval exists.

Q3: What is RRF fusion?

Reciprocal rank fusion: each retriever returns a ranking, and the scores are summed by reciprocal rank—higher ranks contribute more—with no score normalization needed across the two channels. The weighted variant can additionally give BM25 slightly more weight than Dense.

Q4: What happens when the embedding cache is incomplete?

The system silently degrades to pure BM25, so retrieval keeps running without the semantic leg, and results remain available as a workable fallback. Once the cache is complete again, hybrid retrieval resumes automatically, with no manual intervention required at all.