Cross-Encoder Reranking: Worth the Latency

The short version: Cross-Encoder reranking delivers the most reliable relevance lift in the hybrid pipeline, especially fixing the vector-search habit of returning chunks that are semantically near but off-topic. It encodes every candidate, so latency grows linearly with candidate count—use it on the shortlist between initial ranking and Top-N, never on all chunks.

1. What reranking fixes

Start with the difference between the two encoders. A Bi-Encoder (vector search) encodes query and document separately, then compares vectors. That is fast, but the two never see each other, so token-level interaction is lost entirely. The classic symptom: you ask about the definition of a metric and get back an industry overview that merely mentions the term.

A Cross-Encoder takes a different path: it concatenates query and candidate into one input and encodes them jointly. Attention lets the model see how the words in the question actually appear in the candidate, capturing fine-grained interaction and pushing truly relevant items to the top. The division of labor follows: a Bi-Encoder can encode the whole corpus offline and answer with one query encoding; a Cross-Encoder cannot be precomputed and must run online. The most dependable setup in practice: BM25 plus dense vectors fused by weighted RRF (1.6 : 1.0) for recall, then a reranker for ordering.

2. Where latency comes from

The cost is mechanical: every candidate gets its own forward pass, so doubling candidates doubles rerank time—latency grows roughly linearly with count. Widen the shortlist from fifty to a hundred and the recall stage barely notices, while the reranker takes about twice as long. Rerank the whole index of a few hundred thousand chunks and a single query becomes unusable.

So treat the reranker as a judge of a shortlist. BM25 + vectors + RRF first produce a few dozen candidates; the Cross-Encoder reranks only those; the top N go to generation. Shortlist size is your latency dial—shrink it to cut latency, widen it for precision, and the trade stays explicit.

3. When it pays off

Two conditions make reranking worth it: candidates vary in quality, and the answer is highly sensitive to whether the top blocks are truly relevant. Compliance and legal Q&A are textbook cases—one irrelevant block in the top five and the answer loses its credibility, no matter how strong the generator is. There, reranking earns a permanent slot in the pipeline.

On a small corpus with already-clean candidates, the marginal gain is thin, and the saved latency is better spent on generation. Budget end to end: in one equity-research Q&A system, ten questions on stock 000002 took 4.6 seconds in total—under half a second per question. A reranker must fit inside that envelope to stay on the online path. In other words, reranking is not a free gain; it buys ordering quality with latency, and the budget should go where it matters.

4. Decide with evaluation

Do not add reranking on gut feel or a one-off demo. A/B "with rerank / without rerank" using retrieval metrics (HitRate@K / MRR / NDCG@K) and check whether the lift covers the latency cost; then rerun evaluation automatically on every change to chunking or rerank parameters, and even make a failing NDCG@K block the release. That is exactly the value of a regression gate: turning "feels useful" into "proven by data".

Reranking is the last stage of the hybrid pipeline. Initial ranking and fusion in Hybrid Retrieval; Chinese tokenization in CJK Bigrams; metric definitions in RAG Evaluation Metrics.

Decision Guide: Three Criteria for Adding a Reranker

• Candidate-set size: a Cross-Encoder pays off most when the fused candidate set still holds hundreds of items; with a dozen candidates the marginal gain is small while the latency cost remains.
• Query discriminability: when query wording differs sharply from document wording (spoken questions versus written documents), bi-encoder scores lack separation; cross-encoder fine-grained interaction widens the gap between relevant items and noise.
• Latency budget: the measured ten end-to-end questions on 000002 completing in 4.6 seconds shows that reranking before layered generation, combined with time filtering that keeps candidates in the low hundreds, keeps the whole chain within seconds.

Reranking is not an isolated layer: its input quality depends on upstream stages — fusion weights, time filtering, and institution anchoring all determine candidate purity, see Hybrid Retrieval; whether reranking actually helps must be verified by metrics, see RAG Evaluation Metrics.

FAQ

Q1: What does Cross-Encoder reranking fix?

It fixes the vector-search habit of recalling chunks that are semantically near but off-topic. By concatenating query and candidate into one input and encoding them jointly, it captures fine-grained interaction and pushes truly relevant blocks to the top, typically right after RRF fusion.

Q2: Why does reranking cost latency?

Each candidate gets an independent forward pass, so latency grows roughly linearly with candidate count. Reranking the whole index is impossible; keep the shortlist to a few dozen, and treat its size as the latency dial.

Q3: What is the correct way to use a reranker?

Let BM25 plus dense vectors plus RRF produce a few dozen candidates first, rerank only those with the Cross-Encoder, then take the top N into generation. You keep the rerank dividend while total latency stays inside budget.

Q4: When is reranking worth adding?

When candidates vary in quality and the answer is highly sensitive to whether the top blocks are truly relevant, as in compliance and legal use cases. On small, clean corpora the marginal gain is thin; decide with an A/B on retrieval metrics.