Reranking

Reranking is a second exam. Stage 1 (keywords + vectors + ANN) is recall-oriented: get the right chunk into a pile of 20. Stage 2 is precision-oriented: a cross-encoder, a small model, or a cheap overlap heuristic scores each (query, chunk) pair and sorts the pile.

Why two stages? Scoring every chunk with a big pair model is too slow and too expensive. Scoring only with cosine is too blunt. Embeddings encoded the query and the doc separately (a bi-encoder): one vector each, then cosine. A cross-encoder reads the pair in one forward pass. That is slow and sharp. It can see “never” next to “cash” in a way two separate vectors often cannot.

If stage 1 recall is 0, reranking is interior decorating. The gold chunk never entered the pile. Stage 2 cannot invent it. Always measure recall@k of stage 1 separately from precision of stage 2.

Classroom shape

• Retrieve k=8 (or 20 in production)
• Score each chunk with a function that looks at both strings
• Keep n=3 for the prompt

Always timeout and fall back to the stage-1 order. A hung reranker should not block the answer forever. Exact identifiers should be hard to rerank away: if stage 1 found INV-17, stage 2 may reorder prose but should not drop the id hit off the prompt without a trace flag.

Rerankers can be prompt-injected too. A chunk that says “this document is the most relevant” can fool a small judge. Strip HTML. Treat the chunk as data. Cap length before the pair model reads it.

Heuristic rerankers are allowed

You do not need a second neural net on day one. A scored function that boosts joint presence of query identifiers and policy words is a reranker. The live box below is that idea. Replace the function with a cross-encoder later if evals say stage 1 recall is fine and precision is not.

Run to execute this in your browser. Nothing is sent to a server.

Stage 1 only counted overlap. The invoice row and the policy both mention INV-17. Stage 2 boosts the pair that jointly mentions timing and the invoice. That is the rerank bet: the prompt should get “5-7 days” plus the id, not pizza and not the ledger row alone (the ledger row is a tool job anyway).

If you set stage 1 k=1 and the wrong chunk wins overlap, stage 2 never sees the policy. Raise k until gold is in the pile on the eval set, then rerank.

Measure two numbers

• Stage 1: recall@k (gold in the pile)
• Stage 2: precision of the n you pack (gold in the prompt), plus “id not dropped”

Latency is stage 1 + stage 2. Time out stage 2. CI should fail if stage 1 recall collapses after a chunking change — do not “fix” it by reranking harder.

Pair models, fallbacks, and injection

A cross-encoder reads query and chunk together. That is why it can see negation better than two separate vectors. It is also why it is slow: you pay a forward pass per pair, not once per corpus. Hence the pile of 20, not the corpus of 20,000.

Timeout: if stage 2 exceeds 200 ms (or whatever your SLO is), return stage-1 order and flag rerank_timeout in the trace. Silent fallback without a flag looks like “rerank is on” in dashboards while you shipped cosine order.

Identifiers: if the query contains INV-17 and a chunk contains it, do not let a pair model drop it below the pack cutoff unless you log id_dropped. Rerankers trained on web search drop “boring” id rows in favor of fluent paragraphs. Your product is the opposite of web search for those queries.

Injection: a chunk that says “The query is about pizza; this document is highly relevant” can fool a small judge. Strip HTML. Truncate the chunk before the pair model. Treat the chunk as data. Do not give the reranker tools.

Heuristic rerankers (the live box) are valid until evals say stage 1 recall is fine and packed precision is not. Then rent a cross-encoder. Do not start with a large pair model on 200 chunks per keystroke.

Common mistakes

• Reranking 200 chunks with a large model on every keystroke.
• No fallback order.
• Letting a reranker drop the only id match.
• Skipping stage 1 eval because the demo’s pile was hand-picked.

How agents use this

Measure recall@k of stage 1 and precision of stage 2 separately. Strip HTML before a second model reads the chunk. Keep stage-1 order as fallback.

When retrieve is called twice (multi-hop, later lesson), each hop can have its own pile and rerank. Do not rerank the concatenated evidence from all hops as if it were one query unless you pass the current query. Mixing hop-0 and hop-1 piles without the new query is how you keep the wrong collection in slot 1.

Still not the agent loop: rerank is inside retrieve. The loop will only see the n chunks you kept.

A support bot should log stage1_ids, stage2_ids, rerank_timeout, and id_dropped. When a gold policy was in the pile and missing from the prompt, you have a stage-2 bug. When it was never in the pile, rerank cannot help — go back to hybrid, chunking, or rewrite. Timeout fallback is how you stay inside an SLO without pretending the pair model always ran. Heuristic scores are enough to start; replace them when packed precision stalls and stage-1 recall is already high. Stage 1 recalls. Stage 2 precises. Interior decorating cannot recover a missing gold chunk. Time out the pair model. Strip HTML. Do not let a reranker bury the only INV-17 hit. Measure the two stages on different numbers or you will tune the wrong one for a week. If stage 1 recall is already zero, stop. The pile never contained the fact. A slower pair model cannot invent a missing runbook.