Ranking

Classification asks: which label? Ranking asks: which item should come first, second, third?

RAG is ranking. Tool-doc retrieval is ranking. “Which past ticket is nearest?” is ranking. A classifier that says “relevant / not” on each chunk independently can still dump 40 “relevant” chunks and miss the one the user needed at position 1.

The usual scores: cosine, a learned reranker, a cross-encoder. You do not need those names to measure. You need a sorted list and a set of gold items. The usual metrics:

• Precision at k (P@k) — of the first k items, how many are relevant?
• Recall at k — of all relevant items, how many appeared in the first k?
• MRR (mean reciprocal rank) — 1 / position of the first relevant item, averaged over queries

If the right paragraph is 5th and you only stuff top-3 into the prompt, recall at 3 is 0 for that query. The LLM never sees it. That is a ranker bug, not a “dumb model.” Prompting “please cite well” does not move the fifth chunk into the window.

Classification accuracy on “is this chunk relevant?” is not P@k. You can have high accuracy and still rank the useful chunk last among the yeses.

What a ranking eval looks like

Each row is a query plus a gold set of relevant item ids (chunks, tickets, tool docs). You run your scorer, sort descending, compute P@k, recall@k, and the reciprocal rank of the first hit. Average over queries. Also report by type: policy vs table vs log, because averages hide a dead corpus slice.

k is a hyperparameter of the agent, not only of the metric. The k you measure should be the k you stuff into the prompt (or a bit larger, if a reranker will throw some away). Measuring P@20 while you only send 3 chunks is lying about what the generator saw.

Run to execute this in your browser. Nothing is sent to a server.

What printed: rank order by score, highest first. refund-policy at 0.77 is first (hit). refund-faq at 0.71 is second (hit). shipping at 0.41 is third (miss). Then password and office-hours. P@1 is 1.0 (the first slot is gold). P@3 is 2/3 because shipping sneaks into the top three. Reciprocal rank is 1.0 because the first relevant item is at position 1. If you had sorted by the wrong score, P@1 would collapse and the agent would cite shipping.

Change k in your head: if the agent only sees top-1, FAQ never enters. If it sees top-3, both gold chunks can enter, plus one distractor. Distractors are how generators cite the wrong policy.

Rerank vs retrieve

First-stage retrieval might use a fast cosine over many chunks. A reranker then scores the top 50 with a slower function and keeps 4. Measure both stages. If gold is not in the 50, the reranker cannot save you. That is recall-at-50 of the first stage. Raise k, fix chunking, or fix the query text (the question was two turns ago; you embedded the last “thanks”).

Query construction is ranking work: embed the user question, not the whole fluffy chat, unless you have evidence the extra turns help. That evidence is P@k on a freeze, not a vibe.

Gold sets, distractors, and k that matches the prompt

A ranking eval is only as honest as the gold set. If labelers mark every neighbor “relevant,” P@k cannot fall, and you will never see a miss. Mark must-include chunks: the paragraph that actually answers the question. Optional extras can be a second set. Metrics should use the must-include set.

Distractors are the point. A corpus of only the right docs is a toy. Include shipping policy next to refund policy. Include an outdated refund PDF if that PDF still sits in production. If the outdated chunk ranks first, you have a corpus bug, not a generator bug.

k must match the window. If the agent stuffs 4 chunks, report P@4 and recall@4. Also report recall@20 of the first stage if a reranker later cuts to 4: that tells you whether gold never entered the rerank pile. Two-stage systems need two numbers.

MRR cares about the first hit only. That is the right summary when one paragraph is enough. If two chunks must both enter the prompt (a table and a policy), look at recall@k of the set, not only MRR. Pick the metric from the job: “did the needed facts appear in the window?”

Log the ranked names. When the agent cites shipping, you want to see whether shipping was rank 1 (retriever) or rank 5 stuffed anyway (too-large k) or not in the list (generator invention). Ranking metrics without neighbor logs turn into arguments.

Common mistakes

• Measuring classification accuracy of a relevance head and calling it retrieval quality.
• k in the metric not equal to k in the prompt.
• Gold labels that mark every neighbor relevant.
• One global P@k hiding a dead product area.
• Fixing ranking bugs by lengthening the generator prompt.

How agents use this

Log neighbors and scores for every retrieval. Eval with (query, must-include-chunk) pairs. If top-k is 4 and the gold chunk is 5th, raise k, fix chunking, or train a reranker. Do not add “please cite well” to the prompt and call it a retrieval fix.

Tool routing can be ranking too: score each tool doc, pick top-1, abstain if the gap is tiny. Then you still want a confusion matrix on the final pick. Ranking metrics tell you whether the right tool was near the top. Classification metrics tell you whether you called it.

Note:Classification accuracy on “is this chunk relevant?” is not P@k. You can have high accuracy and still rank the useful chunk last among the yeses.