Recall@k and RAG Evals

Recall@k is simple: for each eval question, you know a gold chunk id (or a set of ids). Retrieval succeeds if that id is in the top k after filters, hybrid, fusion, rerank — whatever you actually ship. You do not need a judge model. You need labels and a function.

If stage 1 recall is 0, do not spend a week on the system prompt. Generation cannot cite a chunk that never entered the pile.

You also want, on the same golden set:

• Faithfulness of the final answer (citation substring checker)
• Refuse rate on questions with no gold chunk (should_refuse: true)
• Leak rate on neighbor-tenant questions (must be 0)
• Latency of retrieve + rerank (and hops if you allow them)

Keep a golden set of (question, gold_ids, should_refuse). Re-run it in CI when you change chunking, embeddings, alpha, ANN parameters, or rewrite tables. That is the cheapest RAG eval. The Evals track will add judges and traces at agent scale. This classroom already has a unit test.

Labels are the work

Gold ids must match current chunk ids. If you re-chunk, you re-label or you map via source+offsets. Stale gold is how you “fail CI” after a good ingest. Store gold as source path plus a quote, then resolve to id at eval time if ids churn.

Include hard questions: identifiers, paraphrase, negation, no-hit, multi-hop joins. If 80 items are easy FAQs, recall looks fine. Force the tail.

Do not use production traffic as the only eval. Production has no gold. Sample it to propose new labels, then humans confirm.

Run to execute this in your browser. Nothing is sent to a server.

One number for hits, one for honest refuses. No model in the loop. k=1 vs k=2 shows whether gold was sitting in slot 2. If recall jumps from 0.5 to 1.0 at k=2, packing n=1 was the bug, not the embedder. Print both. Add a tenant leak row in production tests: retrieve as Acme, gold empty, neighbor chunk must not appear.

Word overlap stands in for your real retrieve function. Swap the body; keep the GOLD shape.

Grade retrieval before generation

A faithfulness fail with recall@k=1 is a generate/cite bug. A faithfulness fail with recall@k=0 is a retrieve bug you mislabeled. Split the dashboard. Trace hop queries on failures (agentic RAG). If rewrite changed the query, store both strings in the eval record.

CI should fail the build when recall@k on the critical tag (billing, runbooks) drops below a floor you chose. Do not block on exact-string answer diffs. Do block on gold chunk missing.

Labels, stages, and what not to judge yet

Gold as source path plus quote, resolved to id at eval time, survives re-chunking better than a raw c17. If ids churn every ingest, your eval is measuring ingest noise.

Hard cases: identifiers, paraphrase, negation, no-hit, tenant leak, a join that needs two hops. Easy FAQ-only sets hide every lesson in this track.

Grade retrieve before generate. recall@k on packed ids (after hybrid, RRF, rerank, MMR, pack), not only on stage-1 piles. If pack drops gold, recall@k of the pile can look fine while the prompt never saw the fact. Log both pile recall and packed recall.

Refuse accuracy on should_refuse rows. Leak rate must be 0 on neighbor-tenant rows. Latency of retrieve plus rerank sits next to those numbers. ANN vs brute on a subsample is an index eval inside the same CI.

Do not hire a large model to score retrieval. Gold ids are cheaper and less circular. Judges come later for leftover prose. They do not replace this unit test.

When rewrite runs, store raw and rewritten queries on the eval record. Failures need both strings.

Common mistakes

• Only judging final prose with a large model.
• Gold ids that do not exist after re-chunk.
• No refuse cases.
• Measuring ANN latency without recall.

How agents use this

Grade retrieval before generation. Trace hop queries on failures. When retrieve is a library the later loop calls, this eval still owns the library. The loop evals are extra: did it call retrieve vs get_invoice. Do not skip library tests because you plan a loop.

You do not need an LLM to test retrieval. You need gold chunk ids and a number.

Run the set in CI on every chunking, embedder, alpha, ANN, and rewrite change. Report packed recall@k, pile recall@k, refuse accuracy, leak rate, and latency. Resolve gold from path plus quote when ids churn. Include identifier questions and no-hit questions. When packed recall is 0, stop tuning the generator. When pile recall is 1 and packed recall is 0, stop tuning the embedder and open the packer. Store raw and rewritten queries on the row. That is enough to debug this track without a judge.

Labels are the work. Easy FAQ-only sets hide hybrid and hop bugs. Neighbor-tenant rows must stay at leak rate 0. Stage-1 recall of 0 means rerank is decoration. You can write this eval with lists, sets, and division. No extra numeric library. No judge model. Generation evals wait until gold is in the pile.

Keep gold as path plus quote when ids churn. Re-run in CI. Split the dashboard: retrieve numbers first, then citation substring rate, then refuse. A pretty chat demo is not an eval. A gold id in top k is. Packed recall and pile recall are different numbers. Refuse rows and leak rows belong in the same job. Trace hop queries when a join fails. No LLM is required to know whether the chunk appeared. Gold path plus quote, resolve to id, check membership in top k, check refuse on empty gold, check leak on neighbor tenant. Put that function in CI. Generation waits.