Scores and Thresholds

A retriever that always returns k chunks will always look busy. That is not the same as being right. Top-1 of a bad set is still a miss. Cosine 0.12 is a shrug. Stuffing that chunk into the prompt is how agents invent procedures: the model must say something, the chunk is about “the company” in general, and a cash refund appears.

Set a threshold tau. If the best score is below it, return no evidence. Say “nothing in the corpus matched” (later: structured refuse). Do not lower tau to zero because a demo looked empty. Empty is honest.

Print scores in traces. Cosine 0.81 vs 0.12 is the difference between a hit and a shrug. Do not hide the number behind “here are your results.” Humans cannot debug “the bot was unsure.” They can debug “best cosine 0.14, tau 0.45.”

What the number means (and does not)

On one embedding model, good hits might cluster around 0.7–0.9 and misses around 0.1–0.3. On another model the clusters move. You must calibrate tau on labeled pairs for that model and that corpus. A pair is (question, gold chunk id) plus questions that should hit nothing.

Calibration is not a slide. Sort held-out questions by best cosine. Pick a tau that keeps gold chunks and drops pizza. There is no universal 0.75. Vendors who show a “relevance %” have a private mapping. It still drifts when you change models.

If you change the embedding model, recalibrate. Copying last quarter’s tau is how you either retrieve junk or retrieve nothing after an upgrade.

Scores from keyword search and cosine are not on the same scale. Do not write if bm25 > 0.5 or cosine > 0.5. Hybrid fusion and RRF exist because of this. Thresholding fused scores is a new calibration, not the cosine tau reused.

Always-k vs maybe-zero

Always-k is tempting in a UI that has a “sources” panel. Empty panels look broken. They are not. Retrieved 0 is a first-class outcome. Log it. Eval it. The refuse lesson will turn it into a JSON reason. This lesson is the retriever’s job: do not ship junk as evidence.

Raising k to 50 because tau felt strict does not fix geometry. You add 49 more shrugs. Packing will then drop them, or worse, keep the long ones. Fix ingest, chunking, hybrid, or rewrite. Do not drown the prompt.

Run to execute this in your browser. Nothing is sent to a server.

The pizza query with a threshold returns nothing. The same query with tau=0 returns the least-bad junk (probably menu, still wrong). Agents should prefer empty. The OOM query keeps a real hit. Read the three prints as three product behaviors: hit, honest miss, busy miss.

The or 1.0 on a zero norm avoids divide-by-zero in this toy. A zero vector should still fail the threshold. Do not “fix” zeros by treating them as length 1 without looking.

Logging

Log, per query: top scores, ids, tau, k, kept count, embedding model name. When kept count is 0, that is not a chat failure. When gold id is missing but scores look high, your labels or chunk ids drifted. When gold id is present at score 0.2 and tau is 0.5, your threshold is wrong or the embedding is.

Calibrating without a mythic 0.75

Collect 50–100 questions with gold ids and 20 that should hit nothing. For each, record best cosine (and best keyword score, separately). Sort. Choose tau so that almost all golds sit above it and almost all no-hit questions sit below it. There will be overlap. That overlap is your error rate. Do not pretend a number from a vendor notebook is your tau.

When gold sits at 0.22 and no-hit pizza sits at 0.21, geometry cannot separate them. Lowering tau to 0.20 retrieves pizza. Raising it to 0.30 drops gold. Fix chunks, hybrid, or rewrite. The threshold cannot invent a margin that does not exist.

Per-collection thresholds are allowed if you log which one fired. Runbooks might separate cleanly; Slack dumps might not. Slack dumps often should not be in the same product.

Always-k UIs: show “no handbook match” instead of a sources panel full of 0.12 chunks. If you must show “related,” label it untrusted related, not citations. Citations require the tau gate.

Fused scores need their own tau. An RRF sum of 0.03 is not a cosine. Calibrate after fusion, or threshold each channel before fusion (drop weak cosine rows from the vector list, then fuse). Write down which you did.

Common mistakes

• tau = 0 in production because staging had a thin corpus.
• Showing users the raw cosine as a percent.
• Different thresholds per collection with no note in the trace, so on-call cannot tell which tau fired.
• Thresholding after packing, so you already spent the window on junk.

How agents use this

Calibrate tau on labeled pairs, not on one demo. If you change the embedding model, recalibrate. Log “retrieved 0” as a first-class outcome, not as a failure of the chat UI.

The generate step should not see junk evidence. The refuse lesson teaches the user-facing shape. This lesson is the gate: scores below tau are not sources. The agent loop is later. The gate already belongs in the retrieve function.

Print the best score next to tau in every trace. Recalibrate when the embedder changes. Empty kept-count is a first-class outcome. Do not raise k to hide a missing margin. If gold and pizza overlap in score, fix geometry upstream. Always-k is busy, not right. Cosine 0.12 is a miss even when it is top-1. The pizza query in the live box is the product you ship if tau is zero. Calibrate on labeled pairs for this model and this corpus. Recalibrate on embedder change. Log retrieved-zero. Junk in the prompt is how cash refunds happen.