Accuracy, Precision, Recall, F1

An agent eval is often a yes/no on each ticket: did we pass? did we retrieve the gold chunk? did we call the forbidden tool?

Four counts:

Accuracy is (TP + TN) / all. It lies when most tickets are easy no. A retriever that always says “not found” can look accurate if gold is rare.

Precision is TP / (TP + FP): of the times you said yes, how often were you right? High precision, low recall means you are picky.

Recall is TP / (TP + FN): of the real yes-es, how many did you catch? High recall, low precision means you grab everything.

F1 is the harmonic mean of precision and recall: 2 P R / (P + R). It is 0 if either is 0. Use it when you care about both.

A cutoff on a score (cosine, chance) moves you along that tradeoff. Raise the cutoff: fewer yes-es, usually higher precision, lower recall.

A wrong picture

A wrong picture is: “accuracy is the score to optimize.” Always-no on a set with 80 easy negatives and 20 golds: accuracy 80%, recall zero. Do not ship it. If a metric always goes up when you say yes more often, you are looking at recall (or a cousin), not at quality. Pair it with precision.

Another wrong picture is a percent without counts. Five tickets can print 100% F1. Report TP/FP/TN/FN. Small n cannot tell 0.80 from 0.88 (variance of a proportion).

A third: treating F1 as a physical law. F1 balances P and R equally. A refund bot may want high recall (missed gold chunks hurt). A “delete this account” tool may want high precision (false yeses hurt). The cutoff you pick is a product decision, not a math law.

Division by zero: no predicted yeses means precision is undefined; no real yeses means recall is undefined. In code, return 0.0 and log that the denominator was 0. Do not print 100%.

The formula in words

Accuracy: how many decisions matched the label, over all decisions.

Precision: among predicted yes, fraction that were truly yes.

Recall: among truly yes, fraction you predicted yes.

F1: 2PR/(P+R) — zero if either is zero.

Tiny numeric. Balanced: TP=40, FP=10, TN=40, FN=10. All=100. Acc=0.80, P=40/50=0.80, R=40/50=0.80, F1=0.80.

Always no: TP=0, FP=0, TN=80, FN=20. Acc=0.80, P=0 (no predicted yes), R=0, F1=0.

Always yes: TP=20, FP=80, TN=0, FN=0. Acc=0.20, P=0.20, R=1.0, F1 about 0.33.

Sweeping a cosine cutoff on a handful of (score, gold) rows moves these four counts. That is how you pick 0.35 vs 0.5 vs 0.8.

Moving parts

Report the four counts. A percent without TP/FP/TN/FN is a poster.

A second walkthrough (ten tickets)

Ten retrieve decisions. Three golds. Cutoff 0.50. Predicted yes on four rows; two of those are gold.

Counts: TP=2, FP=2, FN=1, TN=5.

• Accuracy = (2+5)/10 = 0.70
• Precision = 2/(2+2) = 0.50
• Recall = 2/(2+1) ≈ 0.667
• F1 = 2 0.50 0.667 / (0.50 + 0.667) ≈ 0.667 / 1.167 ≈ 0.571

Raise the cutoff to 0.80: maybe only one predicted yes, that one gold. TP=1, FP=0, FN=2, TN=5. Precision 1.0, recall 1/3 ≈ 0.333, F1 0.50. Picky. Lower to 0.20: almost always yes, recall climbs, precision falls. The cutoff is a product choice, not a math law.

Always-no on this set: TP=0, FN=3, TN=7, FP=0. Accuracy 0.70, recall 0, F1 0. Same accuracy as the 0.50 cutoff, useless recall.

Division by zero: no predicted yes → precision undefined. No real yes → recall undefined. Return 0.0 and log that the denominator was 0. Do not print 100%.

A Friday ticket

Friday standup: “retriever accuracy is 94%.” One hundred tickets, six gold chunks. The policy was always-no (cutoff 0.99). TN=94, FN=6, TP=0. Accuracy 0.94, recall 0. They had shipped a polite mute.

The fix was to print TP/FP/TN/FN on the dashboard and to track recall@k on (question, gold chunk) before judging the generator. Accuracy stayed on the chart as a warning label, not as the score to maximize.

Try the counts and a cutoff

Run to execute this in your browser. Nothing is sent to a server.

Balanced prints about [0.8, 0.8, 0.8, 0.8]. Always no: acc 0.8, precision 0, recall 0, F1 0. Always yes: acc 0.2, precision 0.2, recall 1, F1 about 0.333. “Always no” has high accuracy here and zero recall. Do not ship it.

On the six rows, cutoff 0.3 predicts yes for 0.91, 0.72, 0.40, 0.33: TP=3, FP=1, TN=2, FN=0, F1 high. Cutoff 0.5 drops 0.40 and 0.33: you miss a gold at 0.33 (FN), you drop the false 0.40 (good). Cutoff 0.8 keeps only 0.91: very picky, misses 0.72 gold too. Sweeping the cutoff shows the tradeoff on this tiny set. On a real set you plot precision vs recall and pick a point that matches the product.

Evaluate the retriever with recall@k on (question, gold chunk) before you judge the generator. Evaluate a guardrail with precision (false blocks annoy users) and recall (missed blocks leak).

What goes wrong

• Accuracy on rare yes: always-no looks great. Print counts. Pair with recall.
• Percent without n: five tickets at 100% F1. Variance of a proportion cannot tell 0.80 from 0.88 on small n.
• F1 as a law: F1 weighs P and R equally. Refund recall may matter more than F1. Delete-account precision may matter more than F1. Pick the cutoff for the product.
• Cutoff ties: score == cutoff. Decide >= vs > and keep it. Off-by-one on a handful of rows moves F1 around.
• Undefined P or R: denominator 0. Return 0.0 and log it. Never print 100% from an empty yes set.

Production logs: the four counts, P, R, F1, cutoff, and n. Assert denominators, and that a fixture set with a known confusion matrix reproduces those counts. At T>0, average F1 over seeds or decode greedy for a stable eval.

How agents use this

The cutoff you pick is a product decision, not a math law. A refund bot may want high recall; a delete-account tool may want high precision.

• Tokens: these scores are not token counts. They measure decisions. Do not confuse F1 with perplexity. A fluent wrong refusal can have low CE and terrible recall.
• Ranking: recall@k is recall where “yes” means “gold in the k neighbors.” Precision@k is “of the k, how many were useful” if you have usefulness labels. Log scores next to the cutoff.
• Loss: training CE does not maximize F1. If you need F1, sweep a cutoff on a held-out set after you have scores. Nested: no path from F1 into backprop unless you built one.
• Sampling: at T>0, pass/fail is random. Average F1 over seeds, or decode greedy for a stable eval. Report n.

If a metric always goes up when you say yes more often, you are looking at recall (or a cousin), not at quality. Pair it with precision. Print counts.

Tip:If a metric always goes up when you say yes more often, you are looking at recall (or a cousin), not at quality. Pair it with precision.