Calibration

A score is just a number you rank with. A chance is a number that should match frequencies. Calibration asks: when the model says 0.9, is it right about 90% of the time in that bin?

Agents that print “99% confident” on every tool call are uncalibrated. The number is a vibe. Do not put it in the UI. Do not skip a human gate because of it.

Ranking can still work when calibration fails. The right ticket can still come first even if every score is too sure. Use scores to sort. Use a calibrated chance only when you need a yes/no with a costed cutoff, or a sentence that claims a percent.

A cheap check

Bin examples by predicted chance. In each bin, count how often the label was actually 1. If the top bin is only half true, you do not have 99% confidence. You have a loud score.

You need enough rows per bin for the true rate to mean anything. Five rows in the top bin is a story. A few hundred is a plot. Reliability diagrams are that plot: predicted chance on one axis, observed frequency on the other. Perfect calibration is the diagonal.

Overconfidence: mean score 0.95, true rate 0.50. Underconfidence: mean score 0.60, true rate 0.90. Overconfidence is the usual sin of small models and of language models asked to “give a confidence.”

Cosine 0.82 is not “82% true” either. It is an angle. Calibrate with labels; do not invent a percent.

Run to execute this in your browser. Nothing is sent to a server.

What printed: two bins. The low bin (scores under 0.5) has five rows, a mean score around 0.27, and a true rate of 0.2 (one of those five was actually positive). The high bin has five rows, mean score near 0.9, true rate 0.6 (three of five). Those high scores are too sure. A ranking that still puts some true tickets first can be useful. A sentence that says “91% sure” is a lie.

This is ten rows. Do not ship a calibration claim from ten rows. Do ship the habit: table before percent.

What to do about it

Temperature scaling (one number on the logits, then softmax again) is often enough to flatten loud models. You fit that one number on validation chances vs labels. You do not need a new net. You need the plot — or this table — before you trust a number as a chance.

Platt scaling and isotonic regression are richer maps from score to chance. They also fit on validation. They can overfit if validation is tiny. A one-parameter temperature is the right first tool.

If you only need ranking, do not promise a percent. Show the ordered list. Show the raw score. Let a human or a later cutoff decide.

Sampling temperature on a language model is a different knob (how flat the next-token chances are). Do not confuse it with calibration temperature, even though both multiply logits. One is for generation diversity. One is for matching frequencies. Name them in the log so operators do not twist the wrong one.

When a percent is a product lie

Language models, asked to “also give a confidence,” will often emit 99. Language is not a calibrated head. A linear router can be loud too after a few epochs of cross-entropy: chances pile near 0 and 1. Loudness is not skill. Skill is ranking the right tickets first and, separately, matching frequencies if you need a gate.

A cheap reliability table is enough to kill a bad UI: ten bins, or even two as in the live box. If the top bin is 0.9 vs 0.6 true, you may still use the score to sort tool docs. You may not write “I’m 90% sure this should run shell.” For shell, you want a high-precision operating point on a calibrated chance, or you do not use chance at all — you use an allow-list.

Expected calibration error is a fancy name for “average, over bins, of |mean score − true rate|, weighted by how many rows sat in the bin.” You do not need the acronym to compute the table. You need labels. Unlabeled “confidence” is decoration.

Recalibrate when the mix drifts. A temperature fitted in January can be wrong in September if the tool mix changed. Calibration is not a one-time baptism. It is a plot you redraw on a fresh slice, then you freeze a new temperature or you stop printing percents.

Common mistakes

• UI copy that says “I’m 99% sure” from an uncalibrated logit.
• Calibrating on test.
• Calibrating on 20 rows.
• Treating retrieval cosine as a probability.
• Using ranking success as proof of calibration.

How agents use this

Calibration decides when to escalate. An uncalibrated small model that is “sure” it should run shell is how you skip the human gate. Compare: use scores only to rank candidates; use a calibrated chance only when you need a yes/no with a costed cutoff.

Router abstain (“if top p < 0.6, call the LLM”) assumes p means something. If p is loud, you will never abstain. If p is timid, you will always abstain. Measure. Then freeze the cutoff on a slice with natural class rates.

Watch out:Cosine 0.82 is not “82% true” either. It is an angle. Calibrate with labels; do not invent a percent.