Eval the Judge

A judge you have not measured is a second agent with no evals. Label a slice with humans (or with code where the property is machine-checkable). Measure the judge against those labels, especially on safety fails. If the judge and humans disagree, change the rubric, not just the temperature.

You need on the order of 50–100 labeled items to start, more on the harm tail than on easy FAQs. Random 50 FAQs will make any judge look good. Stratify: billing, injection, privacy, must-still-help. Double-label a subset. If humans disagree, the rubric is vague — fix the spec before you tune the model grader.

Pointwise: score one trace (pass/fail or a small integer). Pairwise: pick a winner between two traces. Pairwise is often stabler for “which summary is better” and worse for safety gates. A gate needs an absolute fail, not “this wire is nicer than that wire.” Do not replace forbid-list recall with a preference tournament.

The scary numbers

Treat “judge says fail” vs “gold says fail” as a binary classifier on harm:

Precision = TP / (TP + FP). Low precision: false alarms, ignored suite. Recall = TP / (TP + FN). Low recall: missed fails. For a safety gate, FN is the number that should stop you from shipping the judge as a gate. You can live with some FP if humans review. You cannot live with a grader that blesses wire.

If code already labels forbid hits, use that as gold for those items — do not spend humans on “did refund run?” Spend humans on the leftover the judge was hired for, and still include a few forbid hits to confirm the judge is not undoing code (it should never be allowed to override a code fail).

Walkthrough: one missed fail, one false alarm

Gold fails (True means “is a fail”): [True, True, False, False, True] — three harms, two fine.

Judge: [True, False, False, True, True] — misses the second harm (FN), alarms on a fine case (FP).

pr reports tp, fp, fn, prec, rec. You will see fn = 1. That single miss is the scary number. Precision and recall will both be imperfect. Do not pick the judge that matches the actor most often — that is agreement with the fox.

Calibrate on holdout. The slice you used to edit the rubric is not the slice you quote in a meeting. Blind model names. Shuffle order if pairwise. Store actor output and judge JSON so you can see whether a miss was “ignored the tool list” or “rubric hole.”

Stratify the 50–100. If 80 items are easy FAQs, recall on fails will look fine because there were few fails. Force the tail: injection, privacy, wire-like prose, must-help that a timid judge might fail as “unsafe.” Spend humans where code cannot label. Use code labels for forbid hits as a sanity set: the judge must not pass them if you ever show it those traces, and the runner must not ask the judge to decide them.

Pairwise vs pointwise is a measurement choice. Pairwise “which summary is better?” can be stable and still useless as a ship gate. You cannot pairwise-compare your way to “was this a wire.” Pointwise fail on harm is the gate shape. Use pairwise only for leftover ranking on in-scope summaries, and never let a pairwise win override a pointwise safety fail.

When the actor model family changes, re-measure. Family bias is real. When the rubric changes, the old precision number is void. Treat the judge like a dependency: version it, pin it, eval it, the same way you pin the actor. The FN count is the SLA.

Humans need a rubric too or the gold labels will drift. Train labelers on the same Saturday-new-hire document you give the judge. Measure human-human agreement; if it is low, stop training the judge on noise. Fifty items is a start, not a forever sample. Add labeled fails when a new harm tag appears. Quote holdout FN in the weekly slice next to agent pass rate so a pretty actor cannot hide a blind grader.

Run to execute this in your browser. Nothing is sent to a server.

What printed: a dict with tp, fp 1, fn 1, and precision/recall as floats. One false negative (missed fail) and one false positive. For a safety gate, fn is the number that should stop the judge from shipping as a gate. Empty pred and gold would make prec/rec 0.0 via the same empty-denominator honesty as pass rate.

What goes wrong if you skip this

You will promote a vendor demo rubric to a gate. It will miss injection. It will pass its cousins. It will fail long honest answers (length bias). You will “tune temperature” while the rubric never defined wire-like payouts. Humans will disagree and you will pick the model. The suite becomes two unmeasured policies in a trench coat.

How agents use this

Make “eval the judge” a recurring slice, not a launch checklist item you did once. When the actor model changes, the judge may need re-measurement (family bias). When you add a harm tag, add labeled fails first, then look at FN.

Never let the judge override a code fail. The composition is: code fail → case fail; else maybe judge. Measuring the judge on top of that composition is how you notice the judge “helpfully” passing a forbid hit because the prose was careful.