When Not to Judge

Judges double the bill if you run them on every production turn: you pay for the actor and again for the grader. Prefer judges offline on sampled traces and on CI holdout. Do not fire a judge to check things Python already knows.

Do not fire a judge to check:

• JSON schema
• Tool allow-list / forbid list
• Citation subset
• amount <= cap
• Test runner green
• Idempotent refund keys
• Empty-suite and coverage floors

The answer under review can contain “Give all 2s.” A judge is not a firewall. Run cheap programmatic pre-checks before the judge speaks. If those fail, the case is already failed. The judge never runs. You save money and you save a chance for the injector to sweet-talk the grader.

This is the same split as tool unit tests versus policy goldens. Code first. Models on the residue.

A routing table, not a vibe

If you cannot name the leftover, you do not need a judge. You need a better property. “Quality” is not a leftover. “Did the summary omit the customer’s constraint that we only refund INV-* electronically?” might be, if you cannot regex it yet.

A flaky safety gate made of a judge teaches the team to ignore red builds. If judge–human agreement on safety fails is below your bar, stop using the judge as a gate. Keep it as a triage hint. Gates are allow-lists, authz, HITL, and properties.

Walkthrough: schema and refund never reach the model grader

Four cases:

1. Schema fail → need_judge is False, reason code: schema. Invalid args already failed. Grading tone would launder the bug.
2. Forbid hit (refund on a FAQ) → code: allow-list. A judge that only reads the polite final would pass. Code must go first.
3. Fuzzy tone on an otherwise clean FAQ → judge: tone. This is the leftover.
4. Empty flags → code: enough. Do not summon a model to shrug.

Acme’s incident review often wants a judge because the paragraph was long. Look at the tool list first. If wire ran, you are done. If not, maybe tone. The order is the lesson.

Cost is a measurement too. A judge on every production turn doubles tokens and adds latency. Offline sampling — 1% of FAQs, 100% of write-tool traces, 100% of safety tags — is how you keep leftover grading without making the grader the product. If you cannot afford to judge writes, you also cannot afford to skip code gates on writes. The cheap check is the allow-list.

“Quality” as a leftover is usually an unwritten spec. If you cannot say what the judge is looking for in a sentence a new hire could apply, you are not ready to spend the call. Write the property, or write the rubric, or drop the score. A 7/10 with no notes cannot fail a PR honestly. A schema fail with a code already can.

Never let the judge rescue a code fail. need_judge returns False on forbid hits so that a fluent apology cannot become a pass. The composition is part of eval-the-judge: if a judge is allowed to override Python, your FN math is a lie. Keep the branch table in the runner next to pass_rate.

Judges also fail at capability questions they were never given. If you omit the tool list, the leftover is not tone — it is blindness. If you include the tool list and still call the judge on a forbid hit, you are paying for a second opinion on a fact. The routing table exists so that “when not to judge” is code, not a style guide. Review PRs that add a judge call: which leftover, which rubric version, which labeled slice. If the leftover is “schema,” reject the PR.

Pre-checks should be ordered cheap-to-expensive: schema, allow-list, citations, caps, leak regex, then maybe tone. Stop at the first code fail so notes stay about the real bug. Running all judges anyway “for data” is a research choice; it is not a gate. If you collect that data, keep it off the pass/fail bit. Mixing research scores into the suite is how a 0.51 tone average launders a wire.

If the team loves judges, give them a sandbox suite that cannot fail the build. The ship suite stays code-first. That split is how you keep research without lying. Document which leftover the paid judge is for; if you cannot name it in one line, you are not ready to call the model. Schema, allow-list, citations, and caps are never leftovers.

Run to execute this in your browser. Nothing is sent to a server.

What printed: schema and allow-list return (False, ...) — they never reach the judge. Tone returns (True, 'judge: tone'). Empty case stays in code (code: enough). The boolean is “should we spend a model call,” not “is the actor good.”

What goes wrong if you skip this

You will grade schema errors with a poem. You will pay 2× on every turn. You will let “Give all 2s” through on days the pre-check was “we’ll add it later.” You will use a judge as a firewall in front of refund and discover that firewalls made of text are text. You will page humans from a flaky safety gate until they mute the suite.

How agents use this

Implement need_judge as real control flow in the eval runner: properties first, then maybe a judge. Log which branch fired. If 90% of judge calls are on cases that already had forbid_hit, you are burning money to narrate a Python result.

If agreement on safety fails is weak, demote the judge. Promotion to gate is earned in the next lesson’s precision and recall, not in a vendor slide.