Ground the Judge

If the opponent is rewarded for always attacking, you get nitpicks. If the judge is the same model as the proposer with the same prompt, you get a mirror. If two models agree with no citations, you have a chorus, not a fact. The judge’s job is to fail closed or handoff — not to ship the duet.

Grounding means the judge sees the same snippets the user will see (source ids on the blackboard), checks that cited ids exist, and checks that the answer actually uses those snippets. This is the Agents grounded critic and the RAG citation idea, lifted to a debate node. Debate without this check is a talk show.

Blind the judge to names (“Agent A is our smartest model”) or you will measure branding. Shuffle order; position bias is real: many models pick the first packet. A judge that always splits the difference trains a committee to hedge. Hedged wrong answers still ship. Prefer handoff over a mushy merge.

What the judge must check

The toy below uses a crude “first word of the snippet in the answer” check so you can run it in the browser. Production should use better overlap or a structured field the proposer must copy (delay_days: [5, 7]). Crude is still better than “which essay sounded senior.”

Unknown cite. Proposer cites s9 that is not in the map. Fail closed. Do not let the judge “believe” s9 because the prose is confident.

Ignore cite. Proposer cites s1 (“Paris rain 12C”) then says bring shorts, it is hot. The cite list looks busy. The answer contradicts the snippet. Fail. This is how models launder hallucinations: they attach a real id to a false sentence.

Good. Answer contains rain and the 30-day cash rule, cites s1 and s2, both real. Pass.

Agreement of two models on “bring shorts” with empty cites must not pass just because they match. The quiz is that case. Chorus without ids is still a guess.

Walkthrough: weather plus refund rule

Sources: s1 Paris rain 12C, s2 no cash after 30 days. Three judge calls:

1. Answer restates both facts, cites both ids → ok, winner proposer.
2. Shorts in the heat, cites s1 → errors include ignores s1 (first word “Paris” not in “Bring shorts…”). Fail, winner human.
3. Mentions rain but cites s9 → unknown cite. Fail.

Winner human on fail is a policy: do not let a broken judge silently pick the opponent’s vibe either. A real product might pick opponent if opponent faults are grounded. This toy keeps the lesson small: evidence first, else human.

Run to execute this in your browser. Nothing is sent to a server.

What printed: first call ok, winner proposer. Second call not ok, ignores s1, winner human. Third call unknown cite s9, winner human. Hot-weather advice ignores s1. Unknown cite s9 fails. Grounded rain plus cash rule passes.

The unused text argument is a reminder you can pass the user question without letting it override sources. Do not judge the question’s tone. Judge the answer against ids.

Add a fourth mental case: judge("Paris rain 12C.", [], SRCS, "") → no cites. Two models could both emit that sentence from memory. Still fail if the rubric requires cites. That is the chorus.

Blinding, order, and merge policy

Strip names. Call them Packet 0 and Packet 1. Randomize which packet is proposer. Log the permutation so you can debug, but do not show it to the model judge. If you use programmatic judges only, bias is smaller — still shuffle if any LLM remains in the loop.

Do not average two numbers from two models and call it truth. Do not pick the longer answer. Length is not evidence.

If both packets fail grounding, human. If one passes, you may return that packet’s answer as data, then still run HITL on money. Passing the judge is not apply.

Position bias is not a footnote. If you always put the proposer first, you will ship proposer-shaped errors and call it “the judge prefers our best model.” Shuffle. Log the permutation for debugging. Publish judge accuracy by position in evals. If packet-0 wins 80% of the time, you are measuring order, not policy.

Nitpicks: opponent faults that do not cite a source id are dropped or sent to a nitpick counter. Reward opponents for grounded faults in evals, not for word count of complaints. An opponent that always attacks will look busy and waste the cap. An opponent that concedes on clean answers is doing the job.

How agents use this

Same idea as the Agents grounded critic and RAG citations. Debate without this check is a talk show.

Wire the judge to the blackboard’s artifact map. If the id is missing, the researcher failed, not the judge — fail closed, reassign researcher or human, do not invent s1.

Eval the judge as its own product: goldens where the answer is true but uncited (must fail), true and cited (must pass), false and cited (must fail), chorus with no cites (must fail). If your judge pass rate is 100% on demos and 0% on those goldens, you shipped rhetoric.

Never let the judge write. Never let agreement skip the cite check. When you add swarms next, reduce will also drop children that fail schema — same fail-closed family, different fan-out.