Debate and a Judge

Debate is a pattern where two agents produce competing answers, optionally attack each other, and a judge selects or merges. It can catch confident errors that a single sample misses. It can also produce longer wrong answers at 3× cost.

This is not the planner–worker–critic trio. The critic had a rubric and no writes. Debate adds an opponent whose job is to find faults or concede, plus a judge that sees both packets and the evidence. It is not a group chat with three hats. It is a subgraph you turn on when single-sample workers fail in a known way and you have something to ground the judge.

Do not debate “what is 2+2” or “format this JSON.” That is theatre. Do not debate every FAQ. That is the cost explosion in the failure-modes lesson. Do not use debate to replace unit tests. Tests are cheaper judges.

Use debate when:

• The question is high stakes (money, safety, legal, medical-adjacent policy you are allowed to automate at all)
• You have a judge that is grounded (tests, citations, a rubric over source ids)
• Single-sample evals already fail in a known way (systematic omission of a clause, confident extra promises)

The next lesson is how to ground the judge. This lesson is the shape, the round cap, and when not to bother.

The three roles in a debate subgraph

1. Proposer answers with evidence ids, not with a vibe. Same interface-as-data rule: a packet {answer, cites}.
2. Opponent must find faults or concede — structured: {faults, concede}. “I disagree” with no fault string is invalid. Always-attack opponents become nitpick machines; the judge must be allowed to accept a concession.
3. Judge sees both plus the evidence, not the whole internet again, not the speaker names. Prefer programmatic judging (did they cite real ids? did the number appear in the source?) over “which vibe was smarter?”

Cap rounds. Often one attack is enough. A second round doubles tokens and trains everyone to hedge. If the judge is uncertain, handoff to a human. A mushy merge (“both have a point, refund half”) is how committees ship hedged wrong answers.

Nobody in the debate triangle holds refund. Apply stays on the billing subgraph after the judge, same as sequential-subgraph. Debate is a node, not a new mesh.

Walkthrough: refund policy, proposer lies about a year

Source of truth (artifact on the board): “Refunds take 5-7 business days after approval. No cash refunds after 30 days.”

Question: how do refunds work?

Proposer answers that refunds take 5-7 days even a year later, and cites 5-7. The delay fragment is in the source. The “year later” promise is not. This is the extra-promise failure single-sample models love.

Opponent checks: if the answer says year later and the source has 30 days, that is a fault. If 5-7 were missing from the source, that would be another fault. Here faults is non-empty, concede is false.

Judge: if opponent listed faults, winner is opponent, and the final is a sentence that includes both the delay and the 30-day cash limit — in this toy, written by code from the source, not by another free-form model. If no faults and the proposer’s 5-7 appears in source, proposer wins. Else human.

The judge here is code plus the source string. When you later swap in an LLM judge, keep this evidence in the prompt and eval the judge (evals track). Blind names. Cap rounds to one unless a golden proves you need two.

Run to execute this in your browser. Nothing is sent to a server.

What printed: proposer includes the year-later lie and a 5-7 cite. Opponent lists the 30-day fault and does not concede. Judge winner is opponent, final mentions 5-7 and 30 days, faults echoed. The user-facing product should store both answers plus the verdict. If you only store the winner, you cannot audit the loss when the opponent was wrong next time.

Change the proposer answer to drop “year later” and keep 5-7. Opponent should concede (no faults). Judge should pick proposer. That is the happy path. Do not skip writing it as a golden.

Cost and when to refuse debate

Price: roughly 3× a single worker call, more with extra rounds, plus judge tokens. Tag tickets high_stakes before you enqueue debate. Untagged FAQs go to one worker or a sequential policy node.

If the opponent is rewarded for always attacking, you get nitpicks (next lesson). If the judge is the same model and prompt as the proposer, you get a mirror. If you launch debate because a slide said “team of agents,” you will buy longer wrong answers.

Debate is not ReAct. ReAct is one policy, tools, observations. Debate is two answers and a judge over evidence. Do not wrap debate in a tool-calling loop that lets the opponent edit. No writes in the triangle.

How agents use this

Run debate only on tag=high_stakes tickets. Blind the judge to speaker names. Store both answers — if you only store the winner, you cannot audit the loss.

Put debate as a node on the subgraph that already has HITL for money. Round cap in config, default 1. Uncertain judge → assign: human, not a coin flip.

Eval the opponent for missed faults on fixtures that contain a known lie, and for extra faults on clean answers (nitpick rate). Eval the judge against programmatic labels. A debate system you cannot eval is a talk show.

Baseline still wins if debate does not beat single-sample on the high-stakes slice. Deleting debate from FAQs is a ship. The next lesson makes the judge refuse a chorus without citations.