Ground the Critic

An ungrounded critic says “looks good.” A grounded critic gets:

• The draft answer
• The retrieved snippets (RAG track)
• The tool JSON
• A pass/fail checklist: cites ids, no extra vendors, tests green

If the critic cannot point at evidence, it is a vibe rater. Vibe raters agree with confident lies. They are worse than no critic: they add a stamp that looks like quality.

This is the same idea as observe-before-finish and as citation subset in RAG. The critic is another parser of evidence, not a cheerleader. Reflection without grounding is “are you sure?” Grounding is “show me s1.”

Evidence the user will see

Give the critic the same evidence the user will see. If the user sees snippets s1 and s2, the critic sees s1 and s2. If you hide s1 from the critic but put it in the user answer, the critic cannot catch an ignore. If you give the critic extra secret notes the user will not see, the critic will bless answers the user cannot verify.

Tool JSON is evidence. Tests are evidence. Policy tables are evidence. Private thoughts are not evidence. Yesterday’s Slack is not evidence unless it was retrieved as a snippet with an id.

Checks that mean something

Unknown cite: the answer cites s9 which is not in the snippet map. Fail.

Missing cite: the checklist required s1 and it is not in cited. Fail.

Answer ignores snippet: the cite list includes s1 but the prose does not use the facts (here, a naive word check). Fail.

Extra vendor or extra promise: if policy forbids it, fail.

These are code checks. A model critic can still help with tone, but it must receive the snippets and return structured errors, not “great job.”

Rubber stamps

“Bring shorts, it is hot” with a cite of s1 (Paris rain) is the rubber stamp if the critic only checks that a cite exists. Require that the answer is consistent with the snippet text. The live box uses a blunt “first word of snippet in answer” check so you can see a fail. Production checks will be tighter (numbers, named entities). The shape is: compare draft to evidence, do not score confidence.

A critic that only reads the draft will bless fluent weather that is wrong. That is an ungrounded critic.

Grounding does not invent sources

If there are no snippets, the critic should fail “no evidence,” not invent s1. Reflection cannot fetch. If you need evidence, the main loop must tool-call first. The critic is a gate on a draft that already had a chance to observe.

If tool JSON says timeout, a draft that answers anyway should fail. That is observe-before-finish inside the critic.

Where it sits in the loop

After a finish draft (or a write draft), run the critic before the world changes or before the user sees success. If the critic fails, retry with the errors as an observation, or handoff. Do not send the email and then critique.

Cap retries. Same error twice → stop. The critic is not a specialist society. It is a function. Multi-agent comes next track if you need specialists to talk. You do not need that to check cites.

Tool JSON, tests, and policy are evidence too

Snippets are the RAG-shaped case. Agents also draft from tool JSON: weather dicts, ticket rows, SQL counts. Give the critic that JSON, not a paraphrase. If the tool said temp_c: 12 and sky: rain, the critic can require those tokens or numbers in the answer. If you only pass the prose “it is raining,” the critic cannot catch a 28C lie that still says rain.

Tests are evidence for code drafts: a list of {name, passed} from your runner. The critic checks that required tests are green and that the draft does not claim tests it did not run. It does not run the tests by executing the draft. Policy tables are evidence for money: window_days: 30, cap: 50. A billing draft that approves 80 against cap 50 fails the critic and should have failed a guard. Defense in depth is allowed. A critic that cannot see the cap will bless 80 if the prose sounds fair.

Must-cite is a checklist the product owns, not a model whim. Support answers might must-cite the policy snippet. Weather answers might must-cite the latest observation id. If must-cite is empty and cited is empty, you are back to ungrounded. Empty evidence plus empty cites should fail “no evidence,” not pass “nothing to check.”

Do not ground the critic on private thoughts (“I decided it was hot”). Do not ground it on a Slack dump that the user will never see. Same-evidence is the fairness rule: if the user cannot see it, it does not count as a stamp. If operators need extra notes, put them in the operator trace, not in the ship gate.

Common mistakes

• Critic sees only the prose.
• Critic sees secret evidence the user will not see.
• Cite list checked, facts not checked.
• Blessing answers after tool timeout.
• Second infinite model loop.
• Using a vibe score as a ship gate.

Run to execute this in your browser. Nothing is sent to a server.

Hot-weather advice fails: the answer ignores snippet s1 (no “paris”). Unknown cite s9 fails, and s1 is missing from cited. Grounded rain plus museum passes with empty errors. Read the three dicts. Two are ok: False. That is a critic doing work.

The first-word check is crude on purpose so the box stays small. A real checker might require the temperature number. Crude and grounded still beats eloquent and blind.

How agents use this

Same idea as citation subset (RAG) and as “observe before finish.” The critic is another parser of evidence, not a cheerleader.

Wire critic errors into the trace next to parse errors. When you add eval later, these structured errors are already a gold-shaped signal. This track only needs the gate: no evidence, no stamp.