Self-Check

A self-check is a second pass with a mechanical rubric, not a pep talk. After the draft exists, you verify claims against the context you actually sent:

• Every dollar amount in the answer appears in context
• Every doc_ id is in the allowlist
• JSON still matches the contract
• No tool name that is not in the list
• No “today’s CEO” that was not in the observation

It can be the same model with a different prompt (“here is the draft and the sources; return pass or fail”), or it can be code with no model. Prefer code for numbers, ids, enums, and extra keys. The LLM hallucinations lesson already said substring checks beat “does this sound right?”

Do not ask the same completion “are you sure?” It will often say yes. Confidence is not overlap with sources.

The check must see the sources

A checker that only sees the draft is grading style. A checker that sees the draft and the context can fail $5 that never appeared. Put sources in the check prompt. Put the same sources in the code checker. If you dropped the sources to save tokens, you are not checking — you are rereading.

Eval the checker. A checker that always says pass is a mascot. Include cases where the draft is fluent and wrong. Gate send on empty fails. If the checker is a model, it needs its own tiny golden set, or it will rubber-stamp.

Self-consistency is a cousin, not a substitute

Sample thrice, vote. It triples cost. Use it on high-stakes questions after you already have context, not on every router call. Three fluent lies can still win. Voting does not create a fact that was missing from the window.

Self-check with code is cheap enough for the hot path. Self-check with a second model is a product choice for leftover prose (tone, contradiction in sentences). Do not use a second model to re-decide a refund. That is a policy tool and a human.

Run to execute this in your browser. Nothing is sent to a server.

What printed: grounded is ok — tool_get_job is allowed and 5-7 appears in context. Bad fails on amount $5 and id doc_99. “Are you sure?” would not have caught them. Mechanical overlap with sources would.

This checker is a toy: it only looks at $N and doc_ tokens. Production checkers also fail extra JSON keys, unknown tool names, and empty answers. Same idea. Longer rubric.

What a check is not

• A second adjectives paragraph (“double-check your work”)
• Sampling until you like the vibe
• Asking the model to rate its confidence from 1–10
• Deleting the spec so the checker can “be open minded”

Those are interior decorating. The check is a function that returns a list of fails.

Walkthrough: “are you sure?” on five dollars

The draft says refund $5 today and cites doc_99. Context has no dollar amounts and no doc_99. Asking the same model “are you sure?” yields “yes, confident.” The code checker returns two fails. You do not send. A second-model checker that does not receive the sources also says pass — it is grading tone. The check prompt must include the same context blob you sent the first time.

Self-consistency (vote of three) can still agree on $5. Voting is not a source. Use it rarely, after you already have context, never on the router.

What goes wrong if you skip this

Fluent money leaves the building. Checkers that always pass become mascots. You skip code checks because a model “looks careful.” Ids that never appeared in the window get cited. This lesson is the prompt-layer habit that makes a faithfulness score possible later.

A self-check is a mechanical rubric after the draft: amounts and ids appear in the context you sent; JSON still matches the contract; tool names are on the list. Prefer code for numbers and ids. A second model may grade leftover prose, and only if it sees the sources. “Are you sure?” is not a check. Self-consistency triples cost and can still vote for a lie. Eval the checker with fluent-wrong drafts. Gate send on empty fails.

Common mistakes

Put the checker on the send path, not in a weekly notebook. Empty fail list means send. Non-empty means retry once with the fail reasons, then handoff. Do not let the model negotiate with the checker. Numbers and ids are not a debate. If the draft cites doc_99 and doc_99 was never in context, the answer does not ship. That is the whole rubric.

How agents use this

Put check() in the hot path for ids and money. Use a second model only for leftover prose, and eval the checker. Gate send on empty fails. The check prompt must see the sources, not only the draft. Vibes are not a detector. Confidence scores are not overlap. A fluent $5 that never appeared in context is a fail even if the model swears. Prefer code. Keep the rubric boring on purpose so it can fail a pretty paragraph.

Note:A checker that always says pass is a mascot. Include fluent-wrong drafts in its eval.