Hallucinations

A hallucination is a confident statement not supported by weights-you-trust, tools, or retrieved documents. LLMs produce them because the training game is plausibility. Your job is to make unsupported claims expensive in the product: they fail a check, they do not reach the user, or they are labeled as guesswork.

You will not get a perfect lie detector from the same model. “Be truthful” in the vibe prompt is not a detector. Asking the model “are you sure?” is not detection. It will often say yes. Use process: require citations, compare claims to sources, stop on contradictions between two tool calls, require a tool row in the transcript for any claimed action.

Three families

1. Factual — fake APIs, fake paper titles, fake policy clauses, yesterday’s CEO as if it were today’s.
2. Faithfulness — the doc says 5–7 days; the model says next-day. The source was there. The quote is wrong.
3. Action — claims it called a tool it did not call.

Action hallucinations are the worst in agents. The transcript is the cure: if there is no tool message, it did not happen. Say so in the UI. Do not draw a fake tool card from assistant prose.

Citing a doc and then misquoting it is still a hallucination (faithfulness). A citation id is not a blessing on the whole paragraph.

Checks that survive contact with money

The strongest checks are mechanical: numbers, ids, and dates in the answer must appear in the observation or chunk text. Fuzzy “does this sound right?” judges are extra and can be gamed. Start with substring and schema.

Self-consistency (sample thrice, vote) reduces some errors and triples cost. Three fluent lies can still form a majority. Do not treat vote-as-truth unless you measured it on your abstain set (next lessons).

Run to execute this in your browser. Nothing is sent to a server.

Citations found tool_get_job. Amounts grounded is False because $5 is not in the sources (they say 5-7 days, not five dollars). Fake action is True because the prose claims a call but tool_roles has no "tool". Three independent checks; one pretty paragraph failed two of them. That is why you run checks in code, not with a second poem.

The dollar test is a toy (it misses $5.00 and USD 5). Production uses a tighter extractor and a schema field amounts: [...] you validate against the blob. The idea is the same: claims that look like facts must appear in observations.

Process beats a second poem

A second model that scores “is this true?” is still a guessing policy. It can help as a soft flag after mechanical checks. It cannot be the only check on a refund window. Mechanical first: schema fields, id membership, amounts as substrings, tool row present. Then, if you have budget, a judge on the leftover prose. The Eval track will treat judges as a measurement problem. Here: do not ship an agent whose only grounding is “another LLM said it looked fine.”

Contradictions: tool A says status=failed, tool B says status=ok. Do not average. Abstain or handoff (next lessons). Fluent synthesis of two statuses is a faithfulness failure with extra steps.

Action family is the one you can kill completely in code: the UI and the executor believe the transcript, not the prose. If the assistant says “I refunded you” and there is no refund tool message (and no payment API span), the UI shows “not sent.” Support should never have to guess.

Factual family without a tool is a retrieval or abstain problem. “Who is the CEO today?” with no search observation is not a writing problem. Do not fix it by raising temperature or adding “be truthful.”

Eval sets that include impossible questions

If your eval only contains answerable items, you are training a compulsive guesser. Half the questions cannot be answered from the corpus; the agent must abstain (next lessons). Score a stylish wrong number lower than a plain “I don’t know.”

Put a grounded flag on every final answer. If false, the UI shows a warning or blocks send.

Walkthrough: Maya’s bot cited tool_get_job and then wrote a refund of five dollars “as a courtesy.” The tool blob had no amount. amounts_grounded fails. The send button stays off. A human may still refund from a console. The model does not. That is the product.

Self-consistency is a cost knob, not a truth serum. If you sample three times, you pay three times, and you still need the mechanical checks on the winner. Use it offline if it helps your eval. Do not put it on the hot path until the spend cap says you can afford it.

What goes wrong

• Trusting fluent citations without opening the source.
• UI tool cards driven by assistant text.
• “Are you sure?” loops that add cost and still agree.
• LLM-as-judge as the only faithfulness check. Judges hallucinate too.
• No abstain option in the schema, so the model must invent.

How agents use this

Gate send on grounded or abstain. Humans can override. The model should not. Log which check failed (amount, action, cite). Support needs that, not “the model was having a day.”

The transcript is the action log. The sources are the fact log. Prose is a view. If you cannot point at a row, you cannot ship the sentence as fact.

Tests: an answer that claims called get_job with roles ["assistant"] must set grounded false (or fail the action check). An answer with $5 not in sources must fail. An answer that only restates status=failed from the tool blob may pass. Put those three next to the parser tests.

Watch out:“Be truthful” in the vibe prompt is not a detector. Check the transcript and the sources. Asking “are you sure?” is not a detector either.