Properties Beat String Equality

Agents paraphrase. That is not cheating. “Refunds usually take 5-7 business days” and “You should see the refund in 5-7 business days” are both faithful to the fixture. Byte-for-byte equality with last week’s model sample is a flake generator. The model vendor will ship a comma. Your suite will go red. Someone will disable the test. You will have taught the team that goldens are annoying, not that the policy drifted.

Use exact match for structured fields: order ids, SKUs, invoice numbers, enum statuses, money in integer cents. Use properties for prose and for anything that can be reworded without changing the world.

A property is a predicate over the trace and the output: contains a canonical fact, JSON schema valid, tool args match a bound, citation ids are a subset of retrieved ids, refuse flag true, forbid list empty. The golden row stores the predicate, not the essay.

Which check belongs where

If the expected answer is pasted into the prompt as an example, you are measuring memorization of the suite. Keep a holdout the product engineers do not prompt with. Hidden also means the world fixture is not in the system prompt: the fake database the tools see. If the prompt already contains “5-7 business days,” must_contain: 5-7 does not prove search happened. Pair facts with must_call.

Walkthrough: paraphrase vs “soon” plus a fake cite

The case wants: fact 5-7, exact order_id 99, citations subset of opened kb-44.

Paraphrase. Final: “Refunds usually take 5-7 business days.” Order id 99. Citations ['kb-44']. props_ok returns an empty fail list. The wording moved. The properties held.

Bad. Final: “soon.” Order id still 99 (structured luck). Citations ['evil'] — an id that was never retrieved. Fails: fact, and cite evil. Exact-matching the whole essay would have failed the good paraphrase too. Properties fail only the actual bugs: missing window, invented citation.

A third failure mode: the essay contains 5-7, the id is right, the cite is right, and refund still ran. That is not this function’s job — it is forbid on the golden. Properties are a family. You compose them. You do not replace side-effect checks with a substring.

Canonical facts need a normalization policy. “5-7” should match “5–7” only if you normalize dashes on purpose. “five to seven” might be allowed if legal says so — that is a second predicate, not a fuzzy feeling. Do not reach for a judge to decide whether “about a week” counts. Either the spec accepts it (then write a small normalizer) or it does not (then fail). Ambiguous facts mean the spec is unfinished.

Structured exact match is how you stop “INV-17” becoming “invoice 17” in a ledger write. The customer-facing sentence can paraphrase; the tool arg must be the id. Properties therefore sit on two layers: the final text, and the parsed tool args / state. A citation subset is the RAG cousin of that idea: ids in the answer must be in the retrieved set. Invented kb-evil is a grounding fail even when 5-7 is present.

Holdout exists so prompt authors cannot farm properties. If every must_contain is also an example in the system prompt, you have built a cloze test. Rotate holdout. Keep the world fixture out of the instructions. Pair must_contain with must_call so a memorized window without search_kb still fails.

Substring facts can be gamed: an agent that lists every number it knows will hit 5-7 by accident. Prefer facts that are canonical and rare in the fixture, or require the fact and a citation id, and the tool. For money, never substring-match formatted currency if cents exist as an integer field. Properties should make the cheap cheat fail. If a cheat still works, tighten the predicate, do not hire a judge to “see if it felt grounded.”

Run to execute this in your browser. Nothing is sent to a server.

What printed: [] for the paraphrase (clean). The second print is ['fact', 'cite evil']. Paraphrase with the fact and the real cite is clean. “soon” plus a fake cite fails. The empty list is a pass. Nonempty is reasons — print them next to the golden id.

What goes wrong if you skip this

You will exact-match essays and drown in flakes. You will then loosen to “the model said something,” which cannot fail. You will miss invented citations because the paragraph sounded grounded. You will accept “about a week” when finance needed the 5-7 window for legal copy. You will fight the vendor’s tokenizer instead of the policy.

Skip properties and judges get hired to do substring checks. That is how bills double and gates get fuzzy. Python can fail these cases. Python should.

How agents use this

Write a small library of predicates: contains_fact, exact_field, citations_subset, tool_required, args_ok. Golden rows name the predicates. Runners apply them to traces. When a new model paraphrases more freely, the suite should stay stable. When a new model invents doc ids, the suite should go red.

RAG-style citation subsets belong here even if retrieval was a tool: the ids in the answer must come from the observation, not from a prior. Billing ids stay exact. Mix the two checks on the same row.