Evals & Safety

Why Evals Exist

An eval is a repeatable score over tools, state, and the final answer. Demos, thumbs-up, and live dashboards are not a test suite.

Measure Side Effects

Score what the world did: tools that ran, tools that were forbidden, money or mail that moved, and refusals that should have happened.

Offline vs Online

Offline evals rerun frozen fixtures in CI. Online evals sample live traces. Either one alone is a blind spot.

The Unit Is a Trace

Two agents can print the same sentence. One searched the knowledge base. The other invented a policy. Score the path — the trace is the exam paper.

Write the Eval First

Ten well-labeled cases beat a thousand unlabeled transcripts. If a behavior is not in the suite, the next prompt tweak will delete it.

Golden Sets

A golden set is a versioned table of inputs and expected properties — not screenshots in Slack. Hidden checks, tags, and weights are the threat model.

Pass Rate Is a Fraction

Pass rate is hits divided by n. An empty suite is 0, not 100%. Print the fraction so humans see 2/3, not only 0.667.

Properties Beat String Equality

Agents paraphrase. Exact match on the essay flakes. Check facts, schemas, tool predicates, and citation subsets — exact match only on structured ids.

Quarantine Flaky Rows

A known-bad fixture should not sink pass rate until you fix it. Tag quarantine with an owner; never delete the evidence or leave it silently red.

Coverage by Tag

A 92% pass rate with no injection cases is a vanity metric. Count safety, billing, and injection first — then pass rate.

Unit Tests for Tools

Most agent bugs are tool bugs. Test schemas, authz, timeouts, idempotency, and fixtures without the model — then eval whether the policy called the right name.

LLM-as-Judge

When Python cannot score a fuzzy quality, a judge model grades a trace against a rubric. Then you eval the judge — especially on safety fails.

When Not to Judge

If Python can fail the case, Python should. A judge is not a firewall, it doubles the bill, and it can be talked into all 2s.

Eval the Judge

Label 50–100 items with humans. Measure precision and recall on safety fails. A judge that misses forbidden harm is décor, not a gate.

Traces and Replay

Log thoughts, tools, and observations with digests. Replay is how you debug, eval, and prove what happened — traces are exam papers, not just debug movies.

Safety and Harms

Agent safety is tools plus refusals, not rudeness. Measure fraud, privacy, unsafe calls, and the cases that must still be helped.

Forbidden Tools Are a Gate

A forbidden-tool hit fails the case even if the FAQ fact is present. The allow-list is the exam; prose is not an appeal.

Score Secret Leaks

Score the user-visible answer and the stored trace excerpt for keys, cookies, and other users’ rows. Redact first, then eval that redaction held.

Prompt Injection, Deep Cut

Untrusted text in tools, pages, and memory can steal the policy. Measure that data never becomes a tool call: delimit, allow-list, and goldens with injected docs.

Delimit, Then Allow-List

Stack defenses you can measure: data fences, unknown names cannot parse, observations never join the system prompt, HITL for irreversible tools.

Alignment Basics

Alignment for product agents is a spec you can fail: allowed tools, refusals, honesty, scope — plus enforcement and evals, not a personality paragraph.

Docs Win (On Purpose)

Put docs-vs-prior in the assembler as a switch. If sources exist, do not contradict them. Eval that the prior did not sneak back — including empty-source honesty.

Red-Team Hour Becomes Goldens

A harm taxonomy that never meets an attacker is a slide. Capture wire, cross-tenant, injection, and must-help wins as fixtures the same week.

When the Eval Lies

Empty 100%, drowned injection, stale gold, self-judging actors, uncapped quarantine. Next track is production: CI, traces, and kill switches that use honest numbers.