Traces and Replay
Log thoughts, tools, and observations with digests. Replay is how you debug, eval, and prove what happened — traces are exam papers, not just debug movies.
A trace is the ordered list of events: assembled context hash, model raw text, parsed decision, tool args, tool result (truncated), state, cost. The unit-of-eval lesson said the path is the exam paper. This lesson is how you store that paper so you can replay it. If you cannot replay a run, you cannot improve a run, you cannot write a golden from an incident, and you cannot prove what the tool returned when legal asks.
Recorded observations — do not call live tools; inject the log. Use this to test a new parser or assembler against yesterday’s world. The money did not move twice. The PDF is still the same bytes.
Live tools, frozen policy — same prompt / policy version, new world. That is a regression against reality: finance changed the window, the KB moved. Digests will disagree. That disagreement is a real signal, not a flake, if the policy was pinned.
Fork — change one step from an incident and resimulate. That is how a page becomes a golden: take the injected observation, assert wire must not run.
Do not store secrets in traces (the production track will go deep on exporters). Redact keys, cookies, and raw payment fields before the log is a log. This lesson still needs a digest of the result so you can compare worlds without pasting the whole blob into every dashboard.
Three replay modes, three failure tables
| Mode | What is frozen | What moves | Use |
|---|---|---|---|
| Recorded observations | Tool results | Policy / parser | Deterministic goldens |
| Frozen policy, live tools | Prompt version | The world | Drift vs production |
| Fork | Most of the incident | One step you edit | New golden from a page |
Inject recorded observations. Do not call live Stripe to grade yesterday.
Replay the exam paper| If you skip | You get |
|---|---|
| Digests | “It looked the same” arguments |
| Schema parity with tests | Cannot replay incidents into CI |
| Excerpts plus hash | Either huge PII-filled logs or no proof |
| Sampling policy | Only cheap FAQs stored; writes vanish |
Hashing job_id is how you cannot debug. Hashing a user id may be required; do not hash the job id. Truncate excerpts; keep result_digest over the canonical JSON.
Walkthrough: same world vs status flipped
You record: an LLM event, a get_job tool with job_id 17 and result status: failed, then stop. digest is a short sha256 of canonical JSON (sorted keys) so key order does not flake.
Replay same world. The fake get_job still returns failed. replay returns no mismatches. The exam paper still matches the classroom.
Replay after world change. get_job now returns status: ok. Digests disagree. mismatches names the tool, the wanted digest, the new digest. That is a real regression of the world, or a bug in the fake, but it is not a vibe. You now know the recorded observation is stale relative to this implementation — update the golden, or catch a tool that started lying.
Fork idea (in your head, not in the toy): take the injected PDF observation from an incident, keep it frozen, run the new allow-list policy, expect handoff instead of wire. Recorded observation plus a property: that is a golden.
The digest is identity, not compression. Eight hex chars are a demo; use the full hash in product. Canonical JSON with sorted keys avoids flakes from key order. Do not hash the pretty-printed essay; hash the structured result. If you redact after hashing, two worlds that differed only by a secret will look different forever and you will store the secret in the hash’s preimage — redact then digest.
Sampling is an eval policy. 1% of FAQ jobs, 100% of write tools, 100% of HITL, 100% of safety tags. If you sample uniformly, you will have a museum of “how long do refunds take?” and no exam papers for the refund that actually ran. Schema must match the test runner or forking an incident into a golden is a rewrite. Store trace_id on the user-visible error so the first question in review is answerable.
Replay mismatches are not automatically “agent bugs.” They can be fixture drift (finance moved), tool bugs (status flipped wrongly), or policy bugs (parser changed). The digest tells you inequality. The golden properties tell you whether that inequality is allowed. Keep both.
Recorded-observation replay is the only way to test a parser change without moving money. Live-tool replay is how you notice the KB moved. If you only have one mode, you will call fixture drift a flake or call a parser bug “the world changed.” Store enough of the context hash to know which prompt version produced the paper. Redact secrets before the excerpt is a string you might paste into Slack. The production track will talk exporters; your eval already needs a paper you can share.
Run to execute this in your browser. Nothing is sent to a server.
What printed: events ['llm', 'tool', 'stop']. Same world: [] mismatches. After the status flips to ok: a mismatch dict with name: get_job and two different short hex digests. That is a real regression, not a vibe. The excerpt is truncated JSON; the digest is the identity of the result.
What goes wrong if you skip this
Incidents become Slack archaeology. New parsers cannot be tested without calling live refund. Goldens and production traces diverge in schema (“prettier” test events). You cannot prove what the KB said. You store secrets and then cannot share traces with the people who must grade them. You sample only cheap jobs and never keep write-tool runs.
How agents use this
When an incident happens, the first question is “show me the trace id.” Sample cheap jobs; keep 100% of traces that hit write tools, HITL, or safety tags (after redaction). Production turns this into queues; this lesson is the schema and the digest/replay check.
CI should replay recorded observations for parser and policy changes. Use live-tool replay sparsely as an online drift check, not as every PR. Fork incidents into goldens the same week (alignment part). The exam paper is how measurement becomes a loop instead of a museum.
Check your understanding