The Unit Is a Trace

You do not only score the final sentence. You score the trace: the ordered list of events that made that sentence possible. Tools, arguments, observations, state transitions, and the final text are one object. That object is the unit of evaluation.

Two agents can print “Refunds take 5-7 days.” One called search_kb and quoted a fixture document. The other invented the policy from a prior. Those are not equal products. If your eval format is “input string, output string,” you have rebuilt a chatbot benchmark and put an agent inside it.

In the Agents track, a trace is a debug movie you watch when a job goes weird. Here the movie is the exam paper. You grade it. You store it. You replay it. You do not throw it away after the demo.

What belongs in the exam paper

You do not need OpenTelemetry to start. You need a list of dicts with a kind (or type) you can filter. Later lessons add hashes and replay. This lesson is the idea: same prose, different traces, different scores.

Scoring only the final also hides order. Searching after you already refunded is not the same as searching first. A trace is ordered. A bag of tool names is a weaker exam, but it is still better than the sentence alone. Start with names; add args and order when the case needs them.

Walkthrough: grounded vs invented

Same user question: “How long do refunds take?” Same final text. Different traces.

Grounded. kind: tool, tool: search_kb, then kind: final with the 5-7 sentence. tools_of returns ['search_kb'].

Invented. Only kind: final with the same sentence. tools_of returns [].

same_prose is True. equal as agents is False. The eval that uses same_prose as pass/fail is grading a chatbot. The eval that uses tools_of is grading an agent.

Now add a third trace: search, then refund, then the same sentence. Prose still matches. Tools do not. Side-effect scoring from the previous lesson fails it. The unit is still the trace — you just applied a second predicate to the same paper.

When an incident happens, “paste the last message” is the wrong ask. “Give me the trace id” is the right one. If you cannot reconstruct tools and observations, you cannot write a golden from the page, and you cannot tell whether a fix worked.

Arguments are part of the paper. refund with invoice_id 4412 is not refund with Ada’s id on Bea’s session. A name-only bag would pass both or fail both together. Once you care about authz and caps, predicates on args are the eval: id matches the fixture user, amount does not exceed the cap, query strings are not empty. Order matters when the spec says search then answer: a refund before search_kb is not “the tools were present.” It is a different policy.

Schema parity is a measurement issue, not a taste issue. If production events use type: tool and tests use kind: tool, every replay adapter you write is a place the exam can silently skip questions. Pick one schema. Grade that. The later digest/replay lesson will hash results; it cannot hash a field you never stored.

Cost and steps belong on the paper too. Two traces can both search once and print 5-7; a third can search forty times and then print 5-7. Task-success-only will call them equal. They are not equal products. You do not need a full finance model in this track. You need n_tools and tokens on the event list so a later dashboard can slice them.

Observations are part of the exam, not only tools. If search_kb returned an empty list and the model still said 5-7, that is invented policy with a tool-shaped alibi. A name-only check would pass. Storing a truncated excerpt plus a digest (next part of the track) lets a property say: the fact in the final must appear in an observation, or the case is ungrounded. You do not need to re-teach retrieval. You need the bytes on the paper.

Thoughts or raw model text belong on the paper for debugging, but grading thoughts is usually a flake. Grade parsed decisions and tools. If the parser dropped a call that the raw text “meant,” that is a parser eval — unit-test the parser — not a reason to score inner monologue with a judge. Keep the movie complete; keep the grade on the structured frames.

Run to execute this in your browser. Nothing is sent to a server.

What printed: same prose True. grounded tools ['search_kb']. invented tools []. equal as agents? False. The function names are the lesson: prose can match; the path is the product.

What goes wrong if you skip this

You will build a suite of expected essays. Models paraphrase. The suite flakes. Someone switches to “contains 5-7” — better, still incomplete — and still never notices a missing tool. You will also be unable to replay: production traces will use a different schema than tests (“prettier” JSON, incompatible keys), so incidents cannot become goldens. The next lesson says write the eval first; that eval has to be a trace predicate, or you will write the wrong thing first.

Skip traces and LLM-as-judge becomes a vibe on the last paragraph. Skip traces and injection evals cannot see that wire ran because the PDF said so. The exam paper is the whole track’s substrate.

How agents use this

CI should consume the same schema as production traces: same event kinds, same tool name field, same way you store a final. If the test format is prettier and incompatible, you will not replay incidents. You will maintain two movies and trust neither.

Treat a trace like an audit log you are willing to grade. Redaction is a later safety lesson; schema is this one. Sample cheap jobs; keep traces that hit write tools or safety tags. When two policies produce the same customer-visible sentence, the trace is how you still pick a winner in the suite.