Traces as a Dataset

For agents, a row is usually a whole conversation (a trace): messages, tool calls, errors, tokens, the final answer. ML only starts when you can turn that into x, y, and a split.

This page is the scientific method from lesson 1, applied to production. If you skip the freeze, every dashboard is a story. If you skip the baseline, every model is a hero. If you skip the rubric, every accuracy is a coincidence.

The next track (transformers) is how the big function is built. This track was how you fit, split, and judge any function — including that one. Stay here until the dataset is honest. Architecture will not save a mushy y.

Freeze ids

Give every trace an id. Store three lists: train, validation, test. If you shuffle again next week, you cannot compare two prompts. If a labeler fixes a test row, bump the dataset version instead of silently improving last quarter’s number.

Prefer grouped splits: by conversation id, by user, or by time. Deduplicate on conversation id. Store the id lists in git or an object store next to the rubric. The model code is not the dataset. The ids are.

Run to execute this in your browser. Nothing is sent to a server.

What printed: two different test lists from two seeds — for example seed1 might show ['t03', ...] and seed2 a different trio. Same ten traces, different exams, incomparable scores. A prompt that “won” on seed 2 may have merely drawn easier tickets. Then frozen test ids reprints seed1’s test list: that is the exam paper you publish against. Keep it.

The "t%02d" % i line is old-style formatting, not an f-string. It names traces t00 through t09. Names beat anonymous rows when you debug a single miss.

Write a rubric

Before anyone labels, write one page: when is sql correct even if search would also work? Ambiguous traces go to abstain. Two labelers on a sample: if they disagree a lot, stop modeling and fix the action space.

Sample for humans. Do not treat the agent’s own output as gold except as a weak hint. Script what you can (goal_satisfied, schema, forbidden tools). Humans get the leftover judgment.

A hundred clean, versioned traces beat ten thousand unlabeled dumps. Coverage of tools and failure modes matters more than row count. Stratify the sample: include rare dangerous actions on purpose.

What to put in x

Only what the agent had at decision time: user text so far, tools already called, last error, retrieved chunk ids. Not the final assistant message. Not the close reason. Those are leakage.

If you train a first-tool router, x is the opening state. If you train a reranker, x is (query, chunk) at retrieval time. Different times, different rows. Do not smash the whole trace into one x unless the decision really saw the whole trace.

The loop you actually run

1. Log traces
2. Freeze a split
3. Label (or compute) y
4. Score a baseline
5. Change one thing (rule, prompt, index, small model)
6. Measure on validation
7. Touch test once
8. Watch drift next week

That is every lesson in this track in eight lines. Features, loss, ranking, calibration, thresholds — they plug into steps 3–6. Train vs inference is step 5’s type. Rewards are a form of y for a whole episode.

What a row contains, and how you version it

A practical row is a dict you could print:

• id — immutable
• split — train / val / test, from the freeze file, not recomputed
• x — decision-time fields only (text so far, tools so far, last error, chunk ids)
• y — rubric output (tool, pass/fail, reward, abstain)
• meta — product, language, timestamp, prompt version then in force (for drift slices, not as a feature unless it was known at decision time)

PII does not belong in the feature list you ship to a notebook. Redact. Access-control the freeze. A dataset of traces is customer data with a schema.

Coverage is a table: each tool, each forbidden action, each language you claim to support, a minimum count in val and test. If shell never appears, you cannot report recall on shell. Oversample the dangerous class into the labeled sample on purpose.

Versioning policy: changing a test label, adding ids, or changing the rubric bumps the version. Retraining on the same freeze does not. Publishing a number requires the version string. “Accuracy 0.81” is incomplete. “Accuracy 0.81 on traces-2026-03-01 test ids” can be compared to next week’s experiment.

A hundred rows with coverage beat ten thousand dumps of the happy path. The dump is for clustering and for finding what to label next. The freeze is for judging.

Store who labeled, when, and which rubric version. If two people disagree, keep the disagreement; do not silently pick the model’s answer as a tie-break. That tie-break is how the agent becomes the supervisor of its own exam. Sample disagreements into the next labeling round until the ceiling is a number you can live with.

Common mistakes

• Reshuffling test to make a prompt win.
• Relabeling test in place.
• x that includes the future.
• Ten thousand dumps, no rubric.
• No baseline on the same freeze.

How agents use this

Own the dataset like a product: version, rubric, ids, coverage. The model is replaceable. The freeze is the company memory of what “good” meant.

When a vendor ships a new checkpoint, you rerun this freeze. When policy changes, you bump a version and say so. When someone wants to train, you point at the remainder on this freeze. That is judging an agent with ML instead of with a demo.

Tip:A hundred clean, versioned traces beat ten thousand unlabeled dumps. Coverage of tools and failure modes matters more than row count.