Data and Splits

A model that memorizes its homework and fails the exam is not a model. It is a lookup table. The way you prevent that is simple and often skipped: split the data.

The split is not a formality for a homework PDF. For agents, the “exam” is next week’s tickets, a new user, a new product name. If those rows influenced which prompt you kept, which features you built, or which cutoff you froze, you already peeked. The published number is then a story about the peek, not a prediction of tomorrow.

Three piles, three jobs

If you tune on the test set, it is not a test set. It is a second training set you are lying about.

A common default is 80 / 10 / 10. The percentages matter less than the rule: test examples must not influence any choice, including which features you built, which words went into the bag, which few-shots sit in the prompt, and which cutoff you picked.

Validation is the working exam during development. You will look at it many times. That slowly wears it out: each look is a bit of leakage. That is why test still exists. When validation has been stared at for a quarter, freeze a new test slice from later traces and retire the old one as extra train, or keep it as a historical benchmark with a version stamp.

Shuffle — except when time is the feature

Random shuffle is correct when examples are independent: spam emails, isolated tickets with no user overlap. It is wrong when the future must not leak into the past. Agent logs are a time series. If Tuesday’s incident write-up is in train and Monday’s related thread is in test, you inverted time. If the same user’s style is in both, you may be testing “do we recognize this person,” not “do we route this kind of ask.”

For traces, split by time, by user, or by conversation id, not by random row.

Grouped splits mean: all chunks of one ticket stay together. All messages of one conversation stay together. If you split a ticket into five retrieval chunks and scatter them, the model can “find” the test chunk because its sibling sat in train.

Time splits mean: train on January–March, validate on April, test on May. Product launches, new error strings, and new tools live in the later months. That is the point. A random shuffle would sprinkle May into train and make May look easy.

User splits mean: some customers never appear in train. If you serve enterprises that each have a private dialect, this is the honest exam.

Small data needs honesty more, not less. With 40 traces, a lucky shuffle can put all the hard tickets in train. Freeze the ids. Report them. Do not reshuffle until you bump the dataset version.

Leakage

Leakage means the model saw the answer, or a proxy for the answer, during training or during “eval.”

Common agent leaks:

• The label column left in the feature table
• Retrieval that indexes the eval questions
• Training on the whole chat, including the assistant’s final message, while predicting an earlier tool
• The same ticket in train and test after you split it into chunks
• Few-shot examples in the prompt that are also in the test conversations
• Scaling, vocab, or IDF weights computed on the full file including test
• A human “cleaned” a test label after seeing the model’s answer, then republished last quarter’s number

Deduplicate on conversation id, not on chunk text alone. The same ticket pasted twice is leakage’s cousin. Near-duplicates (“please refund” vs “pls refund”) can still leak if they are the same user and the same order id.

Run to execute this in your browser. Nothing is sent to a server.

What printed: the random split has 14 train rows and 6 test rows, and user overlap is [0, 1, 2, 3] — four of five users appear on both sides (user 4 happened to land on one side only). The user split has 12 train and 8 test, and overlap is []. Random split shares people. User split does not. For support agents, the second one is the honest exam: can you route a new customer, not a customer you already memorized.

The seed is fixed so this page is repeatable. That is the other half of splitting: freeze. Store the list of example ids next to the code. If you reshuffle every experiment, you cannot compare two prompts. Freeze at a timestamp. If someone “cleans” a test label, that is a new dataset — bump the version.

Version the exam

A dataset for agents is not a CSV you overwrite. It is:

• A snapshot of traces (or ids pointing at immutable logs)
• A rubric for y
• A split file: which ids are train, val, test
• A version name

When policy changes (refunds now take ten days), old labels can become wrong. You either relabel and bump the version, or you keep the old exam as “historical January policy” and add a new exam. Silent relabeling of test is how dashboards heal themselves.

Common mistakes

• 90/10 with no validation, then picking the cutoff on test.
• Splitting rows after exploding a conversation into messages, so the same ticket is in two piles.
• Indexing the knowledge base with the test questions “just for the demo.”
• Using production traces that already contain the current prompt’s few-shots.
• Comparing model A and model B on different random splits and declaring a winner.

How agents use this

Your production logs are the dataset. Before you fine-tune or even tune a prompt, freeze a held-out slice of traces. If the retrieval index contains the test questions, every RAG demo is leaking.

Split by conversation id first. Then, if the product is seasonal or launching weekly, prefer a time cut for the test slice: the last two weeks, untouched. Validation can be the two weeks before that. Train is everything older that you are allowed to use.

The scientific method from the first lesson is unusable without this page. Fit on train. Choose on validation. Publish test once. Watch a new slice next week, because the world moves (drift, later).

Watch out:Deduplicate on conversation id, not on chunk text alone. The same ticket pasted twice is leakage’s cousin.