Drift

Drift means the data you see now is not the data you trained on. Labels shift (what “urgent” means). Features shift (people say “down” instead of “crash”). The mix of tools shifts (a new billing API). Documents shift (the policy PDF was rewritten).

A frozen test set from January will not catch a February product launch. You need a fresh slice of traces, on a schedule, with the same metric.

Two names you will hear:

• Covariate shift — the inputs x change (new phrasing, new users)
• Label shift / concept drift — the meaning of y given x changes (the policy for refunds changed)

You do not need the names to act. You need a dashboard that is not only the old eval. The old eval is still useful as a regression suite. It is not a substitute for this week.

What drift looks like in an agent

The router still scores 95% on January’s freeze and 60% on this week’s tickets. Retrieval P@k dies after a docs migration even though cosine code did not change. The keyword “crash” stops firing because the status page now says “down.” A new tool exists and the policy never calls it. Humans changed what “done” means (close the ticket vs send a survey).

Nothing in the training loss of last quarter warns you. Loss is a function of last quarter’s pile. Only a new labeled slice, or a computable metric on fresh traces, warns you.

A/B tests on prompts are drift detectors too. If the user mix changed the same day you shipped a prompt, you do not know which one moved the metric. Segment. Or do not ship on a launch day and call it science.

Run to execute this in your browser. Nothing is sent to a server.

What printed: train acc 1.0 — the word “crash” matches the old urgent tickets. prod acc 0.5 — on this four-row cartoon that is the two easy negatives only; the two urgent rows use “down” and the policy misses them (2/4). Then the sentence: same policy, new words, silent fail. Train accuracy is perfect. Production accuracy is the dummy that always predicts 0 on the urgent class. Nothing in the training loss warned you. Only a new labeled slice would.

Retrain, add features, or write a rule — after you measure. Do not “fix the prompt” in a vacuum if the retrieval corpus is a month stale. That is drift in the index: the geometry is fine, the documents are not.

A weekly habit

Sample live traces every week. Label a hundred, or compute y where you can (goal_satisfied, schema, forbidden tools). Score the router, the retriever (P@k), and the product metric. If last month’s number was 0.81 and this week is 0.64, you have drift or a break.

Then freeze a new eval version (new ids, new timestamp). Change one thing. Re-measure. If you change prompt, index, and model on the same day, you will not know which one answered the drift.

Watch mix: fraction of SQL vs search, fraction of new-user tickets, languages. Mix shift can move a metric without any single example looking weird. Stratify.

Retraining is not automatic virtue

Retraining on the latest month can fit the new words and forget the old incident, or fit a labeling error that crept in. Keep a regression slice of old critical cases. Drift response is: measure, then choose among rule, prompt, index refresh, small-model retrain, cutoff retune. Last on the list is a giant fine-tune.

Slices that name the move

When the number drops, cut the fresh sample before you retrain:

• New strings: “down” vs “crash,” a new product name, a new error code
• New tools: an API that did not exist in January
• New docs: policy PDF replaced, index not rebuilt
• New mix: more VIP tickets, a new language, a seasonal surge
• New labels: refunds now take ten days, so old y is wrong
• New policy in the prompt: you shipped a clause and the world also moved

Each cut points to a different fix. New strings → features or keywords. New tools → action space and labels. New docs → re-embed, P@k. New mix → maybe only the cutoff or the prior, not the whole net. New labels → dataset version. New prompt plus new mix → you cannot credit the prompt until you segment.

Index drift is the quiet one. Cosine code is unchanged. Neighbors are wrong because the corpus is wrong. Score P@k on a freeze of queries against the current index, not against a pickle from launch.

Feature drift can be watched without full labels: the rate of the word “down,” the mean cosine to the billing tool doc, the fraction of traces that call sql. Those are not y. They are alarms to sample. Then you label. Unlabeled dashboards do not replace a hundred honest rows; they tell you when to spend the hundred.

Common mistakes

• Only the January eval on the dashboard.
• Shipping a prompt the same day the product launches and crediting the prompt.
• Refreshing the index without re-running P@k.
• Silent relabeling of old test to match new policy, without a version bump.
• Retraining weekly with no frozen comparison.

How agents use this

The ML move is the same as the ops move: freeze a new eval version, then change one thing. Drift is why “we measured once at launch” is not a quality program.

Put the date on every number. “Router 0.81” is not a fact. “Router 0.81 on freeze 2026-01-15” is a fact. “Router 0.64 on sample 2026-09-14” is a different fact. Drift is the gap. Your job is to explain the gap with slices (new words, new tools, new docs), not with vibes.

Watch out:A/B tests on prompts are drift detectors too. If the user mix changed the same day you shipped a prompt, you do not know which one moved the metric.