Version Everything the Policy Needs

A run is not reproducible unless you record what policy ran. Policy is not only the system prompt. It is the template version, the tool schema the model saw, the model id, the router flags, and the git SHA of the worker that executed tools. If the answer to “what prompt was in prod on Tuesday?” is “whatever YAML was on the box,” you will guess during the incident, and you will guess wrong.

Mismatch is itself an incident: worker sha-abc with prompts p9 you never tested together. Canaries exist so that pair meets production traffic on purpose, with metrics. Accidental pairs meet production as a surprise.

This lesson is not how to write eval goldens. It is how to stamp the bundle so that when a golden fails in prod, you can name the bundle. CI later will refuse untested pairs. Tracing later will show the stamp on every span. Here we make the stamp exist.

How the box actually works

At enqueue or at first worker slice, copy a bundle onto the job. The job carries that bundle until it terminates, even if you deploy in the middle. New jobs may pick a new SHA. Old jobs should not silently switch templates mid-refund.

Owners: runtime stamps the worker SHA. The prompt/ML team owns template and schema versions in a registry, not in an unversioned file on disk. Ops owns flags with history — who flipped tools.refund, when.

Compatible means “this worker SHA was tested with this prompt version.” A table in git or a config service maps sha-abc → {prompt: p12, schema: t4, model: fake-small}. If someone edits production YAML by hand, there is no row, and you cannot replay.

Replay is the point. Given the stamp, you can re-run the job against recorded observations (read-only) or against a fake model in CI. Without the stamp you are interpolating from memory.

Stamp before the first model call, not at the end of the job. A crash at step 2 with no versions object is how you get two stories about which template ran. If the worker is chosen after enqueue (canary percent), the stamp belongs on the worker’s first slice, and the job row must be updated in the same transaction as “I took this SHA.” Gateway-time stamps are fine when the gateway already knows the SHA. They are a lie when pick_sha lives in the worker.

Operator procedure when someone asks “what was live?”:

1. Query jobs in the time window; group by the versions tuple.
2. Open one trace per tuple; confirm spans copied the same fields.
3. Diff BUNDLES (or the registry) for those SHAs.
4. If a job has no stamp, that gap is part of the incident.

A Tuesday incident

Finance asked why refund language changed on Tuesday afternoon. Dashboards showed more HITL rejects. Nobody could answer which template was live. The worker SHA on the pods was from Monday. A well-meaning engineer had kubectl-edited the prompt ConfigMap at 14:10. Traces stored completions but not prompt: p12. The “diff” was a Slack thread.

Afterward every job got a versions object. Prompt diffs moved to pull requests. Rollback was “set bundle to p11”, not “does anyone have last week’s gist?” The Tuesday question became a query: versions.prompt = p12 AND created_at Tuesday.

That is operational versioning. It is not a research paper about eval properties. It is a stamp you can grep.

Run to execute this in your browser. Nothing is sent to a server.

STAMP prints worker sha-abc with prompt p12, schema t4, model fake-small, and the refund flag. Compatibility with p12 is ok. Compatibility with p9 is prompt_mismatch. The stamp lives on the job dict. A later worker, a trace exporter, or a replay script can all read the same object. If BUNDLES has no row for a SHA, you should fail closed — that is an untested pair, not a default to “whatever is on disk.”

What goes wrong

Versioning only the prompt and not the schema: the model gains a wire tool because a worker deploy shipped a new tools list with the old template. Versioning only the SHA and editing prompts in the live ConfigMap. Using “latest” as a model id. Stamping at end of the job so crashes have no versions. Letting canary jobs write the stable SHA because the stamp happened at enqueue on the gateway that did not know the worker pick.

Flags without history: you know refunds were off, not who turned them off. Frozen flags should be versioned too.

How to test it

• Stamp is present before the first model call.
• Replay with a recorded bundle produces the same tool names (fake model).
• CI: worker SHA and prompt version pairs not in the tested matrix fail the build (later lessons expand the gate; here, assert the matrix exists).
• Mid-deploy: an in-flight job keeps p12 even after p13 rolls out.
• Incident query: given a timestamp, you can list distinct version tuples.

Break the bundle on purpose in staging. The job should refuse or error prompt_mismatch, not silently run.

How agents use this

Put versions on every trace span, not only the job row. When cost spikes, group by prompt and model. When a forbidden tool fires, group by schema and worker.

Prompt diffs belong in pull requests with a pointer to the CI gate report. “I tweaked the vibe in prod” is not a version. It is an untracked deploy.

On-call’s first copy-paste is the versions object into the incident doc. If that object is missing, the incident includes “we were flying blind,” and the follow-up is this stamp, not a nicer dashboard color.

Keep the previous bundle on disk or in object storage. Rollback is a pointer change. Archaeology is not a rollback.

When you add a flag, add it to the stamp that same day. A canary percent that is not on the job cannot explain a split-brain of quality. When you add a model id, forbid the string latest. When you add a tool to the schema, bump the schema version even if the prompt text did not change — the model saw a different menu.

A weekly audit that lists distinct version tuples in prod, and checks each against the tested matrix, catches silent ConfigMap edits before finance does.