Every Incident Ends in a Golden
If you cannot write a test that would have failed before the deploy, you do not understand the incident yet. Then game-day: old SHA fails, new SHA passes.
The only acceptable souvenir: at least one golden (and maybe a tool unit test) that would have failed before the deploy; an owner; a tag; a dashboard or alert if you were blind; a spec patch if the spec was wrong. After the golden is merged, run a game day: replay the fixture in staging with the old SHA (should fail) and the new SHA (should pass). If you cannot reproduce, you do not have a fix. You have a story.
Practice on a fake storm in staging and time how long until the flag flips. Production is where eval-track habits block the next merge. This lesson does not re-teach golden design. It demands a fixture that CI can run — the same shape as the refund-without-HITL function below.
Incidents should not end in a pep talk. Same week: golden, owner, tag.
How the box actually works
| Deliverable | Done means |
|---|---|
| Fixture trace or tool args | Frozen in the repo |
| Test that fails on old behavior | Red on old SHA, documented |
| Test that passes on new behavior | Green on new SHA |
| Owner + tag | Named human, billing/safety/authz/… |
| Alert if you were blind | Page or dashboard you actually use |
| Spec patch if needed | The doc matches the new gate |
| Game day | You ran old vs new in staging |
Old SHA fails. New SHA passes. A pep talk is not a close.
Incident ends in a goldenThe test is allowed to be small. refund in tools and no hitl_approved flag → fail. That is both a golden and a spec: refunds need a human flag.
Owners: the person who shipped the bug owns the golden unless you explicitly reassign. Platform owns getting it into the PR gate (fake model / fixture). On-call owns reminding the channel that “over” without a test is not over.
Close the incident ticket only when the fixture is merged and game day ran. “Customers stopped tweeting” is not a close reason. If you cannot fail the old SHA, you do not understand the bug: maybe it was a flag, a payload, or a race. Stay in see until the test is red on old.
The golden can be a tool unit test (tenant filter) or a tiny trace predicate (refund without HITL). It does not need a live model. It needs to be in the PR gate you already built.
An encore ticket
A HITL bypass was “fixed” by a prompt sentence. No test. A week later a worker refactor dropped the gate again. The encore cost more than the first show. The function refund_without_hitl would have failed the old trace and passed the new one. Game day: old SHA red, new SHA green. They added it to the PR gate the same week. The pep talk was canceled.
Run to execute this in your browser. Nothing is sent to a server.
Old SHA prints ok: False with refund_without_hitl. New SHA prints ok: True because the human flag is present. That function is the golden. It is also the spec: refunds need a human flag. Game day is running this against both images, not telling a story about them. If old does not fail, you still do not understand the incident.
What goes wrong
Fix without test. Test that only asserts the prompt contains “be careful.” Test that needs production keys. Game day skipped. Owner “the AI team.” Tag missing, so the PR gate never picks it up. Closing the incident because customers stopped tweeting.
A golden that is not in the PR suite is folklore again. A golden so broad it fails every unrelated prompt tweak will be deleted. Keep it tight: this tool, this flag, this tenant pair. Tight tests survive.
How to test it
The tryit. Plus CI: the new fixture is in the gate; a revert of the fix goes red. Staging game day recorded in the incident doc with times.
Check the old SHA in a worktree or image tag, not in memory. If you cannot run old, you cannot prove the golden. Store the fixture next to the incident id so a year later someone knows why c17 exists.
How agents use this
Same week: golden, owner, tag. Capture red-team and incident wins as merge blockers, not as folklore. The evals track taught you to store fixtures. Production is the last mile: those fixtures fail the build that would have shipped the encore.
If the spec was wrong (you never required HITL), patch the spec and the test. A test without a spec will be deleted as “too strict.”
Same week, not someday: fixture, owner, tag, gate. The encore is scheduled the moment you skip that week. Capture red-team wins the same way — they are incidents that happened in staging.
Name the tag so the PR gate picks the row up. “Misc” is how fixtures rot. Billing, safety, authz, cost — pick one. Put the incident id in a comment above the test so the next delete-the-strict-test conversation has a date and a dollar amount.
If the fix is a flag default, the golden still asserts the trace shape, not only the flag file. Flags get flipped. The test is what keeps the encore off the merge train. Game day without the old SHA is a story; with the old SHA it is a proof. Block the incident ticket on that proof. If old cannot be run, say so in the ticket and keep it open — that is still “we do not understand it yet.” Do not close on vibes.
Check your understanding