Incident Playbooks

An incident is when the agent did something you promised it would not — or stopped doing something you promised it would — at a scale that matters: wrong refunds, data mixed across tenants, a cost spike, a loop that emailed 2,000 customers. You need a playbook before that afternoon. Writing it during the event produces heroics, not control.

Minute 0–5 is contain. Flip the kill switch or the specific tool flag (refund=off). Pause the queue (or that tenant’s queue) if jobs are still applying. Page the owner; open a shared doc with trace ids. Do not start by tweaking the prompt in prod. Contain first.

Agents fail as distributed systems: queues, tools, flags, vendors. “The AI was weird” is not a severity. A tool flag, a prompt version, or a vendor — named — is. Customers can hear “refunds paused; tickets queued.” They cannot hear a persona rewrite.

How the box actually works

Keep a one-page playbook per class of blast (refund, tenant leak, cost, injection, vendor). Each page has the same spine.

Containment levers you already built: tools.refund=false, paused_tenants, agents.disabled, queue pause. Ledger audit is part of contain for money: list rows above a cents threshold, freeze, do not delete.

Status sentence, ready to paste: what is off, who is safe, when the next update is. Update on a clock (15–30 minutes), not when you feel brave.

Owners: on-call runs the spine. Product owns customer sentences. Security owns leak classes. Nobody owns “tweak the vibe in prod.”

Put the playbook next to the flags, not in a slide. A one-pager per class (refund, leak, cost, injection, vendor) with the same spine is enough. Each page lists the first lever, the blast-radius query, who to page, and the status sentence template. If the page is longer than a screen, nobody will read it at 2 a.m.

Game-day the spine quarterly with a fake ledger. Time contain. Time the first status sentence. If either is over five minutes, the path is wrong (permissions, unknown URLs, flags cached).

A prompt-tweak storm

Refunds fired without HITL. The first instinct in the channel was a new system prompt. Refunds continued for 12 minutes. The playbook would have flipped tools.refund and paused the queue in one minute, listed two ledger rows over 100 cents, and noted that a golden “refund without human approval fails” did not exist. Afterward they ran the tryit as a drill: contain prints flags off, radius prints the two jobs, new golden needed? True. The prompt did not change that day.

Run to execute this in your browser. Nothing is sent to a server.

contain shows refund false and queue paused. radius lists both 400-cent jobs. new golden needed? True because the trace has refund without a human flag. Refunds off. Queue paused. Two jobs in the radius. The new golden is “refund without human approval fails.” The playbook print is the spine. Nothing in this function edits a template.

What goes wrong

Prompt art as incident response. Scaling workers to “finish faster” (more refunds). Deleting the ledger. No shared doc, so trace ids live in five DMs. No status clock. Blaming “the AI” in public. Playbook only in a slide deck. Game day never run, so the flag path 404s.

Heroics that skip contain look brave and extend the blast. “I can patch the parser in five minutes” is allowed after refund is off. The playbook order is the product. If two incidents overlap, still contain both; do not average them into one vague channel.

How to test it

• contain / blast_radius / golden-needed unit tests.
• Staging game day: fake storm, time to flag flip, time to a status sentence in the doc.
• Pager payload includes trace ids.
• Ledger freeze does not delete rows.

If you cannot flip the flag in staging in two minutes, the playbook is fiction.

Rehearse the status sentence out loud: what is off, who is safe, next update time. Put a template in the doc so people fill blanks instead of inventing tone. Include a fake finance page in game day so contain is not only a unit test of dicts.

How agents use this

Have a status sentence ready. Do not blame “the AI.” Blame a tool flag, a prompt version, or a vendor — specifically. Keep trace ids in one doc. Contain side effects first. The golden is not optional homework; it is the last line of the playbook (two lessons from now).

Print the spine on the on-call cheat sheet: contain, trace, rollback, write a golden. Put the cheat sheet where flags live.

Do not scale workers to “finish the storm faster.” That applies more refunds. Do not delete the ledger. Do not blame “the AI” on the status page. Name the flag you flipped and the population that is safe. Next update in 15 minutes even if the update is “still contained, still looking at traces.” Keep one incident channel, one doc, one person typing the status sentence. Parallel heroes are how you get two global kills and no traces.

After contain, nobody edits prompts until see is done. Put that sentence in the playbook in bold. The channel will still suggest it; the playbook is how you say no without a debate.