Kill Switches You Have Practiced

A global flag: agents.disabled=true returns a fallback workflow or “a human will take it.” Per-tool: tools.refund=false. Per-tenant: disable a runaway customer without a full outage. Practice flipping them. Game day in staging. A kill switch in a wiki nobody can edit at night is not a kill switch.

Before a breaking worker deploy, drain: stop new jobs on the old SHA, let in-flight slices checkpoint, then switch. Health checks must not spend. A probe that calls the frontier model every 10 seconds is a cost bug and a rate-limit bug. Probe the queue and disk. Synthetic LLM checks belong on a slower cron with a stub.

If a single tenant’s swarm is melting the model bill, the first lever is pause that tenant. Global kill is the next size up. You built three levers so you do not use a sledgehammer on a fly.

How the box actually works

Admission happens before a slice spends.

Workers read flags at slice start, not once at process boot. A kill that requires a restart is too slow.

Owners’ names live next to the switches. 24/7 path to flip: on-call has permission in the config service, not a request ticket that lands Monday. Rollback command lives in the PR template.

Practice: staging game day, time until the flag flips, confirm no new refunds, confirm Beta still runs when only Acme is paused.

Workers must read flags at slice start. Caching flags for the life of the process means a kill waits for a deploy you were trying to avoid. A 10-second cache is a product choice; a 1-hour cache is a bug. Document the freshness.

24/7 path: on-call’s SSO group can flip agents.disabled, tools.refund, and paused_tenants without a change-management ticket that lands Monday. Audit the flips. The wiki can explain the levers; the wiki cannot be the lever.

A wiki-switch ticket

The runbook said “set agents.disabled in the wiki and wait for a deploy.” The deploy took 40 minutes. Refunds continued. Afterward flags were live-read, practiced monthly, and the first lever in a cost incident was paused_tenants=["acme"]. Search kept working for everyone until a later global kill during a vendor 503. Three levers, in order, with names on them.

Run to execute this in your browser. Nothing is sent to a server.

First line: Acme admitted, refund allowed. Tool kill: refund false, search still true. Tenant pause: Acme not admitted, Beta still admitted. Global: even Beta is refused and search is false. Three levers: one tool, one tenant, everyone. Search still worked until the global kill. That order is the on-call script.

What goes wrong

Switch requires deploy. Switch requires a person who is on a plane. Workers cache flags forever. Global kill as the only lever. Never practiced, so the path 404s. Health probes that spend during a kill (you pay to be told you are dead). Drain skipped, in-flight writes duplicate.

A switch that is “scale to zero” kills HITL-waiting jobs without a checkpoint. Pause the queue and flip the tool flag instead. A switch nobody can find because it was renamed in last quarter’s refactor is decorative. Keep the names stable: agents.disabled, tools.refund, paused_tenants.

If flipping refund also silently disables search because of a bad if-statement, you will take down FAQs while containing payouts. Test the three levers independently.

How to test it

• Unit tests matching the four prints.
• Integration: flip refund off, running workers refuse the tool on the next slice without restart.
• Game day timer: flag flip < 2 minutes in staging.
• Probe does not call the model.
• Tenant pause leaves other tenants’ jobs running.

If game day fails, the switch is decorative. Fix it before the real storm.

Test independence: refund off must not change search_kb; tenant pause on Acme must not pause Beta; global kill must stop both. If those three tests are not in CI, the levers will surprise you. Record flag freshness (how old the cached copy may be) next to the game-day timer. A switch that only works after a rolling restart is a deploy, not a kill switch — fail that test.

How agents use this

Keep a 24/7 path to flip flags. Write the rollback command in the PR template. Owners’ names live next to the switches. First lever for a melting bill: pause that tenant. First lever for a refund storm: tools.refund=false and maybe queue pause. First lever for “we do not know”: global kill, then see (next part).

Customers can hear “refunds paused; tickets queued.” They cannot hear “we are finding the prompt.”

Practice until the muscle memory is tool → tenant → global, not global-first. After a real flip, write down how long it took and fix anything slower than two minutes.

Game-day the three levers in order every quarter: tool, tenant, global. Time them. If global is the only one anyone remembers, you will take Beta down for Acme’s swarm. Put owner names next to each switch so paging is not “who has the dashboard.”