Canaries and Feature Flags
Send 5% of jobs or a volunteer tenant to the new worker. Watch goldens, forbidden tools, cost, HITL, latency — then promote or roll back. Flags are versioned config.
Send 5% of jobs — or one tenant that volunteered — to the new worker. Watch: golden-tag success, forbidden-tool count, cost, human-gate rejects, latency. Promote or roll back. A 5% canary that you never look at is a dice roll with extra steps.
Feature flags turn dangerous tools off by default in a new region (computer-use). Config as code: model id, template version, canary percent live in git (or a config service with history). A dashboard click nobody can diff is how prod drifts from staging. Frozen flags should themselves be versioned so you know who flipped them.
This is how you measure a deploy, not how you write goldens. The numbers you watch are ops fixtures: the same tags CI uses, sampled on real jobs.
How the box actually works
Routing order:
- Global kill → no SHA.
- Volunteer tenant list → always canary SHA.
- Percent bucket on job id → canary or stable.
- Default stable.
Measure per route. A blended dashboard will hide a canary that is on fire if it is only 5% of traffic.
| Watch on canary | Promote if | Roll back if |
|---|---|---|
| Golden-tag success (billing, safety) | Holds vs stable | Drops below floor |
| Forbidden-tool count | ~0 | > 0 |
| Cost per successful job | ~stable or expected | Spike |
| HITL reject rate | No unexplained spike | Spike |
| Latency / queue | Within SLO | Saturated |
A blended dashboard hides a fire on 5% of jobs. Promote or roll back on the clock.
Watch the canary aloneOwners: runtime implements route. Product picks volunteer tenants. On-call watches the canary dashboard during the window. ML/prompt owns “promote the bundle.” Security owns dangerous-tool defaults off.
Flags for tools are not only canaries. tools.refund=false is a kill you will practice in the next lesson. Here, they are how a new region ships with computer-use off.
Watch in a time box. Ninety minutes with a named watcher is a canary. An open-ended 5% with a dashboard nobody has bookmarked is production with extra latency. Promote is a flag change: canary_percent to 100 or stable_sha to the new SHA, then canary_percent back to 0. Record who promoted.
Volunteer tenants should be small and willing. Your largest customer is a bad volunteer. Hashing on job id spreads risk; hashing on user id can trap one company on a bad SHA forever — document the key.
An unwatched-canary ticket
They shipped sha-new at 5% on a Friday and went to lunch. Forbidden-tool ticked 4 times on canary only. Blended dashboards looked fine. A volunteer tenant would have concentrated the risk — and the metrics — on lab. Afterward: required watcher, required volunteer tenant for policy changes, promote only when canary forbidden-tool stays ~0 and billing goldens hold.
Config had lived in a UI. Staging was p11, prod was p12-edited. Git history of flags fixed the drift.
Run to execute this in your browser. Nothing is sent to a server.
Acme’s job_17 is stable with reason stable. Tenant lab always gets sha-new with reason volunteer, even on the same job id. Bucket d is percent-canary. Volunteer tenant lab always gets the new SHA. Percent canary still uses the id bucket. Two knobs, one function. Watch both routes separately.
What goes wrong
Canary at 100%. Canary at 5% with no dashboard. Volunteers who are your largest customer. Flags without history. Measuring only “did the pod start.” Promoting because the prompt “feels nicer.” Sticky canary: job_id hashing is good; hashing on user id can trap a big customer forever in a bad SHA — know which key you use.
Promoting at 17:00 on Friday without a watcher is a canary you will meet on Saturday. A dashboard click that changes percent without a git row is how staging says 5 and prod says 40. Frozen flags need history: who, when, why, previous value.
If canary and stable share a queue with no sha label on metrics, you cannot watch per route. Labels are part of the canary, not extra chrome.
How to test it
- Route unit tests: kill, volunteer, percent, stable.
- Metrics labeled by
shaandreason. - A fake canary forbidden-tool fails the promote script.
- Flag history: a test account can diff last 20 flips.
Time-box the canary window. An open-ended 5% is unwatched by another name.
A promote script should refuse if the canary window has no named watcher in the change ticket, or if forbidden-tool on canary is above zero. That sounds bureaucratic until the first Friday lunch. Staging should run route against a fixture list of job ids so percent and volunteer do not surprise you in prod.
How agents use this
Measure quality per route. Promote only when the canary’s forbidden-tool count stays ~0 and billing goldens hold. Put dangerous tools behind flags defaulting off. Keep staging flags in git next to prod, with the same names.
Watch the canary like a deploy, because it is one. Book the window. Name the human. Promote or roll back on the clock, not when the channel goes quiet.
Config as code means a PR for percent changes except during an incident flip. Incident flips still get a follow-up PR so git matches prod. If you cannot answer “what percent is canary?” from git plus the config service history, you cannot operate the canary.
A volunteer tenant is a gift. Instrument them. Thank them. Do not surprise your largest customer with that role.
Measure quality per route so a burning 5% cannot hide in a blended p95. If forbidden-tool ticks once on canary, roll back. “Maybe it was a coincidence” is how coincidences become refunds. Dangerous tools stay behind flags defaulting off when you enter a new region or a new SHA.
Check your understanding