What Blocks Merge
Block on forbidden tools, authz regressions, parser failures, and a drop on a critical tag. Do not block on flaky live search or uncalibrated judge scores. Do not skip the gate.
Must fail the build: any forbidden tool on the safety set; authz regressions in tool tests; parser cannot round-trip the schema; success rate on a critical tag (for example billing facts) drops below a threshold you chose. Should not block on: flaky live-web search; exact-string prose diffs; judge-only scores with no human calibration.
Skipping the gate “just this once” is how safety cases rot. If the golden is wrong, change the golden in the same PR. Promote nightlies that are stable into the PR gate without the live model. The PR gate stays deterministic. The nightly hunts surprises.
This lesson is the policy of the gate, not how to invent goldens. Empty billing rate is a blocker, not a free pass — missing data is fail closed.
How the box actually works
merge_ok reads a report object. Blockers are a list. Ignored noise is recorded so it does not vanish: live-web flakes, uncalibrated judges. Reviewers see both.
| Result | Merge? |
|---|---|
| forbidden > 0 | No |
| authz_fail > 0 | No |
| parser_ok is false | No |
| billing_rate missing or below min | No |
| live_web_flakes = 4, rest green | Yes, flakes listed as ignored |
| Uncalibrated judge unhappy, rest green | Yes, ignored |
Do not skip the gate. Change a wrong golden in the same PR.
What actually blocks mergeOwners: security owns forbidden + authz. Runtime owns parser. Product owns critical tags and the min rate. Platform owns keeping live-web out of the PR job. A gate nobody trusts is worse than a slow suite people actually watch — same dashboard for PR and nightly so product sees trend lines, not only merge blockers.
Skipping the gate is an incident, not a Slack emoji. If the golden is wrong, change the golden in the same PR so git shows policy changed. If live-web flakes, quarantine that row with an owner. If someone needs to ship a parser fix while safety is red, they are not allowed to skip safety — they revert the unrelated failure or split the PR.
Missing billing_rate is fail closed. Empty is not 100%. That bug ships wrong refunds with a green checkbox.
A flake-then-skip ticket
A PR stayed red for a day on live search. Someone skipped the whole gate to ship a refund parser fix. The parser was fine. A forbidden-tool regression hid in the same PR and shipped. Afterward live-web was quarantined, forbidden still blocked, skipping the gate required a named incident. Empty billing in the report blocked (None < 0.9). Flakes were visible, not silent.
Run to execute this in your browser. Nothing is sent to a server.
good may merge: four live-web flakes are ignored, billing is 1.0. bad must not merge: forbidden tool and a 0.5 billing rate. empty billing is a blocker because billing_rate is missing — fail closed, not a free pass. Live-web flakes are ignored. Forbidden tool is not. That is the policy in code.
What goes wrong
Blocking on prose. Blocking on live web. Not blocking on missing billing. Skipping the gate. Quarantine without an owner so the row dies. Two dashboards, so nightly rot is invisible to product.
A second failure mode: the gate is so large nobody knows which blocker class fired. merge_ok returns a list for a reason — put that list in the GitHub check summary. “CI red” is not an actionable state. “forbidden_tool on c1” is.
Calibrated human review of judges belongs on nightly trends, not on merge. If you block on an uncalibrated score, people will lobby to delete the suite. Protect forbidden tools from that lobbying.
How to test it
The three prints. Plus: a PR fixture with only flakes is green; with forbidden is red; with parser_ok false is red. Document skip policy: skipping creates an incident id.
Add a case where billing_rate is omitted and assert merge is false. Add a case where uncalibrated_judge is angry and flakes are high, but forbidden is 0 and billing is 1.0 — merge is true. Those two cases are the policy. If someone “simplifies” merge_ok to not report.failed, the tests should catch it.
How agents use this
A gate nobody trusts is worse than a slow suite people actually watch. Same dashboard for PR and nightly. Quarantine noise. Never skip the whole gate to ship a “small” fix. If the golden is wrong, change it in the same PR so the history shows policy changed, not gate ignored.
When you add a dangerous tool, it enters the forbidden list in the same PR as the tool. When you add a critical product fact, it enters the billing (or equivalent) tag the same week.
Do not block on exact-string prose or uncalibrated judges. Do block on authz, parsers, forbidden tools, and the critical tag floor. Trust is the scarce resource. Spend it on the rows that mean “this would have been an incident.”
Publish the blocker list in the contributing doc so a new engineer does not add live-web to the PR job “for coverage.” Coverage that flakes is negative coverage. Nightly is where surprise belongs; merge is where known contracts belong.
If product asks to block on a new judge score, put it on nightly first until a human calibrates it. Then, if it is stable and tied to a real incident class, promote it into merge_ok with a named owner. Do not sneak it in as a silent extra fails.append. The check summary should quote blockers verbatim so Slack screenshots are unnecessary.
Check your understanding