Quarantine Flaky Rows

Flakes happen. A vendor schema changed. A label was wrong. A clock moved. Finance updated the portal on Tuesday and the golden still wants 5-7. The runner is non-deterministic because someone hit live search. A judge disagreed with itself. Quarantine is the honest middle between two bad instincts: deleting the row so CI is green, and leaving a forever-red row so the team learns to ignore the suite.

Quarantine means: skip the row when you compute pass rate, keep the row in the file, and give it an owner plus a reason plus a date. The evidence stays. The denominator shrinks on purpose. The weekly slice still lists quarantined ids so they cannot vanish.

Deleting the row is how you forget the bug. The injection PDF that failed twice becomes a myth. Next quarter’s model upgrade brings it back. Leaving the row active while it is known-wrong is how you teach “red means whatever.” Both destroy the suite as a contract.

Rules that keep quarantine from becoming a junk drawer

Pass rate on active rows can look healthy while the junk drawer grows. Report both: active pass rate, and quarantine share. The last lesson’s trust() function will fail a report with too much quarantine. You can implement the cap as a second boolean in the runner.

Quarantine is temporary. The honest repairs are: update the expected property (policy changed), fix the product (agent still wrong), or fix the fixture (label was wrong). Quarantine is the holding pen while that commit is in review — not a lifestyle.

Walkthrough: three rows, one quarantined

Rows: a fail active, b pass active, c fail quarantined. Active set is a and b. Hits on active = 1 of 2. Rate = 0.5, not 1/3 (which would punish you for a known-bad fixture you have already flagged) and not 1.0 (which would pretend a does not exist).

Case c is still in the file. Someone grepping for injection still finds it. The next owner can un-quarantine when the policy commit lands. If you had deleted c, the threat model would have a hole and pass rate would look like 0.5 for a different, worse reason: you would have forgotten why 0.5 is not the whole story.

If finance changed the window, do not quarantine forever. Update must_contain from 5-7 to 10 in the same PR that updates the KB fixture. Quarantine is for the afternoon when CI is red and the PR is not ready. It is not for “we do not like this test.”

Flakes that are actually impure runners should not be quarantined as if the label were wrong. Live search, live clocks, live Stripe, unordered JSON in digests, temperature above zero on a “deterministic” suite — those are engineering bugs. Quarantine will hide them until the suite is folklore. Fix the runner: fake world, frozen clock, canonical JSON, recorded observations. Then the row can be active.

The weekly slice should list quarantined ids the way it lists failed ids. Owners get poked. Caps get computed as count and share. If the only injection probe is quarantined, coverage floors should fail even if active pass rate is 1.0 — you no longer have an injection exam. Do not let quarantine punch holes in the threat model without a replacement row.

A known-failing product bug is not a flake. If the agent still wires on the PDF, the row stays active and red until the allow-list ships. Quarantine is for bad labels and broken fixtures, not for “the agent is still wrong and we have a launch date.” Using quarantine to hide harm is how evals lie. The last lesson’s trust() will treat a high quarantine share as untrusted for this reason.

Run to execute this in your browser. Nothing is sent to a server.

What printed: 0.5, then active ['a', 'b']. Rate is 0.5, not 1/3. Case c is still in the file. If all rows were quarantined, score would return 0.0 — the empty-suite rule again. A wall of quarantine is not a pass.

What goes wrong if you skip this

Without quarantine, one broken fixture blocks every prompt PR, and someone will delete it at 5 p.m. Without a cap, everything painful gets tagged quarantine and the suite becomes a FAQ museum. Without owners, quarantines last a year. Without keeping the row, red-team wins evaporate.

Skip this and “flake” becomes a slur you apply to any safety probe that failed. Safety probes should be stable: same fixture, same allow-list. If they flake, your runner is impure (live tools, live clocks), not “the model is creative.” Fix the runner.

How agents use this

Cap how many rows may be quarantined (share and count). The production track can fail a build if the cap is blown; this lesson only demands the field, the owner, and the scoring rule. Print quarantined ids in the weekly slice. Do not hide them behind the float.

Never quarantine a row by editing the prompt to skip it. The row is data. The prompt is not a test filter. If the team is tempted to quarantine because “the model is creative,” freeze the world and the clock first. Creativity is not a reason to drop a forbid list.