Coverage by Tag

Track coverage by tag, not only overall pass rate. A 92% suite with zero injection fixtures is a vanity metric: the happy path drowned the threat model. Tags are how you refuse that drowning. Minimum counts per tag are how you refuse a PR that only adds FAQ paraphrases and calls it “better evals.”

Tags are labels on golden rows: billing, safety, injection, privacy, grounding, hitl. Pick a small set you will actually maintain. Twenty tags with one row each is a folksonomy, not a coverage policy. Three to seven tags with floors (for example: at least two billing, two safety, two injection) will change behavior.

Labeling is work. Double-label a slice and measure agreement. If two humans disagree on whether a case is injection or safety, the spec is vague — fix the spec before you automate a judge. Tags that mean “misc” will attract every awkward fixture and then the floor on injection will still be empty.

Pass rate without coverage is a trap

The coverage check runs before you brag about pass rate. missing tags fail the suite design, even if every existing row passes. That is a different fail than g2 forbid wire. One is “we never wrote the exam.” The other is “we failed the exam.”

Weights and gates still apply inside a tag. Two injection rows that are both paraphrases of the same PDF are not two threats. Diversity of story matters: direct jailbreak, indirect PDF, cross-tenant, must-still-help FAQ. Coverage counts are necessary; they are not a substitute for thinking.

Walkthrough: floors of two

Need: billing 2, safety 2, injection 2.

First set: two billing, one safety, zero injection. Counts show billing 2, safety 1, untagged none. missing includes safety (floor 2) and injection. ok: False. Pass rate on those three rows could be 100% and still be a vanity number.

Second set: add one safety and two injection. Floors met. ok: True. Now pass rate means something — still slice it, still fail on forbid hits, but you are no longer pretending.

Acme’s mistake looks like this: after a demo, someone adds twelve paraphrases of “how long do refunds take?” Pass rate stays high. Injection remains zero. The coverage function is the adult in the room. The PR that only adds FAQs does not “improve the suite.”

Floors are policy. Write them next to the golden schema: billing at least 2, safety at least 2, injection at least 2, privacy at least 1, must-help at least 2. When a new harm class appears (screenshot OCR injection), add a tag and a floor in the same change that adds the first row. A tag with floor 0 is decoration.

Diversity inside a tag matters once the floor is met. Two injection rows that are the same PDF with a different filename are one threat. Aim for: direct jailbreak, indirect document, tool-shaped JSON in an observation, and a must-help FAQ that uses the word “ignore” innocently if that is a real customer sentence. Safety is not only “say no”; it includes “still answer in-scope.”

Untagged rows are a smell. Defaulting to billing because that was the template is how injection never gets a floor. The coverage function should count untagged and you should fail if it is above a small cap. Double-label a slice: if two owners disagree, the tag glossary is wrong. Fix the glossary before you automate tagging with a model — that would be a judge you have not eval-ed, on the suite itself.

Coverage is a gate on suite design, computed before bragging. A PR that raises pass rate by adding twenty billing paraphrases and zero injection should fail missing or fail a “no new rows on starved tags” rule. Publish counts in the weekly slice so leadership sees injection n next to the 92%. If humans will not write injection rows, the threat model is unowned — that is an alignment process fail, not a model quality fail.

Run to execute this in your browser. Nothing is sent to a server.

What printed: the first dict has missing containing safety and injection (safety only had one row). ok is False. The second dict meets the floors; missing is empty and ok is True. Untagged rows would land in untagged and would not help any floor.

What goes wrong if you skip this

Happy-path gravity wins. Every intern knows how to write a FAQ golden. Few people enjoy writing jailbreaks. The suite becomes a mirror of what was easy to label. You will quote 92% to skip HITL. You will be surprised by an injected PDF in production. You will add a judge because “quality feels off,” when the off-ness is that harm was never a row.

Skip double-labeling and tags drift until coverage is a fiction: everything is billing because that was the default in the template.

How agents use this

Minimum counts belong in the runner next to pass rate. A PR that only adds FAQ paraphrases should not “improve” a suite that still has zero jailbreaks. Fail the PR on missing, not on taste.

Publish counts in the weekly slice: billing n, safety n, injection n, then pass rate per tag. Multi-agent: tag by role as well as threat, or a researcher persona will have no money-tool probes because “that is billing’s job” while still having refund attached by a copied config.

You now have goldens, an honest fraction, properties instead of essays, quarantine with owners, and coverage floors. Next: checkers — unit tests for tools, judges you only use when Python is not enough, evaluating the judge, and traces you can replay.