Pass Rate Is a Fraction

Pass rate is hits divided by n: how many cases passed, divided by how many cases you actually ran. It is the first number leadership will ask for. It is also the easiest number to lie with. This lesson is the honest fraction, including the empty-suite trap. Slicing by tag comes two lessons later. Quarantine comes next. You still start here, because if you cannot compute hits over n without cheating, the rest of the dashboard is costume jewelry.

The definition is deliberately boring:

• n is the number of rows in the denominator. For now, all rows. Later, active (non-quarantined) rows.
• hits is how many of those rows have pass: True after check(item, out).
• pass rate is hits / n when n is not zero.
• When n is zero, return 0.0. Do not return 1.0. Do not return None and let a dashboard paint it green. Do not reuse last week’s number.

An empty suite is not “nothing failed.” It is missing homework. Shipping 100% on zero cases is branding. The last lesson of this track will put that lie in a trust() function. Learn the fraction first.

Why a single float is not enough — and still required

A single number hides tags: 90% can be 90/100 FAQs and 0/10 injections, or 9/10 injections and a struggling FAQ. You will slice. You still need the overall fraction so that “we added 80 paraphrases” cannot be confused with “we got better at billing.” Always print hits / n next to the float. Humans read 2/3. They misread 0.666... as “about 70%” or “basically fine.”

Gates and weights are how you stop the FAQ crowd from drowning harm. They do not replace the fraction. They sit beside it: “pass rate 0.91, injection gate fail, forbid-tool count 3.”

Walkthrough: two of three, and an empty list

Three rows: pass, pass, fail. Hits = 2, n = 3, rate ≈ 0.667. Print both 0.666... and 2 / 3. The fail might be the injection case from the golden set. That is a different conversation than “the agent is two-thirds of a chatbot.”

Empty list: pass_rate([]) returns 0.0. A naive implementation returns 1.0 because “zero failures divided by zero” got a special case wrong, or because someone wrote if not rows: return 1.0 to “keep CI green while we add tests.” That special case is how empty suites ship. Do not write it. Do not let a framework write it for you.

A third situation: n = 1, the one case is a FAQ paraphrase you added to feel productive. Rate is 1.0 and still almost meaningless. The number is honest; the suite is not. Coverage-by-tag will fail that suite. Pass rate will not. That is why this metric is necessary and insufficient.

Denominators must be explicit in every chart. “Pass rate this week vs last week” with n dropping from 80 to 8 is not an improvement. Print n on the axis, in the tooltip, in the Slack bot. If a tag has n = 0, that tag’s pass rate is 0.0, not “N/A painted green.” If you exclude quarantined rows, say active n in the title. If you weight forbid hits as suite-fail, say that in the same sentence as the float so nobody quotes 0.91 from the FAQ slice as the company number.

Do not smooth the fraction with a moving average that hides a single wire fail. You may average cost. You may not average a gate. Hits over n is a ratio of cases, not a temperature. When leadership wants one number, give them two: trusted (from the last lesson’s spirit: nonempty, tagged) and the fraction. A lonely 0.667 with n = 3 is a prototype. A 0.667 with n = 90 and floors met is a product conversation.

Integer hits matter more than extra decimal places. 2/3 and 200/300 are different risk. Rounding 0.666 to 67% in a slide is how a tiny suite looks like a majority. Always keep hits, n, and the float together in logs.

Run to execute this in your browser. Nothing is sent to a server.

What printed: a float near 0.667, then 2 / 3, then empty 0.0. Two of three is about 0.667. Empty is 0.0, not 1.0. If your language’s truthiness on empty dicts ever tempts you to skip r.get("pass"), keep the explicit boolean. Missing pass is not a hit.

What goes wrong if you skip this

You will report “we’re good” from a dashboard default. Empty files after a refactor will go green. Someone will quote 94% in a meeting while n dropped from 80 to 8. You will compare floats across weeks with different denominators and call it a trend. Finance will not care that you “improved 3 points” if the 3 points were FAQ clones and the wire probe disappeared from n.

Skip the printed fraction and reviewers cannot sanity-check the float. Skip the empty-suite rule and the rest of this track’s trust() lesson has nothing to hang onto.

How agents use this

The weekly slice is this number plus tags, gates, and cost. Math tracks teach precision and recall on classifiers; here the rows are traces (or their property outcomes). Put pass_rate in the same library as goal_satisfied. Call it in the runner after every case gets a boolean. Refuse to upload a report with n = 0 marked success.

For multi-agent jobs, compute pass rate per role and for the job. A 100% supervisor with a 40% worker is not 70%. It is a job that fails.