Rare Classes and Thresholds

If 1 in 20 tickets needs a human, a model can get 95% accuracy by never escalating. You already saw that. The fix is not a fancier net. The fix is to choose a cutoff for the score, using costs, on validation.

The model outputs a chance p (or a score you can sort). Default is “predict yes if p >= 0.5.” That 0.5 is a hyperparameter. Lower it if missing a yes is expensive. Raise it if extra yeses drown the team.

Imbalance is a property of the world (few jailbreaks, few refunds, few SQL questions). Thresholds are a property of the decision. You can also weight the rare class in the loss. That changes training. The cutoff changes inference. You can use both. The cutoff is cheaper to retune when product changes its mind.

Precision and recall trade

As you lower the cutoff, you catch more real cases (recall up) and you also raise more false alarms (precision down). There is no free number. There is a cost: FP costs a human 5 minutes; FN costs an incident.

A precision-recall curve is that trade drawn on validation: each cutoff is a point. You pick a point, freeze it, then score test once at that frozen cutoff. Picking the cutoff on test is peeking.

Do not pick on the same 10 tickets you will publish. That is test-set peeking with extra steps.

Class weights in the loss (pay more when the rare class is wrong) are another lever. They change the chances coming out of training. After that, you still have a cutoff. Weights are not a substitute for costs. Costs live at the decision.

Run to execute this in your browser. Nothing is sent to a server.

What printed: three cutoffs, three decision lists. At 0.5, pred is four yeses on the last four scores: precision 0.75 (one false alarm at 0.70) and recall 1.0 (all three true positives caught). At 0.3 an extra yes appears at 0.40: precision falls to 0.6, recall stays 1.0. At 0.8 only 0.82 and 0.91 fire: precision 1.0, recall 0.67 because the true positive at 0.55 is missed. Lower bar, more alarms. Higher bar, fewer misses of precision, more misses of recall. Pick using validation costs, then freeze. Do not pick on test.

Ten rows are a cartoon. In production you need enough positives on validation to see the curve. If you have three positives, your cutoff is a coin flip. Get more labels for the rare class, even if you undersample the common class for the plot.

Operating points for agents

“Escalate if the router’s top chance is below 0.6” is a threshold (abstain when unsure). “Refuse the shell tool unless p(safe) > 0.95” is a threshold. “Retrieve if cosine >= 0.35” is a threshold, and cosine is not a chance — you still pick the number on labeled validation.

Log both the score and the decision so you can redraw the line next quarter without retraining. If you only log the yes/no, you cannot retune.

When the mix drifts (next part), the same cutoff can quietly fail. Recalibrate or re-pick on a fresh validation slice. Do not wait for accuracy to move. Accuracy may not.

Sampling and training tricks (stay light)

You can oversample the rare class in train, or undersample the common class. That changes the mix the loss sees. It does not magically create information. It can bias the chances (they may no longer match frequencies). Then you need calibration or a cutoff chosen on a validation set that has the real mix, not the resampled mix.

That last sentence is the bug: train on a 50/50 remix, pick 0.5, deploy into 1/20 world. The cutoff was chosen for a fake world. Keep a validation slice with natural rates.

A costed example you can copy

Suppose a false escalation costs 5 minutes of human time (say 5 dollars) and a missed dangerous action costs 500 dollars. On validation you have 1000 tickets, 50 real positives. At cutoff 0.5 you catch 30 (so 20 misses) and raise 20 false alarms: expected extra cost is 205 + 20500. At cutoff 0.2 you catch 45 (5 misses) and raise 80 false alarms: 805 + 5500. Compute both. Pick the lower expected cost. Freeze that cutoff. Then score test once.

The numbers in your product will differ. The shape will not: two error types, two prices, a curve of cutoffs, one freeze. If you cannot name the prices, you cannot name the cutoff. “F1 was higher” is not a price.

Abstain is a band, not a single line: if p is between 0.4 and 0.6, call the LLM or a human. That band is two hyperparameters. Sweep them on validation with the same cost table. Logging p is mandatory; without it you cannot move the band next quarter.

Rare-class detectors also need enough positives on validation. If you have four jailbreaks, every cutoff is folklore. Oversample for training if you must; keep natural rates for choosing the operating point. That split of jobs is this page in one line.

Common mistakes

• Leaving 0.5 forever because it felt default.
• Picking cutoff on test.
• Picking cutoff on resampled validation.
• No logged scores, so you cannot retune.
• Treating cosine or “model confidence” as already a costed chance.

How agents use this

Safety gates are thresholds. Human escalation is a threshold. Retrieval is a threshold plus a k. Write each as a named hyperparameter, sweep on frozen validation with explicit FP/FN costs, freeze, then touch test once.

If product says “fewer interruptions” next quarter, you may only move the cutoff. That is cheaper than a new model. It still needs a new look at validation, not a vibe.

Watch out:Do not pick the cutoff on the same 10 tickets you will publish. That is test-set peeking with extra steps.