Baselines

A baseline is a stupid, honest predictor you publish next to your fancy one. Majority class. Keyword rules. “Always call search.” Last week’s prompt. A one-line function a new hire could write in a morning.

If the neural net cannot beat the dummy, you do not have a modeling win. You have a demo. Teams skip this because baselines are embarrassing. That is the point. Embarrassment is cheaper than a GPU bill and a quarter of prompt poetry that loses to if "password" in text.

A baseline is also a product decision. If keywords are 96% and the remainder is rare and unstructured, ship the keywords. Training is for the remainder that is large, stable, and labeled.

Majority class

Count the labels in train. Always predict the most common one. On a set that is 90% “not urgent,” that dummy is 90% accurate. Any real model must beat that number, not 50%.

Do not count majority on train-plus-test. That peeks. Do not count it on test and then also report test accuracy of the dummy as if it were a secret you discovered after training. Publish the dummy from train frequencies, scored on the frozen test slice — the same protocol as the real model.

For regression, the dummy is often “always predict the train mean” or “always predict last week’s average latency.” Beating the mean is the bar, not beating zero.

Constant policies and last week’s prompt

Always-search is a policy. Always-finish is a policy. Always-ask-human is a policy. Score them. Always-ask-human may have beautiful safety numbers and terrible cost. Always-search may have decent task success on a doc-heavy product and fail the moment someone asks for a count of users.

Last week’s prompt is the baseline that product actually cares about. A new chain-of-thought template that loses to last week is a regression, even if it beats majority. Freeze the old prompt as a named version. Rerun it on the new eval slice when the data changes, so you are not comparing a new model on new data against an old model on old data.

A random policy (pick a tool uniformly) is sometimes worth printing so people see that 33% on three tools is chance, not skill. A stratified random that respects class frequencies is a slightly sharper dummy.

A keyword policy

Write ten lines that a new hire would write. That is often already a strong router. ML is for the remainder those lines cannot cover.

Do not tune the keywords on the test set. Write them from train examples and documentation, freeze them, score test once. If you keep adding a special case after every incident, you are training by hand on a stream that includes what you will later call eval. Version the rule file.

Run to execute this in your browser. Nothing is sent to a server.

What printed: majority class is 0 (search appears most), accuracy about 0.38. Keyword accuracy is 1.0 on this toy set. Always-search matches majority here, about 0.38. Keywords beat majority. A transformer that scores 0.75 on the same frozen examples is a loss against this keyword policy, not a win. The neural net has to beat 1.0 here, or you do not train.

This file is tiny and the keywords were written while looking at the same rows. In production you would freeze keywords from train and score a held-out slice. The shape of the comparison stays: dummy, rule, then anything fancier.

Cost, latency, and “beats” means all of it

Accuracy is not the only column. A tiny rule that runs in a millisecond and needs no vendor call can lose 2 points of accuracy and still win the product. A fine-tune that beats keywords by 1% and costs 50 times more is a choice, not an automatic promotion.

Publish a small table: method, metric on frozen test, latency, dollar cost per 1000 tickets, and whether it needs labels to refresh. Baselines make that table make sense. Without them, the table is three fancy rows arguing with each other.

When the baseline wins

Ship it. Put the remainder on a dashboard. If the remainder grows and clusters (unsupervised later), promote a new rule or a small model for that cluster. Do not train a global net to avoid writing if "password" in text.

If you cannot beat majority, debug labels, features, and leakage before you buy a larger model. The data is often the dummy’s ally: noisy y, leaked x, or a question nobody defined.

Common mistakes

• Reporting 91% without saying majority is 90%.
• Tuning 40 keyword special cases on the same tickets you publish.
• Changing the eval when the model loses to the dummy, instead of publishing the loss.
• Using a weak dummy (random) when a strong one (last week’s prompt) exists.
• Forgetting that “always escalate” is a baseline with a human-hours cost.

How agents use this

Before you train a router, log one week of traces and score always-search and a 20-line keyword file. Put those two numbers on the dashboard. The ML model’s only job is to beat them on the frozen test slice — and stay cheaper than another LLM call.

Last week’s prompt belongs on that dashboard forever. Regressions against it are incidents. Improvements against it are the only kind of modeling win that counts after you already beat majority.

Tip:A baseline is a product decision too. If keywords are 96% and the remainder is rare, ship the keywords.