Rules, Prompts, or Train

Not every prediction problem wants a neural net. Agents get expensive when teams skip the cheap rungs.

1. Rule. Keywords, schemas, allow-lists. Fast, explainable, easy to test. Ship this if it already beats the bar.

2. Prompt. Frozen LLM, better instructions, few-shot traces. You are doing inference plus hyperparameter search. Good when the task is language-heavy and volume is low.

3. Retrieve. k-NN / RAG when the answer lives in docs that change. You train an index, not a net.

4. Small model. Linear router, tiny classifier, reranker. Train when you have labels, a stable task, and a latency/cost budget the LLM misses.

5. Fine-tune the big model. Last. Needs more data, more eval, more drift pain. Do it when 1–4 cannot close the gap.

The ladder is not a vibe. It is measured on a frozen slice: dummy, rule, last week’s prompt, retrieve, small model, then maybe a fine-tune. Promote a problem up only when the lower rung’s metric, cost, and latency lose. Demote when a new API makes a rule possible again (password reset should not be an agent forever).

Fine-tuning will not fix a broken tool, a stale index, or a fuzzy goal. Train last among those bugs.

Measure the remainder

In production, measure the remainder: tickets the rule gets wrong. If that remainder is 2% and random, you need better labels, not gradient descent. If it is 30% and patterned, then train — or write a better rule for that pattern.

Few-shot examples in a prompt are training data. They leak if they also sit in the test conversations. Version them. They also have a capacity cost: more tokens, more chances to memorize last week.

Run to execute this in your browser. Nothing is sent to a server.

What printed: always-search accuracy 2/6 (only the invoice rows). Keyword rule 1.0 on these six. Then the reminder: a 6-row neural net cannot honestly beat a rule that is already perfect here. Training a net on this table is theater. The remainder is empty. There is nothing to fit.

In a real freeze the rule would be written from train and scored on test, and 1.0 would be suspicious. The lesson stands: beat the next cheaper rung, do not skip rungs.

Stability and labels

“Stable” means the task will still exist next month with the same action space. If the tool list changes weekly, a trained router needs weekly labels. A rule that reads the tool registry may keep up cheaper.

“Labeled” means honest y, a rubric, agreement, a split. Weak labels from the current agent are how you clone its mistakes. Sample for humans before you climb to rung 4.

Cost belongs on the ladder. An LLM prompt that is 2 points better than a linear router and 50 times more expensive is a product choice. Training a small model is often the move that saves inference money, not a science project.

Worked remainder, and when to demote

Imagine 10,000 tickets. Always-search is 62%. Keywords hit 91%. The 9% remainder is not random: a cluster of “how many / revenue / count” that the rule missed because users said “headcount,” and a cluster of password-adjacent phrases without the word password. You now have two patterned remainders. For headcount, add a word to the rule or a bag-of-words feature. For password-adjacent, a tiny supervised router on a few hundred labels may beat another year of prompt clauses. You do not fine-tune the big model on 10,000 traces to avoid adding “headcount.”

Retrieve when the answer moves with documents: policy PDFs, ticket history, SKU tables. Prompt when the procedure is language-heavy and rare. Rule when the action is obvious from a string or a schema. Small model when the remainder is large, stable, and you can label it. Fine-tune when 1–4 lose on the freeze and you can afford drift and eval cost.

Demote aggressively. If finance ships a reset_password API, the agent should call a workflow, not a 12-step search. That is a product win that looks like a modeling loss on last quarter’s taxonomy. Bump the dataset version. The ladder is allowed to go down.

Volume matters. A task with 20 tickets a week will never grow an honest train set. Prompt plus a rule. A task with 20,000 a week and a 15% remainder is where a linear router pays rent.

Common mistakes

• Fine-tuning to avoid writing an allow-list.
• A prompt with 40 few-shots that are the test set.
• Training on six rows because a blog used a neural net.
• Never demoting: password reset still goes through a 12-step agent.
• Climbing the ladder without a remainder dashboard.

How agents use this

Write the ladder in the design doc: rule → prompt → retrieve → small router → fine-tune. Each rung has an owner, a metric, and a freeze. The scientific method from lesson one is this ladder plus a split.

When someone says “we should train,” the questions are: what is the dummy, what is the rule, what is the remainder, how many honest labels, is the task stable, what does it cost if we only prompt? If those have no answers, you are not ready to train. You are ready to measure.

Watch out:Fine-tuning will not fix a broken tool, a stale index, or a fuzzy goal. Train last among those bugs.