Supervised Learning
Every training row has an input x and a target y. No labels, no supervised learning.
Supervised learning means every training example has an input x and a target y. The algorithm’s job is to predict y from x for new rows.
The word is literal: a supervisor (human, script, or downstream system) provided the answers. No labels, no supervised learning. You may still cluster or embed. You may not claim you “trained a classifier.” You may not treat the agent’s own guesses as gold and then celebrate that the agent agrees with itself.
x is what is known at decision time. y is the decision you wish the system had made, or the number you wish it had estimated. Get those two aligned and the rest of the track is search and measurement. Get them wrong and you will fit a function that answers a question nobody asked.
Classification vs regression
| Classification | Regression | |
|---|---|---|
y | A category (spam / ham, tool A / tool B) | A number (latency, price, 1–5 stars) |
| Typical loss | Cross-entropy | Mean squared error |
| Output | Class or chances | Real value |
| Agent examples | Route this ticket; pick a tool | Estimate tokens left; score a trace |
Do not turn regression into classification without a reason. “Latency > 800ms” as a yes/no throws away how late it was. Do not predict 3.7 for a tool id either: that is not a tool call. Ordered classes (1–5 stars) sit in the middle: sometimes a regression, sometimes five classes. Pick from the product, not from habit.
Binary classification is two labels (urgent / not). Multi-class means more than two labels, one of them true (which of eight tools). Multi-label means several can be true at once (billing and outage). Those are different schemas. Mixing them is a silent eval bug: a multi-label ticket scored as if only one tool were allowed will look like model failure when it is schema failure.
If the action space is “call one tool, or finish, or ask a human,” that is multi-class. If the agent may legally call search and sql in one step, that is multi-label or a sequence of decisions. Write the action space down before you label.
What a label is
A label is a decision you wish the system had made:
- The tool a senior engineer would have called
- Whether the final answer is supported by a source
- The JSON that passed the schema
- A thumbs-down from a user
- Whether a human should have been in the loop
Labels are not “the meaning of the ticket.” They are actions and judgments. If two labelers disagree 30% of the time, the ceiling of your model is near 70%, not 100%. You cannot train your way past a fuzzy rubric. You fix the rubric, add abstain, or split the label into two questions.
Agreement is a measurement: two people (or a person and a written spec) on the same sample. If they disagree, modeling is early. If they agree but the labels still feel wrong in production, the rubric does not match the product.
Ambiguous items belong in a third bucket — abstain or ask_human. Forcing a hard class invents noise. Noise looks like a model that “regressed overnight” after a labeling pass.
Noisy labels and weak labels
A noisy label is a lie or a coin flip: the wrong tool tagged, a script that used the current agent as truth, a row where nobody read the policy. A weak label is a cheap hint: the agent’s own action, a keyword, a user thumbs-up. Weak labels can start a dataset. They cannot finish one. Sample for humans.
Do not label data with the agent itself and then treat that as gold. That amplifies the first version’s bias. Every systematic mistake becomes “the right answer” and the next model copies it harder.
Supervised means someone wrote y. Clustering is a different job until you promote a blob into a label.
Each row is an input and a targetRun to execute this in your browser. Nothing is sent to a server.
What printed: clean acc 1.0. noisy acc 0.83 (one of six targets is a lie, so 5/6). Then the sentence that the policy did not get worse — the labels did. The keyword rule is the same function. The score dropped because one y was wrong. That is how noisy agent traces look on a dashboard: the model “regressed” overnight after a bad labeling pass, or after someone used the agent’s own tool calls as gold.
In a real freeze you would not score the rule on the same six rows you used to invent it. The print is about labels, not about validation protocol.
Where labels come from in an agent stack
Good sources: a written rubric plus humans; a replay against a fixture world (goal_satisfied); a schema checker; a forbidden-tool list; an exact match to a required citation id.
Bad sources: “whatever the LLM did last week”; thumbs-up without looking at tools; the retrieval corpus as if every neighbor were relevant; mixing several raters’ private scales.
If you can compute y with a script (JSON valid, required substring present, tool in allow-list), do that first. Humans are for leftover judgment. Script labels still need a split: the script can overfit a prompt if you tune the prompt on the same traces.
Common mistakes
- Calling clustering “supervised” because you named the clusters after the fact.
- Predicting a tool id as a real number.
- Multi-label tickets scored with single-class accuracy.
- A model ceiling of 100% on a task where people agree 70%.
- Training on agent outputs and calling it ground truth.
How agents use this
Every router in front of a big LLM is a supervised classifier, even if you implement it with embeddings. Define the label set as your action space (search, code, ask_human, final). If a label is not an action you can run, it does not belong.
Six demo examples are a story, not a result. Next pages give you a real loss, a search over knobs, and metrics that do not lie when one class is rare. Supervised learning only starts when x and y are honest. The function you pick is secondary.
Note:Six demo examples are a story, not a result. Next: a real loss.
Check your understanding