A Linear Classifier

A linear classifier is the smallest real model: one score per class, or one score plus a cutoff for yes/no. It is not a toy you outgrow on day two. A huge amount of production routing is this shape: a list of features, a list of weights, a chance, a cutoff.

For two classes (urgent / not):

1. score = w * x + b (or a dot product if x is a list)
2. p = sigmoid(score) — a chance between 0 and 1
3. Predict 1 if p >= 0.5 (or another cutoff you pick on validation)

Sigmoid is 1 / (1 + exp(-score)). Large positive scores become chances near 1. Large negative scores become chances near 0. We clip the score in code so exp does not explode. That is numerics.

This is logistic regression. The name is old. The object is a linear score plus a smooth chance. You already know the training rule: take the slope of binary cross-entropy and step downhill. The slope for one example has a kind form: (p - y) * x for w, and p - y for b. You do not need to derive it at 2 a.m. You need to know it is just gradient descent on a classification loss.

For several tools, you use one score per tool (a logit), then softmax. Same idea: linear scores, then chances that sum to 1. The predicted tool is the largest chance, unless you use a cutoff to abstain when the top chance is weak.

What “linear” actually allows

A linear model can only draw a straight cut in feature space. If urgent tickets are “down or refund,” one weight on a single count may not be enough. Then you add features — an extra flag, a product of two flags you computed yourself — or a hidden layer (later). Adding a feature is often cheaper than adding depth.

Linearity is a feature-space statement. If you feed an embedding, the cut is straight in embedding space, which can be a curved story in English. That is why bag-of-words linear routers still work: the space is already a pile of useful flags.

If w goes to a huge number, the sigmoid saturates (p stuck at 0 or 1) and the slope dies. Smaller steps, or regularization, keep it trainable. Saturation looks like “accuracy froze.” Print p on a few rows.

Run to execute this in your browser. Nothing is sent to a server.

What printed: a fitted w (positive) and b (negative-ish). Then five rows. Low “down” counts get a small chance and predict 0. High counts get a large chance and predict 1. The middle of the table is where the cutoff matters. You just trained a router on one feature with 80 steps of gradient descent. No library. Lists of floats.

This run used all five rows as train. There is no test here. The print is to see the S-shape of sigmoid, not to publish a number. On a real freeze, you would fit on train and print this table on validation.

Cutoff is not training

The 0.5 in p >= 0.5 is a hyperparameter. Training put mass on the right class. The cutoff turns mass into a yes/no with costs. Rare classes and thresholds get their own page. Remember the split of jobs: weights from descent on train; cutoff from costs on validation.

Abstain is a second cutoff: if p is between 0.4 and 0.6, call a human or call the LLM. A linear router plus abstain is often the whole architecture in front of an expensive model.

From one number to a list

Replace w * x with a dot product: each feature has a weight. Bag-of-words is this. Embeddings are this (a long list). You can add a bias per class for multi-class. You still print chances. You still need a split. You still need a dummy.

Calibration (later) asks whether a printed 0.9 is right 90% of the time. Linear models are not automatically calibrated. Do not put p in the UI until you check.

Several tools, one linear map

A real router has more than urgent/not. You keep a weight list (or a row of a matrix) per tool. Features go in, one score per tool comes out, softmax turns scores into chances that add to 1, and you pick the largest chance — or you abstain if the top two are close.

That is still a linear classifier. The cut is a set of planes in feature space, one per pair of tools. If “refund” tickets look like a blob that is not linearly separable from “search” — mixed in the same bag-of-words region — no amount of extra epochs will draw a curve. You add a feature (a VIP flag, a cosine to the refund-policy chunk, a “already called search” flag) or you add a hidden layer.

Read the weights after a fit. A large positive weight on the word “select” for the sql class is a story you can test. A huge weight on a rare token that appeared once in train is overfitting you can delete. Linear models earn their keep because you can print the story. If you cannot name a feature that should matter, do not start with a net; start with better x.

The intercept b is the log-odds when all features are zero: the prior, roughly. If most tickets are search, b for search should help search win on a blank greeting. That is not a bug. That is majority class baked into the model. You still publish the majority dummy so people do not confuse “the intercept learned the base rate” with “we found a clever pattern.”

Common mistakes

• Treating tool id as x and fitting a line through 0, 1, 2, 3.
• Training with 0.5 cutoff in mind, then changing cutoff on test.
• Unscaled features so one column owns w.
• Calling it deep learning because you used sigmoid.
• No majority baseline next to the fitted accuracy.

How agents use this

A cheap tool router is often this: embed the user text (or use bag-of-words), multiply by a small weight list or matrix, softmax over tools. You can train it on a few thousand labeled traces and call it in milliseconds. Use the LLM when the router is unsure (chance of top tool below a cutoff). That cutoff is a hyperparameter.

Keep the router small enough to log. Print the score, the chance, the decision, and the features that fired. When it fails, you want “VIP flag and the word invoice pushed sql” not “the net was in a mood.” Linear models explain themselves if you do not hide the weights.

Note:If w goes to a huge number, the sigmoid saturates (p stuck at 0 or 1) and the slope dies. Smaller steps, or regularization, keep it trainable.