Train vs Inference

Training is the loop that changes parameters: forward, loss, backward, step. Inference (also called serving or eval mode) is the forward pass with knobs frozen.

When you call a vendor LLM, you are buying inference. Fine-tuning is when someone runs training on your data. Prompting is neither: you are changing the input, not the weights. It can still overfit an eval. It can still leak a test conversation into a few-shot. It is not “the model learned from this ticket” unless weights moved.

This distinction is how you log a change. If you do not know whether you changed weights, input, index, or code, you cannot debug which one broke Tuesday.

What is frozen

At inference:

• Dropout is off
• Batch-norm uses stored stats
• You do not compute slopes
• You may still sample (temperature, top-p) — that is randomness on the output, not learning
• Weights do not update because a customer was unhappy

At training you shuffle (train only), you compute loss, you step. Mixing a test row into a gradient is leakage with extra ceremony. Mixing a production ticket into a fine-tune without a split is how you un-test your own exam.

model.eval() in frameworks means inference mode. Forgetting it leaves dropout on and makes traces jitter. The same policy, same input, different dropped units: you will chase a ghost.

Sampling is not training. Temperature 0.8 does not update W. It reshapes chances and draws. Run the same frozen model twice and you may get two answers. That variance is inference. Logging the seed and temperature is how you make it less mysterious.

Run to execute this in your browser. Nothing is sent to a server.

What printed: infer score 0.8 from the frozen weights (1.0 1 + -0.5 0 + -0.2). After a fake training step on a copy, the score drops to 0.7 because the first weight became 0.9. The last line shows original W still [1.0, -0.5] and the original score still 0.8. Inference leaves the table untouched — like a production model while you experiment in a notebook.

If production wrote back into W on every ticket, you would have an unsupervised, unbounded, unsplit “online learning” loop. That is a research product with a safety story, not a default. Frozen weights plus logged traces is the default.

Cost

Training needs labels, machines, and a loop. Inference needs a machine (or an API bill) per call. Agents spend almost all money on inference: every tool-planning token is a forward pass. A small local classifier in front of the LLM is a bet that cheap inference can skip expensive inference.

Fine-tunes have a second cost: you must re-eval when the base model vendor moves, and you must watch drift. You bought a new set of weights that will go stale. Inference-only prompting goes stale too, but you can swap a prompt faster than you can rebuild a fine-tune — if you have a frozen eval.

Four kinds of change

Log whether a change was:

• Weights (fine-tune)
• Prompt (input)
• Index (retrieval corpus or encoder)
• Code (tools, parser, cutoff)

Those four fail differently. A fine-tune that looked good offline can still be the wrong move if the prompt and the index were the real bugs — you trained on a symptom. A prompt fix will not repair a stale index. A code fix (schema check) is often the cheapest “model improvement.”

Caching, batches, and “did it learn from this user?”

Inference can still look like learning because of caches and memory stores. A retrieval index that appended today’s ticket is not training the LLM. It is changing the index (one of the four change types). A conversation buffer that carries the last tool result into the next prompt is not training. It is a longer input. Users will still say “the agent learned.” Log the truth: frozen weights, growing context, maybe a growing index.

Batching at inference packs several requests into one forward pass for speed. It does not mix their gradients; there are no gradients. It can mix their timing and their max-token settings if you are sloppy. Keep batching as a serving trick, not as a training story.

Online learning — update weights on every live ticket — needs a split you no longer have, a reward you trust, and a rollback. Default off. If a vendor offers “the model improves from your traffic,” ask whether weights move, whether you can freeze a version, and whether your test ids are excluded. If those answers are vague, treat it as uncontrolled training.

Eval mode vs train mode is not academic. Dropout on at serving makes the same ticket bounce between tools. Batch-norm using batch stats instead of stored stats makes a singleton request look unlike training. For small routers you ship, a unit test that the same x twice yields the same p (with temperature 0) catches this. Sampling models will not be bit-identical; then compare chances, not the drawn token.

Common mistakes

• Calling prompting “training.”
• Leaving dropout on in production.
• Fine-tuning on traces that include the test ids.
• Assuming an API call updates the vendor’s weights on your data.
• One changelog that says “improved the agent” with no which-of-four.

How agents use this

Treat production as inference of a frozen policy. Collect traces. Fit offline on train. Choose on validation. Deploy new weights or a new prompt as a versioned inference artifact. Do not learn on the live path unless you have a separate design for that (and you still need an eval).

When someone asks “did we train on that customer?”, the honest answers are: we ran inference; we stored the trace; we might later fine-tune on a sampled, consented, split dataset. Mixing those sentences is how policy teams lose trust.

Note:model.eval() in frameworks means inference mode. Forgetting it leaves dropout on and makes traces jitter.