Fine-Tuning

Fine-tuning continues training on your data so the model’s default behavior moves. It is not the first lever. Prompting, tools, and retrieval are cheaper and reversible. Fine-tune when the behavior must live in the weights: a dialect, a JSON shape the model keeps missing, a company voice at huge volume, or a small model you will run yourself.

The pretrained game is still next-token. Fine-tuning does not replace that game. It changes which continuations score high on your prefixes. If your demonstrations are mediocre, you will beat the model into mediocrity. If your facts change weekly, you will ship stale weights. If you never measured the base model on a frozen eval, you will not know whether the tune helped.

This lesson is full-weight (or full-last-layers) continuation: SFT and preference training. The next lesson is LoRA: the same idea as a small patch. Limitations after that are what neither will save you from.

A wrong picture

A wrong picture is: “fine-tune so it knows our PDFs.” That is retrieval, unless you like stale weights. Documents that change should not be burned into matrices. Show them at call time.

Another wrong picture is: “SFT teaches the model to be right.” SFT teaches imitation. If the expert sometimes searches twice, the model will too. If the traces contain a leaked key, the model will complete toward leaked keys.

A third wrong picture is: “preference training is a moral compass.” Humans (or a judge model) pick A better than B. The weights raise the winner relative to the loser. This can over-refuse, flatter, or game the judge. It is taste. It is not a proof of safety.

SFT (supervised fine-tuning)

Show (prompt, ideal completion) pairs. Loss is still next-token cross-entropy, usually only on the assistant tokens (not on the user prefix). This teaches format and style. It will also teach whatever mistakes are in the demonstrations. Ten thousand mediocre traces beat the model into mediocrity.

SFT is imitation. Masking the loss on user tokens is so the model does not spend capacity copying the user’s typos as if they were the target. You still need the prefix; you just do not ask the model to predict it.

Do not fine-tune if you do not have clean labels, if the facts change weekly (use retrieval), if you have not measured the base model on a frozen eval, or if you want to “make it know our PDFs.”

SFT on 100% tool JSON can make the model worse at ordinary chat. Mix in a little general data if the same weights must talk to humans. Eval before and after on your traces, a general instruction set, and a safety suite.

Preference (RLHF / DPO / cousins)

Humans (or a judge model) pick A better than B. The weights move to raise the winner’s likelihood relative to the loser. This teaches taste and safety that are hard to write as a single gold string.

For agents, preference data should include tool traces, not just final prose: “this call was the right tool” vs “this one invented an API.” If you only rank paragraphs, you will get nicer paragraphs that still call the wrong tool.

RLHF classically trains a reward model, then scores samples. DPO-style methods skip some of that machinery and push directly on pairs. You do not need the acronyms to use the idea: pairs of completions, one preferred. Garbage pairs become garbage taste.

It can over-refuse, flatter, or game the judge. Eval those failure modes. A model that never calls sql because a judge hated risk is not a safe agent. It is a broken router.

A tiny example in words

Toy vocab: search, sql, finish. Logits all 0.2. Gold is sql (index 1). Cross-entropy is high while the distribution is flat. Bump the gold logit. Loss drops. Argmax becomes sql. Real SFT is this idea on millions of tokens, with a small learning rate, on top of a frozen pretrain.

That bump is also how mistakes get burned in: if gold is the wrong tool in the dataset, you just made the wrong mode.

Bump the gold logit (toy SFT)

Lists of numbers. Cross-entropy by hand. Print before and after.

Run to execute this in your browser. Nothing is sent to a server.

The gold token became the mode. Loss fell. Probability of sql rose. That is imitation of one id. Imagine 50,000 traces that all retry a timed-out tool twice: you just taught a religion.

If you bumped the wrong index, argmax would move to a mistake. Curate. Do not dump production logs into SFT unreviewed.

When not to fine-tune

• Facts that move (prices, staff, tickets) — retrieve.
• A schema you can enforce with grammar and a better prompt — try that first.
• A safety rule that must be auditable — put it in code, not only in weights.
• No frozen eval — you cannot see forgetting.
• Tiny dirty data — you will overfit noise.

Catastrophic forgetting: a fine-tune can erase useful base skills. Eval general questions, not only your JSON. Mix a little general data if the same weights must stay a general assistant.

Keep tokenizer, prompt template, and weights in one bundle. Mixing adapter A with prompt B is a silent shift. The special-tokens lesson is still true after you train.

How agents use this

Fine-tuning an agent on its own unreviewed logs is how loops become religion. Curate. Prefer SFT for schemas and preference data for “don’t call shell on prod.” Keep tokenizer, prompt template, and weights in one bundle.

A tune that fixes JSON and starts offering medical advice you did not ask for is a failed train, even if schema accuracy went up. Eval both.

Do not skip prompting, tools, and retrieval because fine-tuning sounds more serious. Serious is measured. Fine-tuning is expensive to reverse. Prompts you can revert today.

• SFT: copies demos, including mistakes.
• Preference: copies taste; can game the judge.
• PDFs: usually retrieval, not weights.
• Bundle: tokenizer + template + weights.
• Eval: your traces, general skill, safety — before and after.

Watch out:A fine-tune can erase useful base skills (catastrophic forgetting). Eval general questions, not only your JSON.