LoRA

Full fine-tunes rewrite large matrices. LoRA (low-rank adaptation) freezes the base weights W and learns two small matrices A and B so the effective map is W + B A (times a scale). You store a patch of megabytes instead of a full copy of gigabytes. At runtime you add the patch (or merge it).

That is why you can keep one base model and swap LoRA adapters per customer or per skill.

Rank is a hyperparameter: rank 8 is a thin patch; rank 64 is thicker and more able to overfit. LoRA usually sits on attention and MLP projections, not on the whole net. You are not “training a new transformer.” You are training a delta in a few maps, residual-style, in parameter space.

Checkpoints: keep the base, the adapter, the tokenizer, and the prompt template together. An adapter trained for model X on tokenizer Y will look “broken” on model Z — it is in the wrong space.

A wrong picture

A wrong picture is: “LoRA replaces the tokenizer.” It does not. It patches linear maps. The vocab and merge list stay those of the base (unless you did something exotic you should not).

Another wrong picture is: “rank 64 is always better.” Thicker patches fit more, including noise. Rank is a capacity knob. Start small. Eval. Grow only if the eval says so.

A third wrong picture is: “a LoRA that raises JSON accuracy is done.” If it also starts offering medical advice you did not ask for, the train failed. Side effects are part of the eval. Same as full SFT, cheaper to store.

Low rank in words

A full matrix W might be 4096 by 4096. A rank-r patch is B (4096 by r) times A (r by 4096). If r is 8, you store two thin matrices, not the square. The product B A can only express a limited family of updates. That limitation is the point: fewer parameters, less to overfit, less to ship.

Forward pass: compute W x as usual (frozen), compute A x, then B times that, add. Or merge W' = W + B A into one matrix for serving if you do not need to swap adapters live.

QLoRA loads the base in 4-bit and still trains the small patch. Same idea, less memory. The math of the adapter does not change. The base is quantized; the patch is usually higher precision.

Where to attach: typically the query/value (and often other) projection matrices inside attention, plus MLP maps. Papers differ. You will not pick this per request. You pick it when you train.

A tiny example in words

W is a 3 by 3 identity. x = [1, 0, 1]. Base W x = [1, 0, 1]. A rank-1 patch: A is 1 by 3, B is 3 by 1. If A looks at the last slot and B writes into the middle slot, only dimension 1 picks up the adapter. The identity stayed; one direction gained a bump.

That is LoRA: freeze the big map, learn a small route that adds a delta.

Frozen W plus a rank-1 patch

Lists of numbers. Hand matrices. Print base, patch, and sum.

Run to execute this in your browser. Nothing is sent to a server.

The identity stayed; one direction gained a bump. x2 that does not touch the last slot should get a zero patch. The adapter only fires when A sees the feature it was trained to see. Real LoRA is this on bigger maps, with learned numbers, many ranks, many layers.

If you loaded this patch onto a different W shape, matvec would crash. That is the “wrong base model” error in miniature.

Swapping adapters

One base, many patches: staging vs production, customer A vs customer B, “SQL dialect” vs “polite chat.” Do not mix a production adapter that saw customer text into a staging toy. Do not mix an adapter trained on model family X onto family Z.

Merging a patch into W is convenient when you serve one skill. Keeping patches separate is convenient when you swap. Either way the bundle is base + adapter + tokenizer + template.

You can often get the same schema win with constrained decoding and a better prompt. LoRA earns its keep at volume, latency, or a style you cannot prompt into a small model. Try the cheap lever first. Measure.

Common mistakes

Shipping an adapter without the base commit hash. Six weeks later nobody knows which W it was trained against. The patch loads, the shapes match, the generations are garbage because one layer width changed. Put the base id in the adapter filename and in the eval log.

Training on a mix of two chat templates. The special-token ids differ. The adapter learns a dialect that is neither wrap. At run time you pick one template and lose half the gain. Freeze the wrap before the first step of training.

Using rank as a status symbol. Rank 64 on 200 messy traces will memorize the mess. Rank 8 on 5,000 clean traces will often emit the schema more reliably. Capacity is not quality. Quality is the frozen eval plus a side-effect suite.

Merging a LoRA into W and then applying a second LoRA that was trained against the unmerged base. The deltas were not meant to stack that way. If you must combine skills, train a single patch on mixed data, or keep them as separate loads you do not add blindly.

Forgetting that QLoRA’s 4-bit base is an approximation. Tiny numeric drift is normal. Huge behavior drift means the quantization or the pack is wrong. Compare a short greedy completion on the full-precision base plus adapter versus the 4-bit path before you ship.

How agents use this

A LoRA per environment is reasonable: a staging adapter should not contain production customer text. A LoRA that fixes schema and starts offering medical advice you did not ask for is a failed train, even if JSON accuracy went up. Eval both.

Store the bundle. Mixing adapter A with prompt B is a silent shift (special tokens still matter). Mixing adapter A with tokenizer C is a broken lookup table.

• Patch, not a second full model: W + B A.
• Rank: capacity vs overfit.
• Bundle: base, adapter, tokenizer, template.
• Swap: one frozen base, many skills.
• Eval side effects: schema wins that break safety are failures.

Note:You can often get the same schema win with constrained decoding and a better prompt. LoRA earns its keep at volume, latency, or a style you cannot prompt into a small model.