Limitations

Transformers are extraordinary compressors of text. They are not oracles, clocks, or file systems. If you skip this lesson you will ship a demo that dies in week two.

The last lessons taught you how to adapt weights. This lesson is what adaptation will not buy. Hallucination lives in the objective. Cutoff lives in the calendar. Quadratic attention lives in the architecture. You already met each piece. Here they are as landmines.

The next track (hosted language-model products) is APIs, cost, and chat messages. Stay here until the machine is clear: tokens, attention, decoding, adapters. Then go.

A wrong picture

A wrong picture is: “a larger window removes retrieval.” Long context reduces the need to retrieve. It does not remove the need to select. Filling 200k tokens with junk is an expensive way to confuse the residual stream.

Another wrong picture is: “T=0 means true.” Decoding at temperature 0 makes the fake stable, not true. Greedy hallucination is a reproducible lie. That is worse for tests that check fluency, better for tests that check facts — you will fail every time until you ground.

A third wrong picture is: “chain-of-thought is a proof system.” It is more tokens of continuation. If the task is arithmetic, call a calculator. If the task is “what did the tool return?”, quote the observation.

Hallucination is in the objective

Next-token training rewards plausible continuation. When the prefix asks for a citation, a URL, or a Python API, the plausible continuation is often a well-formed fake. Fluency is not a truthfulness signal.

Mitigations live outside the architecture: retrieval with citations, tools that return real data, schemas, abstain phrases, evals that punish unsupported claims. The model can still lie about tool results if you let it; show the observation, then ask it to quote.

Fine-tuning on facts helps until the facts move. LoRA on a polite style does not install a ledger. Preference training can punish “I don’t know” if judges liked confident tone. Then you get confident fakes. Measure that.

Cutoff

Weights freeze. The world does not. “Who is the current CEO?” is a retrieval question after training day. Agents should have a date in context and a search tool, and should be allowed to say “I don’t know as of the tools I have.”

Adapters do not update the news. If you need today’s number, give a tool. Do not buy a larger window and hope.

Quadratic attention

Self-attention compares every token to every token: time and memory grow with n². Practical effects: longer context is not a linear bill; quality is uneven across the window; filling 200k tokens with junk is an expensive way to confuse the residual stream.

The KV cache and prefill lessons were this bill as latency. Here it is also quality: softmax dilutes; the middle fades. Architecture does not excuse a greedy packer.

Other landmines

• Reversal curse: trained on “A is B,” may fail “B is A.”
• Counting and arithmetic without a calculator: tokens are a bad abacus.
• “Ignore previous instructions” as a token pattern, not a legal system. The model continues a jailbreak if that continuation was plausible and you did not constrain the loop.
• Copy errors: UUIDs split into ugly tiles; attention drowned; positions in the dip.
• Forgetting after SFT: you raised JSON skill and lost a base skill. Eval general questions.

A tiny example in words

Naive attention cost n n. A retrieval-shaped cost is query length plus chunks chunk_len. As n grows, the square column becomes vicious. The retrieval column stays tied to how many chunks you admitted. That is why “just paste the wiki” is not a strategy even when the brochure says 128k.

n squared vs a small retrieved set

Print both columns. Lists of integer costs. No extra libraries.

Run to execute this in your browser. Nothing is sent to a server.

The n² column grows viciously. The retrieval column stays tied to how many chunks you admitted. Architecture does not excuse a greedy packer.

If you retrieve 400 chunks of 500 tokens, you built a new monster. Selection still matters. Top-k exists for a reason.

What adapters will not fix

A LoRA cannot make next-token training into a database. It can make the fake more on-brand. That is worse if you were hoping for truth.

A longer window cannot make cutoff into a news feed. It can hold a search result you fetched. Fetch it.

A bigger model cannot make n² cheap. It can make each pair more expensive. Pack anyway.

Preference training cannot make “I don’t know” popular if judges punished humility. If your taste data likes confidence, you will buy confident errors. Read the pairs before you train.

None of this says transformers are useless. It says the engine has a job: continue text. The agent has a job: constrain that continuation with tools, packing, decode policy, and evals. Mixing up those jobs is how week-two demos die.

Design for failure

Verify with tools. Keep humans on irreversible actions. Bound loops. Treat cutoff as a feature of the calendar. The transformer is the engine. The agent is the brakes.

If a task needs today’s number, a citation, or exact arithmetic, give the model a tool. Do not buy a larger window and hope. Do not buy a LoRA and hope. Do not buy temperature 0 and hope.

Stay in this track’s lane: you now know why those hopes fail (objective, positions, softmax, n²). The next track is how hosted products wrap the engine. The brakes are still yours: packing, decoding, tools, evals.

How agents use this

Design for failure: verify with tools, keep humans on irreversible actions, bound loops, treat cutoff as a feature of the calendar. Quote observations. Do not let the model narrate a tool result you never showed.

When something fails, name the landmine: hallucination (objective), cutoff (calendar), dilution (softmax / n²), reversal (A is B), arithmetic (no calculator), truncation (window). Named landmines beat “the model is being weird.”

• Ground: tools and retrieval, then quote.
• Date: cutoff is real; search is the patch.
• Select: do not fill the square with junk.
• Arithmetic: calculator.
• Brakes: loop bounds, human gates, evals.

Tip:If a task needs today’s number, a citation, or exact arithmetic, give the model a tool. Do not buy a larger window and hope.