Why Transformers

Before transformers, the default sequence model was an RNN: read one token, update a hidden state, read the next. To mix token 1 with token 100, the signal had to survive 99 steps. Training was sequential: you could not compute step 50 before step 49.

Transformers mix any pair in one attention layer (in principle). Training scores all positions of the prefix in parallel (with the causal mask). Hardware likes that. That is the boring reason they won.

They are not magic at long distance: n² still hurts, and the middle still fades. They are a better default than a vanishing RNN state.

You now have tokens, attention, blocks, residuals, and decoding. This lesson is why this shape, so the stack in the next lesson feels inevitable instead of fashionable.

A wrong picture

A wrong picture is: “transformers won because they understand language.” They won because they train in parallel and can copy in one hop, at a cost of n². Understanding is a story we tell after the loss falls.

Another wrong picture is: “RNNs cannot do language at all.” They did, for years. They were slower to train on long sequences and worse at long copies. Transformers ate that lunch.

A third wrong picture is: “n² is solved, so architecture no longer matters.” State-space and linear-attention models try to keep the parallel train and drop the n². The stacks you call today are still mostly transformers. Packing still matters.

Three graphs of mix

CNN on text: local window. Token 1 sees 2 and 3, not 100, unless you stack many layers (slow hop).

RNN: a chain. Token 100’s hidden state is a function of 1…99 in order. Long copies must survive a long multiply chain. Train step 50 waits on step 49.

Transformer attention: a complete graph (minus the causal triangle). Token 100 can match token 1 in one layer. Train can score every prefix position at once given the true past (teacher forcing). Complete graphs are expensive. They are also how “the id in the JSON at the start” can copy into the tool call at the end without walking a chain of hidden states.

The MLP still thinks locally after the mix. The residual stream still carries the original embedding. You already met those parts. The new idea vs RNNs is the mix any pair plus parallel train.

A tiny example in words

For length n:

• RNN steps: n (must walk the chain)
• Attention pairs: n * n (or half of that with a causal triangle, still quadratic)
• Train positions in parallel: n (every prefix slot scored together)

Small n, RNN looks cheap. Large n, attention’s pair count explodes, but training still uses the GPU in parallel across positions. That trade is why transformers own language and why your 100k dump is expensive.

Count steps vs pairs vs parallel slots

Print a table of integer costs. No extra libraries.

Run to execute this in your browser. Nothing is sent to a server.

RNNs scale with depth of time; attention scales with pairs; training still parallel. The causal triangle is still quadratic. At 512, pairs are already huge compared with 512 RNN steps. Hardware still prefers the transformer train because those pairs are a big matmul, not a 512-step Python loop of hidden states.

When a tool argument must copy a UUID from the user message, attention is the mechanism that can copy it in one hop. When it fails, the UUID was too far, too drowned, or split into ugly tokens — not “the RNN died.” Fix the sequence and the tokenizer, then the copy.

What did not go away

Vanishing long-range mix is reduced, not deleted. The middle still fades. Softmax still dilutes. Positions still need a signal. You still pack a window.

CNNs are still useful for local patterns. RNNs still show up in small on-device models. Linear-time sequence models will keep trying to replace n². Your agent-engineering levers stay: tokens, packing, decode policy, adapters. Those levers do not vanish if the mixer changes.

Teacher forcing plus causal mask is why a transformer can train on a whole book in parallel and still be a valid next-token model. That pair (parallel train, honest generate) is the design win.

Common mistakes

Blaming “the RNN in the API” when a copy fails. Hosted chat is almost never an RNN. The copy failed because tiles split, the id drowned, or it sat in the middle. Reopen those lessons.

Assuming one-hop mix means one-hop reliability. In principle any pair can match. In practice softmax shares mass with thousands of other keys. One hop is permission, not a guarantee. Shorten the haystack.

Waiting for linear attention to make packing optional. Even if mix becomes linear, positions and dilution-like effects can remain. Your suitcase policy still pays.

Writing a toy RNN in the live box, seeing it copy a length-4 string, and concluding you do not need attention. Length 4 is not a UUID at position 0 copied into a tool call at position 800. The hop distance is the point.

How agents use this

When a tool argument must copy a UUID from the user message, attention is the mechanism that can copy it in one hop. When it fails, the UUID was too far, too drowned, or split into ugly tokens — not “the RNN died.” Fix the sequence and the tokenizer, then the copy.

Do not wait for a linear-attention product to save a greedy packer. The boring reason transformers won (parallel train, one-hop mix) is also why a fat prompt is a fat matmul. Shorten.

• One hop: any pair can match in a layer (in principle).
• Parallel train: all prefix positions scored together.
• n²: the bill; pack anyway.
• Not magic: middle fades; still pack.
• Copy bugs: sequence and tiles, not a vanished RNN.

Note:State-space and linear-attention models try to keep the parallel train and drop the n². The stacks you call today are still mostly transformers.