The Transformer Block

A transformer block (decoder style) is a residual pipeline. Tokens walk through many copies of the same block. Depth is “how many times we mix neighbors and think locally.”

Attention moves information between positions. The MLP (feed-forward net) processes each position. Residuals add the original stream back. LayerNorm (or RMSNorm) keeps sizes civilized. Unembedding at the end — next part of the track — maps the last vectors to vocab-sized logits.

GPT-2 small stacks 12 blocks. Frontier models stack many more. The interface stays: sequence of vectors in, sequence of vectors out, same length. You do not add or drop tokens inside a block. You only change the lists of numbers at each position.

A wrong picture

A wrong picture is: “each layer rewrites the sentence from scratch.” Residual style means each layer learns a delta. The original embedding still flows. A 96-layer model can still look like “the embeddings plus a pile of patches.”

Another wrong picture is: “more layers is automatically better at my tool schema.” After a point you are buying fluency, not obedience. Evals still rule. A deep stack on a missing spec still cannot attend to a sentence you never sent.

A third wrong picture is: “the MLP is another attention.” The MLP does not mix positions. It is the same small net applied at every index. Mix happens in attention. Think-locally happens in the MLP.

Diagram (top to bottom)

x  (one vector per token)
|--+
|  LayerNorm
|  Multi-head self-attention  (tokens mix)
|--+  add  (x = x + attn)
|--+
|  LayerNorm
|  Feed-forward MLP  (same token, wider, then back)
|--+  add  (x = x + mlp)
v
x'  to the next block

Two ideas make this trainable:

• Residual adds — the original x still flows. The block learns a delta, not a whole new picture from scratch.
• LayerNorm (or RMSNorm) — keeps vector sizes from exploding as depth grows.

Modern stacks usually normalize before attention and MLP (pre-norm), which trains more stably at depth. Older stacks normalized after (post-norm). You will not switch this from a prompt. You will remember that deep nets need a size-taming step.

Mixture-of-Experts replaces the MLP with several MLPs plus a router: more parameters without running every expert on every token. Your agent tool router is the same idea at a coarser grain. Failure modes rhyme: the router ignores an expert, or all traffic hits one expert.

A tiny example in words

Two tokens, width 2. Fake attention mixes 70% self and 30% the other token. Then a tiny MLP with ReLU expands to 4 numbers and back to 2. Residual adds after each step. The printed vectors should still resemble the inputs plus a nudge. That is residual style: identity plus a patch.

If the residual were missing, two nonlinear maps could smash the picture. With the residual, you can still see the old axes in the new lists.

Residual attention, then residual MLP

Lists of numbers. Small hand-written matrices. Print after each add.

Run to execute this in your browser. Nothing is sent to a server.

The printed vectors still resemble the inputs plus a nudge. Length of the sequence is still 2. Width is still 2. That is the block contract: same shape in and out.

After residual attention, each token has borrowed a bit of the other. After residual MLP, both lists get a nonlinear patch, still added on top. If a number looks huge, you are seeing why LayerNorm exists (next lesson). This toy is shallow enough to stay finite.

Depth is repeated mix-and-think

One block: mix neighbors, think locally. Two blocks: mix again, including things that only became visible after the first mix. That is how information can hop. In principle, L layers can move a fact about L hops. In practice residual streams and attention patterns are messier. Still, depth is repeated opportunity to mix, not a separate “reasoning module.”

When people say “the model reasoned in latent space,” they mean these residual streams were mixed again and again. You cannot inspect that cheaply from a typical hosted call. You can inspect the text you stuffed into the first embeddings. Garbage in the window is garbage in every block.

The last block’s vectors go to unembedding (later). Only then do you get logits over the vocab. Until that map, you still have lists of numbers, not words.

Common mistakes

Thinking of depth as “more thinking steps” the way a human would count. The model does not pause and reflect between blocks. Every block runs. Every token position gets a new vector. There is no skip-if-easy. If you want fewer steps, you want a smaller model or early-exit research, not a prompt that says “only use layer 3.”

Stuffing a chain-of-thought into the prompt and assuming that is the same as extra blocks. Extra tokens are extra keys for attention. Extra blocks are extra mix-and-think on whatever tokens you already sent. They are different knobs. You control tokens. You do not control block count on a hosted stack.

Copying a block diagram into twenty lines of Python without residuals or norm, then deciding transformers “do not work.” The diagram omitted the two ideas that make depth trainable. The next lesson is those two ideas slowly.

Expecting Mixture-of-Experts to pick your tool. The expert router picks MLPs inside a layer. Your tool router is a decode decision on ids. Do not conflate them. They rhyme. They are not wired together unless you built that.

How agents use this

When a call fails, people blame “the model.” Named parts of the block help you assign the failure:

• Wrong copy of an id: attention mix (and token split, and position).
• Fluent but empty: the MLP and the unembedding can still write pretty tokens.
• Policy ignored: often the policy tokens never entered x, or they drowned, not “layer 47 refused.”

You cannot peel one block off a hosted response. You can only change the text that becomes the first x. That is context engineering — already implied here.

More layers is not a substitute for a schema, a stop rule, or a tool that returns a real number. After a point you are buying fluency, not obedience. Evals still rule.

• Shape: sequence length does not change inside the block.
• Mix then think: attention across positions, MLP at each position.
• Delta: residuals keep the old stream.
• MoE: extra MLPs plus a router; same grain as a tool router, finer.
• Inspect: you inspect tokens in and tokens out, not the residual stream.

Watch out:More layers is not automatically smarter at your tool schema. After a point you are buying fluency, not obedience. Evals still rule.