The Stack an Agent Actually Runs

You now have the machine. An agent call is this pipeline:

1. Chat template wraps roles into special tokens
2. Tokenizer turns text into ids
3. Embedding table + positions turn ids into vectors
4. N blocks mix (attention) and think (MLP) with residuals
5. Unembedding turns the last vector into logits
6. Decode picks a token (greedy for tools, sampled for prose)
7. KV cache stores past keys/values; repeat 6 until EOS or stop
8. Ids map back to text (a tool call, a sentence, a refusal)

You do not configure heads at runtime. You configure the sequence that becomes step 1–2, the sampler at step 6, and the stop / schema that ends step 7.

This is the recap lesson. Stay in this track’s lane: the machine. Hosted product choices (which vendor, which price table, which HTTP shape) belong to the next track. Here you assign failures to named steps.

A wrong picture

A wrong picture is: “the model is a blob; when it fails, switch blobs.” Sometimes you should switch size or adapter. Often the failure is packing, template, decode, or stop. Named steps beat blob-swapping.

Another wrong picture is: “I can inspect the residual stream from a typical hosted call.” You cannot. You can always inspect the tokens you sent and the tokens you got. Start there.

A third wrong picture is: “autonomy is temperature.” Autonomy is the loop, the tools, and the stop rules. Temperature is decode noise. You already paid for that lesson. Do not forget it at the recap.

What you control vs what is frozen

Frozen in a hosted stack: vocab, merges, embedding rows, block weights, LayerNorm, head count, RoPE, unembedding. Maybe an adapter you loaded. You do not set LayerNorm eps per ticket.

You control:

• which strings enter the template
• whether the template matches the model
• packing (pin, tail, refuse if must-have does not fit)
• decode policy (greedy tools, milder prose)
• stop / schema / max new tokens
• whether the prefix is byte-stable for the cache
• whether you adapt (SFT / LoRA) on curated data — off to the side, not per request

If those are green and it still fails, then change model size, LoRA, or stack. The next track is that product layer.

Assign the failure to a step

Draw this pipeline on the design doc. Assign each failure to a step. Named steps beat vibes.

Same observation, different prefix

A cartoon function: pinned spec vs dropped spec. Fake logits over three actions. Lists of strings as toy tiles. No hosted client.

Run to execute this in your browser. Nothing is sent to a server.

Same observation, different prefix. The stack did not grow morals. The pinned tokens changed the logits. Noisy decode ignored the better mode and picked search. That is this whole track in one function: sequence, then sampler.

When the spec is dropped, sql can stay the mode and “delete them” looks like a sql job. Pinning flipped finish up and sql down. Packing is safety. Decode is whether you take the mode.

Checklist before you blame “the model”

• Tokenizer and template match the model
• Spec is pinned and still in the window
• Latest observation is at the end, not drowned
• Tool names decode greedily (or with a grammar)
• Stop rules cannot fire inside legal JSON
• Cache-friendly prefix is stable across turns
• You measured on a frozen eval, not the demo
• Pad/collator is clean if you trained
• Adapter bundle includes tokenizer and template
• Facts that move go to tools/retrieval, not a hope in the weights

If those are green and it still fails, then change model size, LoRA, or vendor stack. You earned that move.

Common mistakes

Skipping the checklist and swapping models first. You spend a week and keep the silent truncation.

Logging only the decoded string. You cannot see whether the pin was present, whether decode was greedy, whether the stop fired early. Log tiles in, tiles out, decode policy, and “was spec prefix still there.”

Treating this recap as an API lesson. HTTP shapes, price tables, and vendor SDKs are the next track. This track is why those calls are slow, why JSON truncates, and why a LoRA did not install a ledger.

Editing the spec to “be helpful” mid-incident. You miss the cache and you move the policy. Freeze the pin. Change the tail.

How agents use this

Draw this pipeline on the design doc. Assign each failure to a step. “Hallucinated a tool” is often decode + missing schema (step 6–7). “Forgot the policy” is often truncation (step 1–2). “Slow” is often prefill on a fat prompt (cache). Named steps beat vibes.

You cannot inspect the residual stream from a typical hosted call. You can always inspect the tokens you sent and the tokens you got. Start there. Count them. Print the pin. Print the last observation. Print greedy vs sampled. Then change one thing.

The next track is large language models as a product: chat wraps, cost, choosing a hosted brain. This track was the engine: tokens, attention, decoding, adapters. Keep the engine picture when the product layer gets loud.

• Sequence: template, tokenize, pack.
• Mix: embeddings, positions, blocks.
• Pick: unembed, decode, stop, cache.
• Adapt off to the side: SFT / LoRA, bundled.
• You: packing, sampler, stops, evals.

Tip:You cannot inspect the residual stream from a typical hosted call. You can always inspect the tokens you sent and the tokens you got. Start there.