Prefill and Decode

One model call has two phases.

Prefill — run the whole prompt through the stack, fill the KV cache, produce logits for the next token. Cost grows with prompt length (and with n² attention unless the kernel is clever).

Decode — sample one token, append it, score the next, repeat until stop. Cost grows with output length. Each step is cheaper than prefill but you may do hundreds of them.

Time to first token is mostly prefill. Time to last token is prefill plus decode. Streaming shows decode as it happens; it does not shrink prefill. Users who say “the model is slow to start” are often holding a fat prompt. Users who say “it streams forever” are holding a large max-output (or a missing stop).

This is the KV-cache lesson as a clock. Same cache, now with a toy bill.

A wrong picture

A wrong picture is: “latency is max_tokens.” Output length is only the decode part. A two-sentence reply after a 40k-token dump is still slow to start.

Another wrong picture is: “streaming makes prefill cheaper.” Streaming changes when bytes arrive, not how much work prefill did. The first token still waits on the prompt.

A third wrong picture is: “speculative decoding and draft models make a junk prompt free.” They make decode faster by guessing ahead. They do not make a 200k junk prompt free.

Two clocks

TTFT (time to first token): dominated by prefill on long prompts. This is the pause before the cursor moves.

TTLT (time to last token): TTFT plus every decode step. This is when the JSON is complete.

Agent loops often emit short outputs (a tool name, a small object) after a long prompt (spec + tools + history). Then TTFT is the pain and TTLT is TTFT plus a blink. People still “optimize the sampler” and ignore the suitcase.

Parallel tool calls still serialize into one next prompt — one more prefill — unless you cache the shared prefix. Several tools in one turn can mean one fat observation block at the end. That is good for the cache head and bad for prompt length. Summarize tool dumps before the next prefill.

Output tokens are often priced higher than input tokens in hosted products. This track is not that product lesson. Still: decode is where the JSON is born, and long rambling decode is a bill you chose with max-tokens and stop rules.

A tiny example in words

Toy: prefill cost grows with n * n a little plus linear in n. Decode cost adds a bit per extra cache slot each new token. Doubling the prompt hurts first-token time more than doubling a short 80-token answer. That matches the “RAG dump feels slow to start” story.

The numbers are fake. The shape is the lesson.

Toy milliseconds for fat prompt, short answer

Lists of numbers as lengths. Print prefill vs decode. No extra libraries.

Run to execute this in your browser. Nothing is sent to a server.

Doubling the prompt hurts first-token latency more than doubling a short answer in this toy. Doubling output grows decode, not the start pause. A 500-token prompt prefill should look cheap next to 4000.

If your agent’s output is 20 tokens of JSON and the prompt is 8,000, do not start by cutting max-tokens from 256 to 64. Cut the prompt.

What to look at when the user waits

Look at prompt tokens, not only max_tokens. A router that runs a 2k-token spec every turn should keep that spec cacheable. A tool that returns 10k tokens of logs should summarize before the next prefill.

If first token is slow and prompt is huge: packing and cache. If first token is fine and the stream crawls: output length, stop rules, or a memory-bound cache. If both are slow: both.

Speculative decoding, draft models, and clever kernels are serving tricks. They do not excuse a greedy packer. Your lever from outside the weights is still the sequence and the stop.

Common mistakes

Cutting max output to “make it faster” when TTFT is the complaint. Max output is decode. TTFT is prefill. Cut the prompt.

Enabling a stream and calling the problem solved. The user still waits the same prefill. They just see a cursor sooner after it.

Logging only total latency. You cannot tell prefill from decode. Split the clock: time to first token, time to last, prompt tiles, output tiles. Then you know which lesson to reopen.

Running five tools, concatenating five fat dumps, and acting shocked the next call’s first token is slow. That next call is a new prefill. Summarize dumps before you concatenate.

Leaving max new tokens at 2048 for a router that should emit one enum. Decode will ramble if the stop never fires. Stop on schema complete.

How agents use this

If the user waits, look at prompt tokens, not only max output. A stable pinned prefix is a latency feature (KV-cache lesson). A tool that returns 10k tokens of logs should summarize before the next prefill. Parallel tool results still become one next prompt — one more prefill.

Reserve output room in the window (last lesson) so decode has space. Then stop so decode does not fill that room with poetry when you needed one JSON object.

• TTFT: mostly prefill; fat prompt.
• TTLT: prefill plus decode.
• Stream: does not shrink prefill.
• Short JSON, long prompt: optimize the prompt.
• Stop: schema complete, so decode ends.

Note:Output tokens are often priced higher than input tokens. Decode is where the poetry (and the JSON) is billed. Packing still dominates the start pause.