Decoding

The model outputs logits. Decoding is the policy that turns those scores into the next token, again and again, until you stop.

This is not a footnote. The same weights can be a deterministic clerk or a chaotic poet depending on the sampler. Agents die here: a tool name sampled from a flat tail is a production incident; a JSON key sampled with high temperature is a parse error that looks like “the model is bad.”

You already have logits. This lesson is the policy. Greedy. Temperature. Top-k. Top-p. Stop rules. Repeat.

A wrong picture

A wrong picture is: “raise temperature to make the agent more autonomous.” Autonomy is the loop and the tools, not noise on the logits. Temperature reweights mistakes. It does not add knowledge.

Another wrong picture is: “greedy is always boring and therefore always worse.” Greedy is the mode. If the mode is the right tool name, you want it every time — a gift for tests. If the mode is the wrong tool, you will get that wrong tool every time — a curse until you fix the prompt. Sampling would have hidden the bug as flakiness.

A third wrong picture is: “stop when the answer feels done.” Stop when you see EOS, hit max tokens, match a stop string, or the schema is complete. Feelings are not a stop rule. A stop string that appears inside legal JSON will cut the model off mid-argument.

Greedy

Always pick argmax(logits). Reproducible. Tends to repeat and to choose safe, high-frequency wording. Good for JSON fields and tool names. Bad for variety in a poem. Agents that emit discrete actions want greedy (or a grammar that only allows legal tokens).

Greedy is still a distribution: it is the mode. Tests love it. Prompt bugs cannot hide behind a lucky sample.

Temperature

Divide logits by T before softmax.

• T near 0 approaches greedy (the max dominates)
• T = 1 uses the model’s native distribution
• T > 1 flattens; rare tokens get more mass

Temperature does not add knowledge. It reweights the same logits. At high T, a wrong tool name that had a small chance becomes a sometimes-event. For a router, that sometimes is an incident.

Top-k and top-p

Top-k: keep only the k highest logits, zero the rest (or drop them), then softmax and sample. Cuts the long tail of nonsense ids.

Top-p (nucleus): sort chances, keep the smallest set whose mass is at least p, drop the rest, renorm, sample. Adaptive: peaked rows keep few tiles; flat rows keep more.

Both are tail clips. They do not replace a schema. A grammar that only allows legal JSON is stronger than top-p = 0.9 when the job is a tool call.

Stop

Stop when:

• you sample EOS
• you hit max new tokens
• you match a stop string (after decode, or as ids)
• the structured object is complete (schema / grammar)

Agents should stop on schema complete, not on vibes. Greedy for tool JSON, a little temperature for the user-facing paragraph, is a deliberate split — two decode policies, one system.

Best-of-n samples multiply cost. Use them on hard items after a cheap filter, not on every turn. You are paying decode (and maybe prefill) n times.

Sample the same logits at three temperatures

Lists of numbers. A tiny vocab. Print greedy, chances, and samples. No f-strings. random.Random is seeded so you can rerun.

Run to execute this in your browser. Nothing is sent to a server.

At low T, search dominates. At high T, sql and even wait sneak in. For an agent router, that sneak is a production incident.

Rerun: greedy never changes. Samples at T=2 should look more mixed than at T=1 with the same seed start. That is the whole knob.

Two policies in one product

Tool names, enum fields, JSON keys: greedy or a constrained grammar. User-facing prose: a little temperature if you want variety, still clipped with top-p so you do not sample garbage ids.

Do not share one high temperature across both. The JSON will break and the router will wander. Split the decode policy even if you use one set of weights.

Repetition: greedy and low T can loop (“wait wait wait”). A repetition penalty is an extra decode hack. Summarizing the transcript is usually a better fix than a magic penalty — you are feeding better prefixes into the same logits.

Common mistakes

One global temperature for tools and prose. The router wanders. The JSON breaks. Split the policy even if the weights are one.

A stop string copied from a blog (} or \n\n) that also appears inside legal arguments. You will ship truncated objects and blame the model. Test stops against a valid tool call.

Top-p at 0.95 plus T at 1.2 plus no grammar, then wondering why the tool name is Searchh. The tail was invited. Clip it or forbid it.

Calling best-of-n on every turn “for quality.” You paid n times. A cheap filter plus one greedy decode would have caught the empty JSON. Save n for the hard slice.

Changing T to debug a prompt bug. The bug is in the prefix. Greedy makes it show every time. Sampling hides it. Debug greedy, then set T for the product.

How agents use this

Default chat temperature (often in the 0.7–1.0 rumor range) is for conversation. Tool calls want 0 or near 0, plus a JSON grammar if the stack offers it. Do not “increase temperature to make the agent more autonomous.” Autonomy is the loop and the tools, not noise on the logits.

Log the decode policy next to the trace: greedy or T, top-p, stop strings. Otherwise you cannot reproduce a bug. A test suite that samples T=1 will flake. A test suite that decodes tools greedily will catch prompt regressions.

• Discrete actions: greedy (or grammar).
• Prose: mild T, clipped tail.
• Stop: schema complete; stops that cannot appear in legal JSON.
• Tests: greedy so bugs cannot hide.
• Best-of-n: costs n; use after a cheap filter.

Watch out:Best-of-n samples multiply cost. Use them on hard items after a cheap filter, not on every turn.