Decoding Knobs

You already saw temperature in the Transformers track as a math knob on next-token scores. In an API, it is a product knob, sitting next to max_tokens, stop sequences, and sometimes seed, top_p, and penalties. Wrong knobs look like model failures. They are often configuration.

Change one knob per eval run. If you twist temperature and top_p and the spec on the same day, you will not know what fixed JSON.

Temperature and top_p

For tool calls and JSON, use temperature 0 (or the vendor’s minimum) and native structured output. For user-facing prose, 0.3–0.7 is a common band. Above 1.0 is a party trick, not an agent policy.

If you set temperature and a tight top_p, you can accidentally double-cut the tail. top_p=0.9 keeps the smallest set of tokens whose probabilities sum to 90% and drops the rest. Combined with temperature 0 it does almost nothing (the mass is already a spike). Combined with temperature 1.5 it is the only thing between you and noise. Change one variable at a time on an eval set.

This lesson will not re-teach softmax. You need the product rule: JSON wants argmax-like decoding. Creativity belongs in user-facing prose, if anywhere.

max_tokens and stops

max_tokens is the completion budget, not the window. A 128k window with max_tokens=16 still cannot emit a long JSON object. A 8k window with max_tokens=4096 can ramble until you pay for the tail.

Stop sequences cut off when the model starts imitating your scaffold (User:). Frequency and presence penalties reduce repetition in stories. They can also wreck code and repeated ids (job_id mentioned twice). For agents, prefer stop sequences, max tokens, and lower temperature — not a penalty that fights legitimate repetition.

Seeds are best effort, not a scientific guarantee under load. Evals should tolerate small drift or use greedy plus structured tools. Do not advertise “deterministic” to legal if the vendor says seed is best effort.

Store knobs per step type: router temperature 0, writer 0.5, tool max_tokens 200, final 800. One global temperature for the whole agent is how JSON grows poetry. When latency spikes, check whether max_tokens is 4096 “just in case” — you pay for the long tail of rambling, and you wait for it if you are not streaming.

Run to execute this in your browser. Nothing is sent to a server.

Greedy picks index 1 (the 0.8 peak) — the JSON/tool default. The toy sample walks until it passes 35% of the mass and may pick an earlier, less likely index. The numbers are a cartoon of temperature vs argmax, not a vendor-accurate sampler. The print line is the policy: per step type. A router and a poet should not share a config struct.

Penalties, n, and eval

Presence/frequency penalties fight loops in stories. They also fight legitimate repetition of job_id. Leave them off for tool steps unless you measured a win.

n>1 is an eval trick: generate several completions, score with your validator, pick a winner. Using another LLM as a judge is a later, loaded choice (Eval track). Do not do it on the hot path by default.

What goes wrong

• Playground defaults copied into prod (temperature 1, max tokens huge).
• Creative JSON. Extra keys, invented enums.
• Stops that appear inside URLs or JSON strings.
• Tuning knobs instead of the spec when the model lacks a tool.
• Declaring reproducibility from a seed on a busy hosted API.

How agents use this

A config map: step_kind -> {temperature, max_tokens, stop, top_p}. Tool kinds: 0 and 200. Writer kinds: 0.5 and 800. Log knobs on the span with the model name. When JSON fail rate spikes, look at knobs and spec version before you swap vendors.

Tests do not need a real sampler. Tests need: the HTTP wrapper actually sends temperature 0 on tool steps. A miswired default is a common bug.

Do not copy playground defaults. Playgrounds are for humans exploring. Agents are a policy. If a new hire pastes temperature: 1 because the docs example used it, your JSON step is now a creative writer. Code review the knobs like you review SQL.

top_p: leave it unset (vendor default) unless you have a measured reason. Setting 0.9 and temperature 0 is cargo cult. Setting 0.9 and temperature 1.5 is a sampler you must eval. Penalties: off for tools. Seeds: log them if you send them; do not promise bit-identical replays on a hosted API.

Latency: max_tokens is an upper bound the model can use. Some vendors bill only what was produced; you still wait through a ramble if you are not streaming and you asked for 4096. Tool steps should not wait on a novel.

Store knobs per step kind in config, not in comments in a notebook. When JSON fails, diff knobs on the span against last week’s baseline before you blame the weights. A Tuesday bump in temperature is a deploy, not a mystery.

JSON steps: temperature 0, small max_tokens, native schema if you have it. Prose steps: a modest temperature, a larger cap, stops that match your scaffold. Never one global struct named LLM_DEFAULTS copied from a playground.

Note:Change one knob per eval run. n>1 is for offline eval unless you measured a hot-path win. For agents, prefer stop sequences and max tokens over frequency penalties.