Sampling and Temperature

Temperature T > 0 rescales logits before softmax: use z_i / T.

• T → 0: the largest logit dominates; you approach greedy argmax (repeatable, given ties).
• T = 1: the model’s native distribution.
• T > 1: the distribution flattens toward uniform; more entropy, more chance of a weird token.

Temperature does not “add creativity” as a slogan. It changes the categorical you sample. At high T you spend entropy; at low T you copy the mode.

Sampling: draw u uniform on [0,1), walk the cumulative sum of p_i, pick the first index where the cumulative meets u. That is the same walk as the cost-sampler in the expectation lesson.

Tiny walk: chances [0.50, 0.30, 0.20]. Draw u = 0.55. After index 0 the cumulative is 0.50 (not enough). After index 1 it is 0.80 (enough). You pick index 1. If u = 0.02, you pick index 0. If u = 0.99, you pick index 2. The last index is a safety net when rounding leaves a hair of mass unused.

A wrong picture

A wrong picture is: “temperature changes which token has the biggest logit.” It does not. Argmax of logits is invariant to dividing by T > 0. Softmax chances change; the winner of greedy does not (ties aside). Raising T does not flip the softmax winner. It flattens chances so samples diversify.

Another wrong picture is: “high T is smarter” or “low T is always safer.” Low T is repeatable, which evals love, and it can trap loops (“I’ll search again”). High T makes a ditherer: more wasted tools, more weird tokens. Moderate T plus a stop tool and a max-step budget is a typical compromise.

A third: comparing two prompts at T>0 without a seed (or without many draws). “Which prompt won?” is then a coin flip. Seed evals when you compare prompts. Score the distribution of plans, not a single lucky run.

Clip T above a tiny floor so you never divide by 0.

The formula in words

Scale logits by 1/T. Softmax. Draw.

Tiny numeric. Logits [2.0, 1.0, 0.2, -1.0] for search, sql, finish, wait. At T=0.2 the gap 2 vs 1 becomes 10 vs 5; search eats almost all mass. At T=2 the gap becomes 1 vs 0.5; sql, finish, even wait show up in 12 samples. Same logits, different agent. If your production temperature is accidentally 2.0, “the model got worse” is a sampling bug.

Greedy: pick search every time from these logits.

Moving parts

Argmax of logits does not move when you change T > 0. The chances move. Samples follow the chances.

A second walkthrough (top-k and top-p)

Start from chances [0.50, 0.30, 0.15, 0.05] on search, sql, finish, wait.

Top-k = 2: keep the two largest, drop finish and wait, renormalize. Remaining mass is 0.80. New chances [0.50/0.80, 0.30/0.80] = [0.625, 0.375]. You will never sample finish. The tail is gone.

Top-p = 0.90 (nucleus): walk largest-first until the cumulative meets 0.90. Sorted: 0.50, then 0.80, then 0.95. After three tokens you have 0.95 ≥ 0.90, so the nucleus is search+sql+finish. Drop wait (0.05). Renormalize by 0.95: about [0.526, 0.316, 0.158]. Wait is gone; finish still lives.

Draw u = 0.55 on the original list: cumulative 0.50 (not enough), 0.80 (enough) → sql. Same u after top-k=2: cumulative 0.625 (enough) → still sql. Same u is not a different universe if the cutoff did not remove the winner of that u.

Tie at greedy: logits [2.0, 2.0, 0.0]. Two winners. Pick a rule (first index, or prefer finish). Do not leave it to “whatever Python’s max does on a list.”

A Friday ticket

Friday eval bake-off. Prompt A beat prompt B on 12 tickets at T = 0.8 with no seed. Monday they reran. B won. The logits had not changed. The draws had. After they seeded 20 traces per prompt, pass rates matched within a few points. The “winner” had been one lucky u.

A second Friday: production temperature was accidentally 2.0 after a config merge (default in a client library). Tool entropy jumped. Wait and finish showed up on refund tickets. The logits were the same as Thursday. Sampling was not.

Implement softmax with T and a draw

Run to execute this in your browser. Nothing is sent to a server.

At T = 0.2 almost every sample is search (chance about 0.993 on search, tiny on the rest). At T = 1 search still leads but sql appears. At T = 2 you will see sql, finish, even wait. Greedy prints search. Seed 3 makes the 12-draw lists repeatable. Rerun with another seed to see T>0 move; greedy will not.

If T is 0.2 and you still see lots of wait, you did not divide logits (you might have multiplied). Check the chance table before the samples.

What goes wrong

• T = 0: division by zero. Clip T above a tiny floor (for example 1e-5) if you want “almost greedy.” True greedy is argmax, not softmax at T=0.
• Multiplied instead of divided: T=2 should flatten. If it peaks harder, you used z * T. Print the chance table before you look at samples.
• Cutoff then forgot to renormalize: remaining mass is 0.80, but you still draw u against the old list. Some u land in the dropped tail and hit the safety-net last index. Always renormalize after top-k / top-p.
• Ties: two equal max logits. Greedy needs a written tie-break. Sampling will split them 50/50 at any T.
• Unseeded evals: one lucky u is not a prompt winner. Seed, or average many draws.

Production logs: T, seed (or unseeded), chosen token/tool, p_chosen, and entropy of p. Assert T > 0, unit sum after softmax and after any nucleus cutoff, and that the last-index safety net rarely fires (if it fires often, mass does not sum to 1).

How agents use this

Temperature on tool logits is a policy knob: near-greedy makes a stubborn specialist; high T makes a ditherer. Log the chosen tool and its chance. A long tail of 0.15 decisions is an entropy problem you can fix with a better prompt, fewer tools, or lower T.

• Tokens: production often uses T near 0 for tools and slightly higher for prose. Split them if the API allows. Do not raise T on tool names to “add creativity.”
• Ranking: retrieval is usually greedy top-k, not sampled. Sampling documents is a rare product choice; if you do it, seed it and log the scores.
• Loss: train at the model’s native distribution (T=1 in the loss). Do not train with a high T and then decode greedy without measuring the gap.
• Evals: seed when you compare prompts. Unseeded T>0 makes “which prompt won?” a coin flip. For diversity, many seeded draws, then score the set of plans.

Top-k and top-p are extra filters on the same p list, not a different softmax. They cut the long weird tail without driving T all the way to 0. Seeded sampling with T > 0 is how you generate diverse traces for evals: same prompt, many draws, then score the distribution of plans rather than a single lucky run.

Tip:Seed evals when you compare prompts. Unseeded T>0 makes “which prompt won?” a coin flip.