Reference/Math
Probability and sampling
Bernoulli entropy, expected cost, greedy vs weighted sample — decoding in one page.
LLMs emit a distribution over tokens. Decoding turns it into a choice.
Bernoulli entropy (bits)
H(p) = -p log2 p - (1-p) log2(1-p) with H(0)=H(1)=0.
Max at p=0.5 (1 bit). Low entropy = peaked = greedy is almost enough.
Expected cost
If each step costs c tokens in + d out, expected cost of a loop with random length L is E[L] * (c+d). Cap L. That is max_steps.
Greedy vs sample
| Mode | Rule | Use |
|---|---|---|
| Greedy | argmax | tools, JSON, classifiers |
| Temperature | softmax(z/T) then sample | prose |
| Top-k | sample from k largest | cap tail |
| Nucleus (top-p) | smallest set with mass ≥ p | modern default |
Temperature T→0 ≈ greedy. T→∞ ≈ uniform. Never sample tool names if you can greedy + validate.
random.choices
python
import random
random.choices(range(len(w)), weights=w, k=1)[0]Seed in tests. Do not seed in production traffic.
Tip:Structured output = greedy + schema. Sampling belongs in the user-facing sentence, not in {"tool": ...}.