Cross-Entropy, KL, and Perplexity

Cross-entropy asks: if the truth is distribution p, and the model predicts q, how many bits do we spend on average to name the outcome using q?

H(p, q) = -sum p_i log q_i

If p is a one-hot (the true next token is index k), this collapses to -log q_k. That is the usual token loss. A model that put chance 0.5 on the right token pays 1 bit. Chance 0.25 pays 2 bits. Chance near 0 pays a huge bill.

KL divergence D_KL(p || q) is extra bits beyond the entropy of p:

D_KL(p || q) = H(p, q) - H(p)

It is 0 when p = q. It is not a distance: D_KL(p || q) is not D_KL(q || p). Training a classifier is often “drive KL (or cross-entropy) down on the training labels.”

Perplexity is 2^{H} when H is in bits, or exp(H) when H is in nats. It is “how many options does this feel like, on average?” Perplexity 10 means as confused as a uniform choice among 10 tokens. Lower is sharper (on that data). Lower on training data can also mean memorization. Use a held-out set.

A wrong picture

A wrong picture is: “training cross-entropy is quality.” Perplexity does not know if the answer was useful. A model can pay few bits copying the training set and still fail the ticket. Use eval tasks. Held-out perplexity is a health check (prompt, tokenizer, or model id changed), not a trophy.

Another wrong picture is: “KL is a distance.” Distances are symmetric. KL is not. Print both directions if you compare two policies. Distillation often minimizes KL from teacher to student — the order is a product choice.

A third: log(0) on the true class. Infinite loss, NaNs in the log. Guard q with a tiny floor, or use log-softmax. That is why people clip q away from 0 before they log.

You cannot beat H(p) on average: cross-entropy is at least entropy of the truth. The extra is KL. If labels are noisy, the model will still try to put mass there. Garbage labels, garbage q.

The formula in words

Cross-entropy: weighted average of -log q_i with weights p_i. One-hot p: only the true index survives, so -log q_true.

KL: cross-entropy minus entropy of p. Extra bits the model wastes relative to a perfect encoder of p.

Perplexity: 2 to the power of bits of cross-entropy (or e to the nats). Units: “effective number of choices.”

Tiny numeric. True one-hot [0, 1, 0]. Good q = [0.05, 0.90, 0.05]: CE = -log2(0.90) ≈ 0.152 bits, perplexity 2^0.152 ≈ 1.11. Bad q = [0.40, 0.20, 0.40]: CE = -log2(0.20) = 2.32 bits, ppl about 5. Flat 1/3: CE = -log2(1/3) ≈ 1.585.

Asymmetric KL: p=[0.7, 0.3], q=[0.6, 0.4] vs swapped — two different numbers.

Moving parts

One-hot p: only the true index survives, so CE is -log q_true. That is the usual token loss.

A second walkthrough (soft labels)

Truth is not one-hot this time: p = [0.80, 0.20], model q = [0.60, 0.40].

-log2(0.60) ≈ 0.737, -log2(0.40) ≈ 1.322.

CE = 0.800.737 + 0.201.322 = 0.590 + 0.264 = 0.854 bits.

Entropy of p: -log2(0.80) ≈ 0.322, -log2(0.20) = 2.322.

H(p) = 0.800.322 + 0.202.322 = 0.258 + 0.464 = 0.722 bits.

KL = 0.854 - 0.722 = 0.132 bits. The model is close, not equal. Swap p and q and you get a different KL (the tryit prints both). Distances are symmetric. KL is not.

If q were one-hot on the wrong class, CE is huge (clip at 1e-12 → about 40 bits). If q matches p exactly, KL is 0 and CE equals H(p). You cannot beat H(p) on average.

A Friday ticket

Friday 11:00. Training loss was still falling. Held-out perplexity jumped from about 12 to about 31 overnight. The incident title was “the model got worse.” A tokenizer config had changed; token ids on the held-out set no longer matched the ids the model was trained on. Train CE kept looking healthy because it was computed on the new ids against the same files, now misaligned in a way that still had a frequent token to copy.

They started logging ppl_heldout_bits on a frozen transcript set at T=1, with the tokenizer version next to it. Shipping on training CE alone was banned. Perplexity is a health check, not a trophy.

Code the three numbers

Run to execute this in your browser. Nothing is sent to a server.

H(true) one-hot is 0.0 (we passed [1.0] as a peaked distribution). CE good about 0.152, ppl about 1.111. CE bad about 2.322, ppl about 5.0. CE flat about 1.585. KL(p||q) and KL(q||p) print two different small numbers (about 0.031 vs 0.033 — order matters). The good predictor pays fewer bits than the bad one. Clip q away from 0 before you log, or a single zero chance on a true token is infinite loss.

Fine-tunes minimize cross-entropy on “the right next token” (or the right tool name). If your labels are noisy, the model will still try to put mass there.

What goes wrong

• log(0): infinite loss, then NaNs. Guard q with a tiny floor, or use log-softmax so you never exp then log a zero.
• Mixing bits and nats: math.log vs log2. Perplexity is 2^H only if H is in bits. If H is in nats, use exp(H). Label the unit.
• Train CE as quality: falling train CE can be memorization. Held-out CE / perplexity is the health check. Eval tasks are the trophy.
• Asymmetric KL: printing one direction and comparing it to a paper that used the other looks like a regression. Distillation order is a product choice: teacher→student is not student→teacher.
• Softmax twice before CE: you score a different q than the model. Loss then trains the wrong object.

Production logs: mean token CE (say bits or nats), held-out perplexity, tokenizer/model id, and a canary that CE stays finite on a batch with a rare token. Assert q_i >= floor, unit sum, and that KL ≥ -tiny (floating noise). A sudden ppl jump is a prompt, tokenizer, or model-id change until proven otherwise.

How agents use this

Perplexity on a held-out transcript set is a cheap health check: a sudden jump means the prompt, tokenizer, or model id changed. Distillation and some ranking heads minimize KL from a teacher distribution to a student.

• Tokens: usual training loss is token cross-entropy. Average it over a batch (sums lesson). Log in nats or bits; say which.
• Ranking: a softmax over candidate chunks plus CE toward the gold chunk is a listwise ranking loss. Pairwise “gold should beat distractor” is a cousin.
• Loss: this is the loss for classification and next-token training. It is not F1. It is not dollars. If you want those, put them in eval, or add a term (chain rule: no path, no optimize).
• Sampling: decode T does not change the trained q unless you divide logits at train time too. Measure CE at T=1.

You cannot beat H(p) on average: cross-entropy is at least entropy of the truth. The extra is KL. If labels are noisy, the model will still try to put mass there. Garbage labels, garbage q.

Watch out:log(0) on the true class is infinite loss. Guard q with a tiny floor, or you will debug NaNs instead of agents.