Logs and Exp

exp means e to a power. In Python that is math.exp(x). It grows fast. exp(0) is 1. exp(1) is about 2.718. exp(10) is already about 22,026. exp(1000) is too big for a normal float: you get inf.

log undoes exp. math.log(x) is the natural log (base e). math.log(x, 2) is log base 2, measured in bits. We will use bits for entropy later. Natural log is measured in nats. Same story, different unit. Be consistent.

These two show up every day in agents:

• Softmax uses exp to turn logits into chances.
• The chance of a whole sentence is a product of token chances. Products of small numbers hit 0.0. log of a product is a sum of logs. Sums stay safe.

A wrong picture

A wrong picture is: “log of a probability is still a probability.” It is not. Chances live in [0, 1]. Log of a chance is zero or negative (for chances ≤ 1). log(1) = 0. log(0.5) is negative. People still say “logprob” on dashboards. That number is not a chance. You must exp it if you want a chance back (and even then, a sum of logprobs is the log of a product, not a chance of a single token).

Another wrong picture is: “I can exp any logit.” exp(1000) overflows. Softmax subtracts the biggest logit first. The chances do not change, because the same shift hits every term and cancels. You will code this in the softmax lesson. Here, just see that exp is touchy.

A third wrong picture is taking log(0). That is not a real number. In code, skip zeros or add a tiny floor before you log. Training that puts chance 0 on the true token is infinite loss. That is a feature of the formula, not a GPU mystery.

The formula in words

• exp(x) is “e multiplied by itself x-ish-ways.” Bigger x, much bigger output.
• log(x) asks “what power of e gives x?” Only for positive x.
• They undo each other: exp(log(x)) = x when x > 0, and log(exp(x)) = x.
• Log turns a product into a sum: log(a * b) = log(a) + log(b). That is why long sentences are scored as a sum of token logprobs.
• Log turns a divide into a subtract: log(a / b) = log(a) - log(b).
• Exp turns a sum into a product: exp(a + b) = exp(a) * exp(b). Softmax lives here.

A tiny example

Three token chances: 0.5, 0.25, 0.25. Product is 0.5 0.25 0.25 = 0.03125. Sum of logs (natural) is log(0.5)+log(0.25)+log(0.25). Exp of that sum is again 0.03125. The sentence chance is the product. The number you store is the sum of logs.

log2(8) = 3 because 2^3 = 8. That is 3 bits. log2(0.5) = -1. A fair coin landing heads is 1 bit of surprise (entropy lesson will say “minus log”). For now, bits are just log base 2.

Subtract-the-max: logits [1, 3, 2]. Max is 3. Shifted: [1-3, 3-3, 2-3] = [-2, 0, -1]. Exp of those is a short list of ordinary numbers. Exp of the raw logits is larger, but ratios match. Softmax cares about ratios after you divide by the sum.

Rules you will use

log is only for positive numbers. log(0) is not a real number. In code, skip zeros or add a tiny floor.

Why we subtract the max

exp(1000) overflows. Softmax does this trick: subtract the biggest logit first. Try the box.

Run to execute this in your browser. Nothing is sent to a server.

Read the prints. exp(0) is 1.0. exp(1) is about 2.718. log(exp(2)) returns 2.0. Product of chances is 0.03125. Exp of the sum of logs matches it. log2(8) is 3.0. log2(0.5) is -1.0. Shifted exps are about [0.135, 1.0, 0.368] — the max logit became exp(0)=1. The last line shows raw exps in the same ratios as the shifted ones (divided by the smallest raw). That is why the trick is safe: chance is exp divided by sum of exps, and a common factor cancels.

The product of the three chances matches exp of the sum of logs. That is why training talks about log-likelihood: it is a sum you can add, not a product that vanishes.

If you ever multiply 200 token chances like 0.4, 0.3, 0.2, … the product hits 0.0 in float math long before the sentence is impossible. The sum of logs is a perfectly ordinary negative number, like -80. Dashboards plot that.

How agents use this

A model scores a token with a logit. Softmax is exp, then divide so the list sums to 1. Sequence chance is a product. You log it. Cost dashboards that plot “average logprob” are this sum, averaged. If you ever see 0.0 for a long prompt’s chance, you forgot to log.

Agent connections, in one place:

• Tokens: next-token chance p, stored as log(p). Beam search and scoring add those logs.
• Ranking: some retrievers use log of a score; most of this track uses cosine (a different formula). Do not mix them without knowing which.
• Loss: cross-entropy is -log(q_true) for a one-hot label. Small q_true means a large loss. You will code it later.
• Sampling: temperature divides logits before exp. That is still this pair of functions.

Never exp a huge logit without subtracting the max. The page may print inf or nan. Never log a chance that rounding pushed to a tiny negative — clip first (next lesson).

Watch out:Do not exp a huge logit without subtracting the max. The page may print inf or nan.