Softmax and attention weights

Softmax maps logits z to a probability vector.

p_i = exp(z_i) / Σ exp(z_j)

Stable form

Subtract m = max(z) first: exp(z_i - m). Same p, no overflow.

import math
def softmax(z):
    m = max(z)
    e = [math.exp(x - m) for x in z]
    s = sum(e)
    return [x / s for x in e]

Attention (one head, one query)

1. scores = query · key_i
2. weights = softmax(scores / sqrt(d)) # scale is optional in the toy
3. output = Σ weights_i * value_i

Weights must sum to 1. If you implement this in a Try it box, print(sum(weights)).

Cross-entropy loss (classification)

For true class y: L = -log p_y. Low p_y → large loss. Agents do not train this in Joeven, but eval logs often quote it.

Note:Tool choice among 3 tools is a 3-way softmax in the model. Your code should still allowlist names.