Gradient Descent

Gradient descent is the algorithm behind almost every neural net you will use: measure how the loss changes when you nudge each parameter, then nudge the parameters downhill.

For a scalar parameter w, the gradient is the slope. Update:

w = w - learning_rate * slope

If the slope is positive, w is too big; we decrease it. If negative, we increase it. The learning rate (lr) is how brave the step is. Too large: you jump over the valley. Too small: you wait forever (and pay the cloud).

You do not need to love calculus to use this. You need the picture: the loss is a landscape; parameters are coordinates; the slope says which way is up; we walk the other way a little. Automatic differentiation computes that slope for millions of knobs. On this page we do one knob by hand so the loop is visible.

A one-parameter universe

We fit y ≈ w * x with MSE. True w is 2. We start at 0 and only look at the data.

The slope of MSE for one example is 2 (wx - y) * x, then average. You can also estimate the slope with a tiny bump (finite difference): raise w a hair, lower it a hair, subtract the losses, divide. If analytic slope and finite difference disagree, your formula is wrong — a debugging trick that still works on tiny agent-scoring functions you wrote yourself.

The loop is always:

1. Forward — compute predictions from current knobs
2. Loss — one number
3. Backward — slope of that number with respect to each knob
4. Step — move knobs opposite the slope

Print loss every few steps. If it becomes inf, your lr is a dare. If it never moves, the slope is near 0 (a bug, a dead unit, or lr too small) or you are already in a flat spot.

Run to execute this in your browser. Nothing is sent to a server.

What printed: at step 0, w is 0 and loss is large (the targets are 2, 4, 6, 8 and you predict all zeros). At step 1, analytic gradient and finite-difference gradient match to several decimals — the formula is not a typo. Then w climbs toward 2 and loss falls: after 25 steps you should be near the true slope, with a small leftover depending on lr. Bigger models have millions of ws and use automatic differentiation, but the loop is the same: forward, loss, backward, step.

If you set lr to 1.0 in your head, w would leap past 2 and might diverge. If you set lr to 0.0001, 25 steps would barely move. That is the next lesson’s table. Here, notice the minus sign: we subtract the slope because we want descent.

Mini-batches and noise

Full-batch uses every example in the gradient. The slope is stable and potentially slow. SGD (stochastic gradient descent) uses one example or a small batch. The gradient gets noisy; that noise sometimes helps escape a bad valley. For agents scoring traces, a “batch” might be 16 conversations — the same idea.

Shuffle train between epochs so the batches are not always the same order. A frozen unlucky order can stall. That shuffle is not a test-set shuffle. It is only train.

Epoch means one pass over the training set. People quote epochs because it is easy. What the optimizer actually sees is steps. A huge dataset with tiny batches can do many steps per epoch.

Non-convex losses (neural nets) have many valleys. Gradient descent does not promise the global best w. It promises “from here, go downhill.” Early stopping on validation is how practitioners stay honest: when the working exam worsens, stop, even if train loss still falls.

When the slope lies

The slope is the derivative of the loss you wrote, not of the product. If the loss ignores forbidden tools, descent will not avoid them. If the loss is next-token surprise, descent will make fluent text. If labels are noisy, descent will fit the noise given enough knobs.

Vanishing slopes (sigmoid stuck at 0 or 1, ReLU units that never fire) look like “training did nothing.” Exploding slopes look like NaN. Clipping gradients (cap the length of the slope list) is a seatbelt, not a model. Print. Do not guess.

Schedules, clipping, and scoring functions

People often decay the learning rate: start brave, then take smaller steps so you settle in a valley instead of hopping out. A simple version is “use 0.02 for 50 steps, then 0.005.” That decay is a hyperparameter. So is clipping: if the slope list is huge, shrink it before the step. Neither is magic. Both are reactions to a printed loss that jumped.

Momentum (keep a running average of the slope, then step along that average) smooths noisy batches. Adaptive methods (one effective step size per knob) exist in every library. You do not need to implement them here. You need to know they still do w = w - something * slope. If the loss is the wrong product, a fancier stepper will descend the wrong hill faster.

Agent work often has no analytic slope. You have a scoring function on a freeze of traces: schema penalty plus missing citation plus forbidden tool. You cannot backprop through a hosted model easily. You can still walk downhill: change one prompt sentence, rescore the freeze, keep the winner. That is coordinate descent. The gradient-descent lesson still applies: define the number, take small steps, print, stop when validation turns. Do not take a huge prompt rewrite and call it one step — you will not know which sentence moved the score.

When you can train a small router, print three numbers per N steps: train loss, validation loss, and a product metric (recall on ask_human). If train loss falls and the product metric does not, your loss is a bad cousin. Change the loss, not the stepper.

Common mistakes

• Forgetting the minus sign and climbing the loss.
• A learning rate that worked at w = 0 and explodes later.
• Computing the gradient on validation or test (leakage with extra ceremony).
• Comparing two runs that used different step counts and calling it architecture.
• No print of loss, then a week of “maybe it is learning.”

How agents use this

You will rarely write GPU kernels. You will often write a scoring function and a search: prompt candidates, tool-order candidates, chunk-size candidates. If you can compute a numeric score, you can descend — even if the “gradient” is “try the neighbor and keep the winner” (coordinate search). The ML habit is: define downhill, then walk.

Fine-tuning is this loop on someone else’s architecture. You still own the loss, the split, and the print. If you cannot say what downhill means on a frozen validation slice, you are not training. You are burning compute.

Watch out:A learning rate that works at w = 0 can explode later. Print loss every N steps. If it becomes inf, your lr is a dare.