Optimization

Optimization means pick weights to make a scalar as small as possible (or as large — then flip the sign). Training is optimization of a loss. Many agent knobs are optimization too: find a cutoff that maximizes F1 on a dev set.

Gradient descent is the algorithm: start at x, compute g = grad f(x), set x ← x - lr * g. The learning rate lr is the step size. Too large, you bounce. Too small, you crawl. Gradient ascent drops the minus and is used when you maximize.

A local minimum is a valley that may not be the deepest on Earth. For the convex bowl below, local is global. For real nets, you settle for a good valley and evals, not for a philosophical minimum.

A wrong picture

A wrong picture is: “more steps always better.” After a point you overfit (eval-scores, ML track). Infinite descent is a bill. Stop when the gradient is small, when f stops improving, or when you hit a step budget — the same three families as agent stop conditions (success, plateau, max steps).

Another wrong picture is: “the optimizer shares my unstated values.” The optimizer is loyal to the scalar, not to citations, not to latency, not to “be nice.” Agents that self-improve by hill-climbing “shorter answers” will cut citations. Choose f as carefully as you choose a loss.

A third wrong picture is: “a huge learning rate is faster.” On a bowl, too-large lr increases f. Same gradient, different step. Most “training is unstable” bugs in small models are this diagram. Also: optimizing pass-rate on 20 hand-picked tickets overfits as surely as 10,000 epochs on 20 images. Hold out evals.

The formula in words

Repeat: measure gradient of f at the current point. Subtract learning-rate times that vector. That is one step.

Tiny numeric. f(x, y) = (x - 2)^2 + (y + 1)^2. Floor at (2, -1) where f = 0. Gradient [2(x-2), 2(y+1)]. Start (0, 0), f = 4+1 = 5, gradient [-4, 2]. The update is minus learning-rate times gradient: minus 0.2 * [-4, 2] = minus [-0.8, 0.4] = add [0.8, -0.4]. New point (0.8, -0.4). Closer to (2, -1). f drops.

With lr = 1.1 on this bowl you can jump over the floor and climb the other wall. Print f and watch it fail to fall.

Moving parts

Descent is x ← x - lr * g. Drop the minus and you ascend. That sign bug is common.

A second walkthrough (one dimension)

f(x) = (x - 3)^2. True slope f' = 2(x-3). Start at x = 0, f = 9, slope -6.

Step with lr = 0.25: x ← 0 - 0.25 * (-6) = 1.5. New f = (1.5-3)^2 = 2.25. Down.

Next slope 2(1.5-3) = -3. x ← 1.5 - 0.25*(-3) = 2.25. f = 0.5625. Toward 3.

Same start, lr = 1.1: x ← 0 - 1.1*(-6) = 6.6. f = (6.6-3)^2 = 12.96. Up. Same gradient, different step. Most “training is unstable” bugs on a toy bowl are this diagram.

Stop when |g| is tiny and f is near 0 (here, you reached the floor). Stop when f plateaus even if g is noisy (batches). Stop at a step budget so infinite descent is not an infinite bill.

A Friday ticket

Friday they “optimized” max_steps by hill-climbing pass-rate on 20 golden tickets. Pass-rate hit 100% after the agent learned to skip search (shorter traces still passed those 20). Monday’s held-out tickets needed search. Pass-rate fell. Same loop as 10,000 epochs on 20 images.

The scalar was wrong for the product, and the set was too small. They froze a held-out eval, put a step budget on the search, and stopped treating 20 goldens as the world.

Descend a bowl

Run to execute this in your browser. Nothing is sent to a server.

With lr = 0.2 the values walk toward (2, -1) and f drops: start f = 5, then about 1.8, 0.65, 0.23, … approaching 0. x climbs toward 2, y falls toward -1. With clumsy lr = 1.1, f can jump (watch the second table). Same gradient, different step.

If your print grows instead of shrinks on the first loop, you dropped the minus. That is the sign bug again.

What goes wrong

• Sign: adding lr * g when you meant to minimize. f rises. People blame the learning rate. Print f after one step on a bowl first.
• Huge lr: f jumps. On this bowl, 1.1 already fails. Clip lr, or clip the gradient’s length (keep direction, cap magnitude).
• Tiny lr: crawl. Looks like “not training.” Check that f does drop, just slowly.
• Wrong scalar: shorter-answer ascent cuts citations. Latency not in f will not fall. Choose f as carefully as a loss.
• No holdout: optimizing pass-rate on 20 tickets overfits. Same geometry as memorizing 20 images.

Production logs: f, gradient length, lr, step index, and a held-out scalar that is not the training f. Assert f is finite, g has the same dimension as x, and that one downhill step on a fixture bowl lowers f. Stop conditions should match agents: success (small g / good f), plateau, max steps.

Stopping, batches, schedules

Stop when the gradient is small, when f plateaus, or at a step budget.

Stochastic gradient descent replaces f with a batch estimate. The direction is noisy; you still go downhill on average. That noise is why you log smoothed loss, not only the last mini-batch. A moving average (sums lesson) is the smoother.

You can schedule the learning rate: start larger to make progress, shrink later to settle. Momentum methods add a fraction of the previous step so you do not zigzag in a ravine. For this academy, master the minus-gradient update first; fancy variants are the same geometry with extra memory.

Discrete search (which prompt, which tool set) is not this loop. There is no honest gradient. Use eval scores, not backprop. Still optimization: you pick the option with the best scalar on a held-out set.

How agents use this

You will not train GPT from this page. You will train small things: a linear scorer on embeddings, a calibrated cutoff, maybe a prompt via discrete search (not slope-based — use eval, not backprop). When a vendor fine-tunes, their loop is this loop plus automatic gradients.

• Tokens: max_tokens and max_steps are budgets, not learning rates. Do not “descend” them without measuring quality. A tiny eval finite-difference (derivatives lesson) is the honest slope.
• Ranking: train a projection or a linear ranker with descent on a pairwise or softmax loss. Then freeze and retrieve. Re-index after weights move.
• Loss: the scalar is the product. If you add a length penalty, answers get shorter. If you only optimize training loss, held-out perplexity can still jump (cross-entropy lesson).
• Sampling: greedy decoding is argmax, not descent. Temperature is a knob you can search with evals. Do not backprop through production sampling without a research setup.

Log f and |g| every step on anything you actually descend. If |g| explodes, shrink lr or clip. If |g| is tiny and held-out f is still bad, you are on a flat of the wrong hill — or you overfit the 20 goldens.

Agents that self-improve by hill-climbing a metric on stored traces are doing ascent on that metric. If the metric is “shorter answers,” they will cut citations. Choose f as carefully as you choose a loss. Hold out evals. Twenty golden tickets are not a world.

Watch out:Optimizing pass-rate on 20 hand-picked tickets overfits as surely as 10,000 epochs on 20 images. Hold out evals.