Derivatives

The derivative of a function f at a point x is the slope of the graph there: how much f changes if you move x a little, divided by the size of the move.

In words: rise over run, for a tiny run. For a line f(x) = mx + b, the derivative is the constant m. For f(x) = x^2, the derivative is 2x: steep far from zero, flat at the bottom of the bowl. That is why gradient descent can crawl in a valley and fly on a wall.

You do not have to memorize a table to use derivatives in software. You can estimate them: pick a small h, compute (f(x+h) - f(x)) / h. That is a forward difference. A central difference (f(x+h) - f(x-h)) / (2h) is usually more accurate for the same h.

A wrong picture

A wrong picture is: “the derivative is how important a feature is, globally.” It is a local slope at one point. At another point the slope can flip sign. A weight that is “important” on Monday’s batch can be flat on Tuesday’s. Sensitivity analysis with a nudge is useful; it is still local.

Another wrong picture is: “smaller h is always better.” If h is huge, you measure a chord across the continent, not the tangent. If h is tiny like 1e-20, floating point rounds x+h back to x and you get 0 or noise. Values around 1e-5 are a reasonable start.

A third wrong picture is: “to minimize, move with the slope.” If f'(x) > 0, increasing x increases f. To minimize f, you move against the slope: x - step * f'(x). That one line is gradient descent in 1-d.

The formula in words

True derivative: the limit of rise/run as the run goes to 0. In code we sneak up on it with a small h.

Forward: [f(x+h) - f(x)] / h. Cheap (two evaluations if you already have f(x)). Biased a little to one side.

Central: [f(x+h) - f(x-h)] / (2h). Two evaluations. Usually closer.

Sign: positive slope means the graph rises to the right. Downhill to the left. Negative slope: downhill to the right.

A tiny example

f(x) = x^2 at x = 3. True slope is 2 * 3 = 6. Forward with h = 1: (16 - 9)/1 = 7 — close-ish, not 6. Forward with h = 0.01: about 6.01. Central with h = 0.01 is even closer to 6. At h = 1e-12 floats get messy.

To minimize at this point: slope is +6, so decrease x. A step x - 0.1 * 6 = 2.4. f(2.4) = 5.76, less than f(3) = 9. Downhill worked.

Check a slope with a loop

When you later write a formula for a slope (or trust a library), check it against a tiny nudge on a few points. Mismatches mean a bug, not a philosophical fight.

Run to execute this in your browser. Nothing is sent to a server.

Watch forward differences approach 6, then get messy at 1e-12. At h = 1 forward is 7.0 and central is 6.0 (central is exact here because x^2 is a nice parabola). At h = 0.1 forward is 6.1. At h = 0.01 forward is 6.01. Central stays on 6.0 until floats bite. The derivative is not a vibe. It is a limit you can sneak up on with a loop.

Sign matters. If f'(x) > 0, increasing x increases f. To minimize f, you move against the slope: x - step * f'(x). We will run that after gradients in several variables.

A finite difference needs two (or more) runs of f. Never nudge a function that hits a paid API in a tight loop without a budget. For local knobs (temperature, cutoff), nudge on a small held-out set.

How agents use this

Loss is a function of weights, prompts, even knobs. “If I raise temperature by 0.1, does eval score go up?” is a derivative of an eval function (noisy, but the same idea). Tiny nudges on a small eval set are how you sanity-check that a knob does what the dashboard claims.

A useful 1-d derivative is cost with respect to max_steps: add one allowed step, measure dollars and quality. If quality is flat and dollars rise, the slope says stop. You do not need a GPU to compute that from logs.

• Tokens / dollars: treat total tokens as f(max_steps) or f(temperature) on a fixed ticket set. Finite difference is two complete runs. Budget it.
• Ranking: “if I raise the cosine cutoff by 0.05, what happens to recall?” That is a slope of an eval, with a jump (cutoff is discrete-ish). Sweep a few cutoffs rather than a tiny h.
• Loss: training uses analytic slopes (next lessons). Finite differences check those slopes on a toy f. If analytic and numeric disagree, the training code is wrong.
• Sampling: temperature’s effect on pass rate is noisy. Average several seeds before you believe a slope. A single draw is not a derivative.

Do not confuse a derivative with a causal story about users. It is how this function responds to this input nearby. If you differentiate the wrong scalar, you will energetically optimize the wrong product.

Watch out:A finite difference needs two (or more) runs of f. Never nudge a function that hits a paid API in a tight loop without a budget.