Hyperparameters
Learning rate, epochs, batch size, and seeds are knobs you choose — on validation, not on test.
Parameters are numbers the training loop fits (w, b, millions of weights). Hyperparameters are numbers you choose: learning rate, number of epochs, batch size, cutoff, temperature, top-k.
If you pick them by looking at the test set, the test set is no longer a test. Pick on validation. Touch test once.
This distinction is the most violated rule in agent work, because “hyperparameters” sound like training, and prompts sound like English. Temperature, top-k retrieved chunks, max steps, and “escalate if p < 0.6” are hyperparameters of the agent. Sweeping them on the published eval is the same crime as tuning lr on test.
The knobs you will actually turn
| Knob | If too small | If too large |
|---|---|---|
| Learning rate | Loss barely moves | Loss explodes or jumps |
| Epochs | Underfit | Overfit (next lesson) |
| Batch size | Noisy, slow wall-clock if tiny | Smooth, may need more memory |
| Seed | — | Different seed, slightly different run |
| Cutoff | Too many yeses | Misses the rare class |
| Top-k | Gold chunk never enters the prompt | Extra junk drowns the model |
An epoch is one pass over the training set. A batch is the slice you use for one gradient step. More epochs is not more virtue. It is more chances to memorize train.
Weight decay, dropout rate, hidden size, and prompt length are hyperparameters too. Anything you choose rather than fit belongs on the list. Write the list. Change one knob at a time. If you change lr, batch, and prompt in one commit, you will not know which one helped.
Tiny lr barely moves. Good lr finds the valley. Huge lr blows up. Only the knob changed.
Same 25 steps, three learning ratesRun to execute this in your browser. Nothing is sent to a server.
What printed: three outcomes on the same data and the same 25 steps. Tiny lr barely moved: w still near 0, loss still large. Good lr found w near 2 with a small loss. Huge lr returned exploded — the update jumped so far that w became nonsense (w != w is the NaN check, or the absolute value blew past a million). Only lr changed. That is why people plot loss.
The w != w test is a Python trick: NaN is not equal to itself. You do not need a library to see a failed run.
Seeds and “it worked once”
random.Random(0) makes shuffle and init repeatable. A result that only works on seed 7 is not a result. Report the seed. Run more than one if the set is small. Agent evals with 40 traces are small. One lucky split plus one lucky temperature is a story.
Seeds are hyperparameters you should not optimize. Do not search 200 seeds and keep the winner on test. Freeze a seed for reproducibility, then run a few extra seeds on validation to see spread.
Search without cheating
Grid search: try a few learning rates, a few cutoffs, write a table on validation. Random search: pick random combinations when there are many knobs. Either is fine at this scale. Nested cheating is not: picking the best prompt on test, then picking the best cutoff on the same test.
Use a budget: you may look at validation N times. After that, freeze. If you must keep searching, get a new validation slice from later traces and treat the old one as worn.
For agents, log every trial: prompt version, k, temperature, cutoff, seed, validation score. The log is the experiment. Memory is not.
What counts as a knob on an agent
Write a one-page catalog so the team stops arguing about “the model” when they mean a cutoff.
- Generation: temperature, top-p, max tokens, whether you sample or take the greedy token
- Loop: max steps, retry counts, whether
finishis allowed before a required tool - Retrieval: encoder name, chunk size, overlap, top-k, score cutoff, prefixes
- Router: decision cutoff, abstain band, class weights used in training
- Training: learning rate, epochs, batch size, weight decay, seed
- Prompt: template version, few-shot ids (those ids are data, and a hyperparameter of which data you stuffed)
Each row needs: default, range you are willing to try, which split you tune on, and whether changing it requires a new dataset version. Temperature 0 vs 0.7 can move a tool-calling agent more than a new base model. Treat it with the same respect as lr.
Nested mistakes look like science: you pick the best of 20 prompts on validation, then pick the best cutoff on the same validation, then pick the best k, and you report the winner as if it were one pre-registered model. Each look wears the slice. Budget the looks. When the budget is spent, freeze and take the test number once. If you must keep searching, get a new validation slice from later traces.
A result that moves by 2 points when you change the seed on 40 traces is noise. Report spread, or get more traces, before you ship a knob change as a win.
Common mistakes
- “Fixing” the demo until the published number moves.
- Searching temperature on the live customer mix and calling it science.
- One huge change-set of knobs.
- Treating seed search as modeling.
- No record of what was tried, so you retry last month’s failure.
How agents use this
Prompt length, temperature, top-k retrieved chunks, max steps, and “escalate if p < 0.6” are hyperparameters of the agent. Sweep them on a frozen validation slice of traces. Do not “fix” the demo until the published number moves. That is the same crime as tuning on test.
When a vendor ships a new model, those knobs may need a re-sweep on validation. The old temperature is not a law of nature. Re-sweep does not mean peek at test. It means the working exam again, then one test run.
Tip:Change one knob at a time. If you change lr, batch, and prompt in one commit, you will not know which one helped.
Check your understanding