Choosing Models

There is no “best model.” There is a best model for a step, under latency, cost, privacy, and quality. Treating the frontier chat model as the only employee is how bills and latency explode. The last lessons packed the window. This one picks which brain reads that window.

A smaller model that emits valid JSON 99% of the time beats a genius that essays 30% of the time. Averages lie. A small model that is always 400ms beats a large model that is 2s until it is 20s under load. Queueing, rate limits, and cold starts are part of “which model.”

Keep complete(messages, tools) behind your interface. Swap models as a config change. Re-run the eval. Tokenizers differ. Privacy: if traces cannot leave, the catalog shrinks to what you host. That constraint beats a leaderboard.

Axes that matter for agents

Measure the hosted small model on your JSON. A public “arena” rank is entertainment. Your tickets are the job.

Distillation and fine-tunes of small models are how grown-up teams lock in a router. Do not start there. Start with measurements. Fine-tune last (ML and Transformers already said why).

Run to execute this in your browser. Nothing is sent to a server.

For json, the large model fails the 0.95 gate and scores -1.0, so small wins. For plan, the gate is skipped and large can win on quality. Toy weights; real teams plug in their fail rate, their p95, their dollars. The shape is the lesson: hard gate on JSON, then a score. Do not average JSON fail rate into a soft penalty and hope.

p95 means: 95 percent of calls are faster than this. Users feel the slow tail. Timeouts fire on the tail. If you only watch the mean, you will pick a model that is “fine” until Friday traffic.

Run your tickets, not an arena

A public ranking answers “which model sounds better on a mix of internet questions.” Your agent answers “which model emits valid get_job JSON on these 200 frozen tickets, under this spec, with this tokenizer.” Those are different contests. Steal the method: freeze the tickets, freeze the spec id, change one model, score with your validator and your grounded/abstain checks. Do not score with a second LLM unless you have measured that judge (Eval track).

Protocol that fits on one page:

1. Pick the step kind (JSON tool vs plan vs prose).
2. Run 100–300 tickets (or all you have).
3. Record JSON-ok rate, abstain-on-impossible rate, p50/p95 latency, tokens, dollars.
4. Hard-fail a candidate below the JSON gate for JSON steps.
5. Among survivors, pick on p95 and dollars, not on a single “quality” vibe.

Tokenizer swap: after you change models, re-estimate packing. A spec that was 800 tokens may become 1,100. Stop sequences that were one token may become three. Prompt cache prefixes may break because bytes include different wrapping. Re-run build_context tests with the new estimator. Choosing a model is also choosing a tokenizer.

Cold starts and queues belong in p95. A self-hosted 7B that is fast when warm and 8 seconds when the GPU worker scaled to zero is not a 400ms model. Measure under the same autoscale policy you will ship.

A walkthrough: the JSON gate

Maya’s team loved a large model’s explanations. JSON-ok was 70%. The executor spent the week in handoff. They switched tool steps to a small model with native tool-calling: JSON-ok 99%, explanations worse. They split the step: small emits the tool JSON; large writes the user-facing paragraph after the tool result exists. Cost dropped. Latency dropped. Support stopped seeing invented job ids. That split is choosing models. It is not a single winner.

Privacy, region, and rate limits

If traces cannot leave the VPC, the catalog is local. If the user is in a region that forbids a vendor, the catalog shrinks again. A crew of agents is a burst: one user click can be 8 POSTs. Rate limits are part of quality. A perfect model you cannot call is an error outcome (429 lesson).

What goes wrong

• One frontier model for classify, extract, plan, and poetry.
• Choosing from a blog table without running tickets.
• Ignoring tokenizer change after a swap (stop sequences, token budgets).
• Ignoring p95.
• Fine-tuning to avoid measuring.

How agents use this

Choosing models is an architecture decision: which brain at which node in the loop. Next lesson: a router. If every node is the most expensive model, you did not design a system. You rented one.

Log model= on every span. A week later you will know whether “always large” was fear or evidence.

Keep a short approved catalog: small JSON, large plan, embed, maybe a fallback region. A developer pasting a new SKU into one call site is how you get an unbilled surprise and a tokenizer you never packed for. Catalog changes are PRs with an eval attached.

If two models tie on JSON-ok, pick the cheaper and faster. Quality theater is how you stay on the frontier for a classifier. If they tie on JSON-ok but the large model abstains better on the impossible slice, that is a real reason to keep large on the final-answer node only.

Note:Distillation and fine-tunes of small models are how grown-up teams lock in a router. Do not start there. Start with measurements on your JSON and your p95.