Routing Models

A router is a cheap decision: which model runs this step? It is not a personality. It is a function you unit-test.

• Extract fields, classify intent, fill tool JSON → small
• Multi-step debugging, ambiguous policy, novel code → large
• Embeddings → an embedding model, not a chat model
• Rerank 20 chunks → a reranker, not a 70B chat

Fallback: if the small model’s validator fails, retry once on the large model. Measure how often you escalate; that is your real savings. 80% escalate means you do not have a small model. You have a delay.

Rules, embeddings, or a tiny LLM can be the router. Start with if kind in .... Add ML when the ifs rot. If you have no eval yet, do not route in production — start with one model and log what a router would have chosen.

When to go large anyway: the small model fails a frozen eval slice; the user-facing explanation is the product; safety-sensitive gray areas (still with a human gate).

Run to execute this in your browser. Nothing is sent to a server.

always large is the higher bill. routed is cheaper because classify and JSON tool calls went small. decisions should show small, small, large, large. Real traces are mostly classify-and-fill, not novel science. Even this toy mix wins. Your production mix will win harder if you actually look at it.

Escalate path (not in the toy): small JSON fails validate → one large retry → still invalid → handoff. Infinite small retries burn tokens and still fail. One measured escalate is cheaper than a poetry loop.

Do not route embeddings through chat. Do not ask a 70B to score 20 chunks when a reranker exists. Those are different products (RAG track). The router’s job is to not send them to chat.

Shadow first, then cut traffic

Before a router changes production, log what it would have chosen for a week. That is shadow routing: the live call still goes to today’s model; the span records would_have=small or would_have=large. If the validator would have failed on the small model’s hypothetical output, you cannot know without actually calling small — so a honest shadow is: send a sample of traffic to small in parallel (read-only steps only) and compare JSON fail rate and latency. Never shadow a write. Refunds are not a/b tests.

A week of spans answers the only question that matters: what fraction of steps are classify/extract/JSON, and what is small’s fail rate on those steps? If 90% of steps are JSON and small’s fail rate is 2%, you will save money. If 90% are “debug this incident,” a router that defaults to small will escalate constantly and you will pay for two models per hard ticket.

Escalate once, then stop

Write the fallback as code, not as a prompt that says “try a smarter friend.” Sequence:

1. Small model, temperature 0, schema on.
2. Validator fails → one large call with the same messages plus the schema error.
3. Still invalid → handoff. Do not return to small. Do not raise temperature.

Count escalated=true on the span. Alert if the daily escalate rate jumps. A spec change that made JSON harder should show up here before finance sees the large-model bill.

Spend caps still apply. Escalating to large can be the call that trips max_usd. If the cap would be exceeded by the large retry, skip it and hand off. The cap is not “unless we are almost done.”

Embeddings and rerankers are not chat

If task.kind == embed, you call an embedding endpoint. If task.kind == rerank, you call a reranker. Putting those through complete(messages) is how you pay chat prices for a vector. The router should refuse unknown kinds rather than defaulting to large chat. return "large" at the bottom of the toy is a safety default for chat tasks, not a sink for every string named “model.”

A crew of agents can burst 8 POSTs per click. Route the 7 cheap ones to small so the 1 hard plan still has rate-limit headroom on the expensive SKU.

What goes wrong

• LLM router with a huge prompt that costs more than the small model you were saving.
• No escalate cap.
• Routing in prod with no eval.
• Same model string for embed and chat because the vendor dashboard listed both.
• 80% escalate ignored because the architecture diagram looked smart.

How agents use this

The router is a function you unit-test. Log model= and routed_as= and escalated= on every span. A week later you will know whether “always large” was fear or evidence. Next tracks put tools, RAG, and planners on those nodes.

Escalation rate is a KPI. Treat it like JSON fail rate. If it spikes after a spec change, the spec got harder or the small model got worse — not “users are dumber.”

Unit-test route with the four kinds in the tryit plus: missing kind (should not crash; default large or reject), embed (must not return a chat SKU), tool_call without schema (decide: small-with-prompt-JSON or large — write it down). Tests are cheaper than a month of “we thought JSON went small.”

When you add a new step kind, you add a row to the router table and a row to the eval. A kind with no eval is how “debug” silently becomes 40% of spend.

Config the model names outside the function: SMALL=..., LARGE=.... The router returns a role, not a vendor string. Swapping vendors then does not rewrite if-statements. Choosing-models and this lesson share that adapter: complete(messages, model=role_to_id[role]).

Tip:Start with if-statements. Add ML when the ifs rot. 80% escalate means you bought a delay, not a router.