Cost and Latency
Agent cost is steps times growing input tokens. Cap it, trim the scratchpad, route easy jobs small, and write SLOs that admit the loop is slower than a FAQ.
Agent cost is steps × (input tokens + output tokens) × price, plus tools, plus retries. Input tokens dominate because you resend the transcript. Latency is serial model calls plus serial tools, unless you parallelize independent work. If you do not cap this, a single stuck job becomes a finance event.
Users feel time to first token and time to first useful event (a tool result or a status line). They do not feel your average model latency. Research jobs are not FAQs. Say so in the product. An SLO that pretends every agent turn is a 200 ms autocomplete will make you cheat with worse answers or unpaid heroes.
This lesson is how spend and wait actually form on the job. The next lesson turns those numbers into stop conditions. Routing between small and large models is an ops lever here — not a recap of how to train a router.
How the box actually works
The default cost curve is transcript growth. Step 1 sends a short prompt. Step 12 sends step 1 plus eleven observations. Retries multiply both dollars and wall time. A 503 that you retry three times on a 20k prompt is a choice.
| Lever | What it does | Owner |
|---|---|---|
| Route easy tickets to a small model | Cuts price and often latency | ML platform + product |
| Short observations; blobs out of band | Stops 8k HTML dumps in the window | Tool authors |
| Summarize / slot the scratchpad | Caps tokens in per call | Runtime assembler |
| Fewer tools in the prompt | Schemas are tokens | Product |
| Do not loop when one call would do | Workflow vs agent | Product |
| Cache deterministic tool results (TTL) | Avoids repeat search | Runtime |
| Parallelize independent tools | Cuts wall time, not always dollars | Runtime |
Each step resends the window. Cap it, or one stuck job is a finance event.
Transcript growthEstimate before the loop using a crude planner (expected steps × typical tokens). Update after each span from real tokens_in / tokens_out. Show both on the job. Kill when you would exceed the cap (next lesson). Courtesy emails to finance are not a control.
Latency budget: admission (enqueue), queue wait, each complete, each tool, HITL. Chart them separately. “The model is slow” is often “we sat in queued for ten minutes.”
Owners: runtime owns estimates and assembler trim. Product owns “this job type is a research SLO, not a FAQ SLO.” Finance owns unit price tables. ML owns which model id a route may pick.
Interactive vs batch is a latency control you will encode as two queues later. Here, know the numbers: FAQs should show a first useful event in a few seconds (even if that event is “queued”). Research jobs should say “this can take minutes” in the UI so the SLO is honest. Mixing them on one worker is how a PDF swarm makes refund questions look like a model outage.
Retries belong in the estimate. A policy of three tries on a 20k prompt is a product choice with a price. Count them on the job as spend, not as free resilience.
A stuck-job ticket
A single “summarize these 40 PDFs” job sat in a loop, stuffing each PDF into the scratchpad. Twelve long steps at ~4000 input tokens estimated well above a 0.02 dollar cap in the toy rates, and much worse in real rates. Nobody had a cap. The job finished six dollars later with a vague summary. Queue wait for everyone else climbed because the worker was busy.
The fix was not a sterner prompt. It was trim (keep recent events under a token budget), blobs by id instead of inline text, a small-model route for “extract title,” and a cap. The product copy changed too: “this can take a few minutes” instead of a FAQ-shaped spinner.
Run to execute this in your browser. Nothing is sent to a server.
The long transcript fails a 0.02 cap (cap check FAIL). Trimmed context keeps the newest events that fit ~120 tokens; trimmed events is smaller than 4. After trim, twelve steps cost less because each prompt is shorter. Retries would multiply both pictures. The lesson on the screen: uncapped growth fails first; trim is an ops control, not a style choice.
What goes wrong
No estimate. Estimate once at the start and never update. Trim that drops the goal or the spec. Caching tool results forever so policy changes never appear. Routing every job to the frontier model “just in case.” Parallel tool calls that duplicate writes. SLOs copied from a JSON API.
A cheap model that fails billing goldens is not cheap. You pay in HITL and in refunds. Measure cost per successful job, not cost per call.
Verbose “reasoning” traces billed as output. Cap them. They are a product choice.
How to test it
estimatefixtures: known steps and tokens → known dollars.- Trim: under budget; does not drop the latest observation if it fits; drops oldest first.
- Cap check: the 4000-token path fails 0.02; the trimmed path can pass.
- Span integration: after a job, summed
usd_estmatches the job’s running total within rounding. - Latency: a test job records queue_ms vs llm_ms; a queue-only delay must not be blamed on the model in the summary.
Replay a production-shaped transcript with trim on/off and diff dollars.
How agents use this
Put the estimate on the job before the loop, update it after each span, and kill the job when it crosses the cap. Show users time-to-first-event. Separate interactive vs batch queues so a PDF swarm does not steal FAQ latency (next part).
Route easy tickets to a small model in config, with a version stamp, not in a prompt that says “be cheap.” If billing goldens fail on the small model, do not silently stay there — that is the next lesson’s silent-downgrade bug.
When you add a tool, paste a realistic result size into the estimator and multiply by expected steps. If it cannot fit the cap, the tool must return less, or the job type must declare a higher cap on purpose.
Show cost on the staff admin for every job, even tiny ones. Habit beats a quarterly surprise. If product wants longer “reasoning” output, that is a flagged, versioned, capped choice — not a model default you forgot to turn off.
Check your understanding