Spend Caps

max_steps is not a budget. An agent can burn a week of tokens in eight verbose steps: a huge handbook, three searches, a reasoning novel, a retry. Put money (or a token ceiling you convert to money) next to steps.

A simple rule:

1. Each call costs prompt_tokens in_rate + completion_tokens out_rate (plus any tool you pay for separately).
2. Before you call, estimate the next prompt (you already have the messages) plus a worst-case completion of max_tokens.
3. If spent + next_estimate > cap, do not call. Handoff or fail closed.
4. After the call, add actual usage to spent. Estimates are for the gate; usage is for the books.
5. Cap keys in the vendor dashboard too. Two layers. A runaway loop should hit your code first and the vendor second.

Fail closed means: stop, tell the user you cannot finish, do not guess the last action to be “helpful.” A refund you cannot afford to think about is not a refund you invent.

Caps are stop conditions

You already want max_steps so the loop cannot run forever. max_usd is the same idea in currency. Some teams cap tokens instead of dollars so they do not rewrite code when prices change. Either works. What does not work is a spreadsheet that nobody reads at runtime.

Alert when daily tokens exceed a baseline (yesterday’s p95, or a fixed quota per tenant). Per-trace max_usd belongs next to max_steps. Per-tenant daily caps stop one noisy customer from eating the account. Vendor dashboard caps stop a leaked key from becoming a five-figure night.

Run to execute this in your browser. Nothing is sent to a server.

Early steps are ok and spent climbs. A later fat prompt would push over CAP, so the log records stop with the step number, the spent so far, and the call you did not make. That stop is a success. You did not keep calling. In production, convert that into a user-visible handoff and a trace event budget_exhausted.

The toy uses actual usage as if you knew it before the call. Real code should gate on an estimate (prompt known, completion assumed max_tokens), then correct with usage. If you only add usage after a 4k completion, you can overshoot. Set max_tokens small on tool steps so the overshoot cannot be a novel.

Estimates when adding a tool

When you add a tool, estimate typical result size — that estimate is the unit economics of the path. A get_job that returns four fields is cheap. A dump_logs that returns 20k tokens is a budget weapon. Maybe the tool should return a summary plus a link. Maybe only a debug role may call it.

Retries count. Two 429-wait-then-success cycles are two LLM bills if you retried the model, plus whatever the vendor billed for partials. Put attempt numbers on the trace next to spent.

Per-tenant daily caps sit next to per-trace caps. A noisy customer with 2,000 tickets should hit a daily ceiling without taking down everyone else’s budget. Vendor dashboard caps are the last backstop when your process dies and the loop does not.

Estimate the next completion as max_tokens, not as “it usually writes 80.” The gate is conservative on purpose. After the call, add actual usage. If you only add usage after the fact, one 4k surprise overshoots. Keep tool max_tokens small so the surprise cannot be a novel.

Rates live in config, not in comments. When the vendor changes prices, you change config and the cap still means dollars.

A walkthrough: the 3 a.m. loop

A planner never emits finish. Each step writes a long thought and a search. max_steps is 30. Without a dollar cap, 30 fat prompts land at 4 a.m. With max_usd=0.50 on the trace, the loop stops at step 6 and pages a human. The bug is still real (no stop rule in the spec). The cap is the blast radius. Blast radius is what production is for.

What goes wrong

• Caps only in a spreadsheet. Wishes do not stop HTTP.
• Gating on steps only. Verbose steps still ruin the month.
• Using list prices from last year. Rates move. Config the rates.
• One global cap for a multi-tenant product. Noisy neighbor problem.
• Hiding remaining budget from developers. Then nobody notices a packing regression until finance does.
• Continuing after the cap “just this once.” That is how exceptions become the policy.

How agents use this

Show remaining budget on the trace UI for yourself, not necessarily the user. Finance will ask. “The model was thinking” is not an answer. usage on every step is.

When the next call would exceed the cap, skip it. Same code path as hitting max_steps. Do not lower temperature and try anyway. Do not delete usage from the log to “look cheaper.”

Tests: feed a fake client that returns huge prompt_tokens and assert the third call never happens. Caps that are not tested are decorations.

Show remaining budget on the trace UI. When the next call would exceed the cap, skip it — same path as max_steps. Do not lower temperature and try anyway. Do not delete usage from the log to “look cheaper.”

Wire the stop to user copy: “I could not finish this within budget; a human will take it.” That sentence is a feature. Wandering is not.

Tip:Caps that only live in a spreadsheet are wishes. Caps in code stop the loop. Vendor dashboard caps are the backstop, not the only layer.