Tokens and Cost

Vendors charge for tokens, usually both directions, often at different rates. Input (the prompt: system, history, tools, observations) is cheaper. Output (the completion) is the tip. An agent loop resends the growing prompt every step, so the bill is closer to sum of prompt sizes than to “one chat with a final answer.”

If input is 3 dollars per million tokens and output is 15, a 4k-prompt / 200-completion turn is cheap once. At step 15, when the prompt is 20k because you never summarized, you pay for 20k input again, plus another completion. The user still sees one ticket. Finance sees a furnace.

True counts need the model’s tokenizer — the same one the Transformers track introduced. Character/4 is a rumor. Code, JSON, and non-English text are often worse. For planning, a crude estimator is better than nothing. Split on spaces, then treat long words as extra pieces. For billing, believe usage on the response. Mixing a notebook estimate with a production invoice is how you “stay under budget” until the card statement.

One turn vs a loop

A single chat is: pay for this prompt, pay for this completion, done. An agent is: pay for prompt 1, completion 1, then prompt 2 which includes prompt 1 plus the tool result plus completion 1, and so on. Even if completions stay short (“I’ll call get_job”), input climbs.

That is why “we only generate 80 tokens per step” can still be expensive. The 80 is the tip. The iceberg is the resent spec, the tool docs, the RAG chunks, and every past observation.

Run to execute this in your browser. Nothing is sent to a server.

single turn is tiny. Then each step prints a climbing prompt_tok. The completion string never grows; the prompt does. Loop total is many times the single turn. Toy rates, toy tokenizer — the shape is real. If you paste a realistic tool payload into a vendor tokenizer UI and multiply by expected steps, you will feel the same curve with honest numbers.

Hidden multipliers

These do not show up in a playground screenshot of “one reply”:

• Retries and timeouts that still bill a partial completion
• Sampling n greater than 1 (three completions for one question)
• RAG: you pay to embed and to stuff chunks into the LLM
• Vision: images tokenize as many tokens, often more than you expect
• Playground clicking without max_tokens, then copying that habit into prod
• Long “reasoning” traces the vendor streams into the bill as output tokens
• Tool result dumps: a 40-line stack trace in a tool message is a 40-line prompt tax on every later step

Output is often priced higher than input. Long reasoning can help quality on some tasks. It definitely helps the invoice. Cap it. Treat “thinking tokens” as a product choice with a budget, not as free intelligence.

Compare vendors on your mix of input vs output, not on a blog’s “dollars per million.” A model that is cheap on input and verbose on output can lose to a pricier quiet model. Measure completion tokens per successful task, not per call. A call that fails JSON and retries is two calls.

Cached input tokens are sometimes cheaper. Caching helps only if the prefix is stable — next lessons in this part.

Per-task metrics beat per-call vanity: tokens per resolved ticket, dollars per successful get_job, dollars per failed JSON retry. A dashboard of “average tokens per HTTP” hides the loop. Group by trace id, sum usage, then average those sums.

Vision and audio, if you add them later, are token bombs. Count them before you put screenshots in every support ticket. Embeddings have their own meter; stuffing eight chunks still hits the LLM meter too.

A walkthrough: search tool economics

Before adding a search tool, paste a realistic result into the vendor tokenizer. Eight snippets of 400 tokens each is 3,200 tokens. Called three times, that is 9,600 tokens of observations, resent as the prompt grows, plus the snippets still sitting in history if you never trim. A goal that cannot be reached within your cap should fail closed, not wander through a fourth search.

Maya’s support agent had a handbook dump in every call “in case it helps.” Quality did not move. Input tokens tripled. The fix was packing (later) and this lesson’s habit: estimate before you add.

What goes wrong

• Budgeting in characters. JSON and code will blow the window and the bill.
• Ignoring loop shape. “Average tokens per call” hides step 12.
• Uncapped output. Reasoning novels, then a JSON afterthought.
• Logging full prompts to a paid log product and paying twice: once to the LLM, once to logs.
• Comparing vendors on list price only. Your mix and your verbosity decide the winner.

How agents use this

Store usage on every step. Dashboard: tokens per successful ticket, not tokens per HTTP. Before a new tool, write the expected result size in tokens and the expected call count. If that product exceeds the cap, the tool is too chatty or the path is wrong.

A spend cap (next lesson) needs these numbers. Without usage, a cap is a vibe. With usage, a cap is arithmetic.

Trim observations. Summarize history as decisions and ids, not as chain-of-thought. Put fat RAG chunks only on the step that needs them. Those are context-engineering moves; they exist because of this cost curve.

Before a tool PR merges, paste a realistic payload into the vendor tokenizer (or your local tokenizer for open weights) and multiply by expected steps. Write the number in the PR. If nobody can name the number, the tool is not ready.

Note:Cached input tokens are sometimes cheaper. Caching helps only if the prefix is stable — next lesson after spend caps.