Reference/LLM APIs
Tokens, context, and cost
Rough token math, why loops multiply cost, and what to trim first.
You pay for input + output tokens every call. Agent loops resend the transcript.
Rough counts (English)
| Thing | Tokens (rule of thumb) |
|---|---|
| 1 English word | ~1.3 |
| 4 characters | ~1 |
| JSON keys | worse than prose |
| Code | worse than prose |
Classroom estimate: int(words*1.3) + 4 per message.
Cost
USD = n_in rate_in / 1e6 + n_out rate_out / 1e6
A 20-step loop with a growing transcript is not 20× a one-shot; it is triangular (1+2+...+20 message-sizes). Trim.
Trim order (first to delete)
- Duplicate search hits
- Old tool bodies (keep last k)
- Thoughts (keep side trace)
- Few-shot examples
- System tool docs you already enforced in code
Never trim the user goal.
Context window
If the window is 8k and your handbook is 6k, RAG is mandatory. If the handbook is 800 tokens, you may stuff it — still cite chunks.
| Knob | Effect |
|---|---|
| smaller model for routing | cheaper steps |
| larger model for finish | better last mile |
| max_steps | hard cap on triangle |
Tip:Log n_in, n_out, usd, step on every call. You cannot optimize what you do not meter.