Curriculum/Tools & Function Calling
Timeouts, Retries, and Size Caps
Hung tools must die. Huge outputs must truncate. Retry reads. Do not blindly retry writes.
Every tool needs numbers:
- Timeout — kill the call, return
{"error": "timeout"} - Output cap — bytes, rows, or both
- Retry policy — reads yes (with backoff), writes only if idempotent
- Wall budget — max time for the whole batch, not only one call
Without numbers, “sandbox” is a mood. Without timeouts, one hung HTTP call freezes the whole agent. The loop is a client waiting on dispatch. Dispatch must not wait forever. The worker that runs the function owns the timer. A prompt that says “be fast” does not own a timer.
Hung tools must die. Huge outputs must truncate. Retry reads, not blind writes.
Timer owns the callJoeven’s try-it boxes already live under a ceiling: stdlib, no network, no pip. Production tools need the same kind of ceiling with explicit milliseconds and byte counts.
Timeouts are fail-closed
When the timer fires, you return timeout. You do not return the last partial byte as success. You do not hang the worker until the upstream feels like answering. You cancel if the client library can cancel. You still assume the world may have applied a write. Timeout is unknown outcome, not a clean miss.
Reads: retry with backoff, still under a total budget. Writes: retry only with the same idempotency key, or poll status. Do not mint a new refund because the socket dropped. The idempotency lesson is the reason this policy is safe. Without a key, poll or hand off. Do not “try again.”
Backoff is not a tight loop of twenty calls. Exponential, capped, with jitter if you have a clock. This classroom does not sleep. Production must.
Size caps are fail-closed too
Bytes, rows, or both. A search that returns 50,000 hits is a timeout of attention. Cap hits at 10. Cap JSON bytes at a number you pack anyway. If the handler cannot cap (a dumb SDK), the packer still truncates and flags. Prefer handler caps so you do not pay to download the ocean.
Row caps and byte caps catch different monsters. Ten huge rows still blow the window. A million tiny rows still blow the worker. Set both on list tools.
Error observations are small by construction. Do not attach the 2 MB body to too_large. Attach size, a preview, and a hint.
Retry is a table, not a feeling
| Error | Read | Write without key | Write with key |
|---|---|---|---|
| timeout | retry | poll or hand off | retry same key or poll |
| rate_limited | backoff | backoff if safe | same |
| invalid_args | no (fix args) | no | no |
| denied | no | no | no |
| not_found | no (usually) | no | no |
| internal_error | maybe once | poll | same key |
“The model asked nicely” is not a row. Session caps still win: max retries per call, max tools per turn, max writes per hour.
Classroom limits
Duration over 200 ms is timeout. JSON over 40 bytes is too_large with a truncated preview. Retry helper: get_job and search may retry timeout; refund only with a key. The numbers are tiny so the prints are obvious. Production numbers are larger. The branches are the same.
Run to execute this in your browser. Nothing is sent to a server.
What printed: a fast get_job is ok. A slow one is timeout. A huge log is too_large with a truncated slice. Retry get is true. Refund without a key is false. Refund with a key is true. After a real refund timeout, that last line still means “same key,” not “new invoice.”
What goes wrong
Timeouts only in the HTTP gateway so the worker still waits. Retries in three layers (client, dispatcher, model) that multiply. Caps in the prompt. Treating timeout as not_found. Retrying invalid_args. No total budget so backoff still runs for minutes. All of these freeze or double-submit.
How to test limits
Fake a slow handler with a duration argument — you already do. Assert timeout does not include ok: true. Fake a huge body, assert size and truncated keys. Table-test should_retry. For writes, assert a timeout path does not increment a “charged” counter unless a key replay is intended.
Numbers live on the worker, not in three other layers
If the HTTP client retries, the dispatcher retries, and the model retries, a single timeout becomes a storm. Pick one layer for write retries: the dispatcher with a key. Let the HTTP client disable automatic POST retries. Let the model see timeout and follow the table instead of inventing a new invoice id. Document that split or you will debug multiplied charges.
Budgets are the outer timer. A batch of reads can each be under 200 ms and still burn 30 seconds of wall clock. Cap the batch. Cap the job. When the budget is spent, stop even if the model wants one more search. Stopping is a runtime decision. Hung tools that ignore cancel still need the worker to abandon the wait and return timeout. You may still have to assume a write landed.
Size caps belong in the handler and in the packer. Handler caps save download cost. Packer caps save the context window. Row caps and byte caps catch different floods. List tools need both. too_large must not attach the ocean as detail. Attach size, a short preview, a hint to narrow.
Reads may retry timeout with backoff and jitter, still inside the budget. Writes retry only with the same key or via poll. invalid_args and denied never retry. Copy that table into tests. A feeling is not a policy.
How agents use this
Put the same limits in the worker that runs tools, not only in the prompt (“please be brief”). The model cannot enforce a timeout. The loop should treat timeout as a first-class observation and follow the retry table. If the budget is spent, stop. Stopping is a runtime decision.
When you add a tool, fill timeout_ms and max_bytes before you write the description. Missing numbers mean “defaults,” and defaults should be strict.
Watch out:A timeout is not “nothing happened.” Writes need a key or a status check.
Check your understanding