Error Recovery

Tools fail. The loop must not fail as a personality. “Try harder” is not a policy. Timeouts, retries with a cap, circuit breakers, and fail-closed are a policy.

The tools track already cares about retries and keys. This lesson is what the loop does when the executor returns an error observation. You log the error as an observation the assembler can see. You do not hide failures in a thought. You do not pretend the tool returned success. You do not eval a fallback.

A circuit breaker stops calling a tool that just failed N times. The model does not get a 12th try at the same flaky API. Caps and breakers are recovery. Infinite retry is a furnace.

Timeouts

Every outbound call has a timeout, including the model. A hung search is a hung step. Normalize to {"error": "timeout"}. Retry once if the tool is a read. Then skip or handoff. Writes: retry only with an idempotency key. Counting the retry toward the step budget is mandatory. A “free” timeout retry is how 8 steps become 40.

Jobs: timeout can be a wakeup. Park, retry later, do not hold the worker.

429 vs 400 vs 500

400 — bad args. Repair once (parser/schema) or fail. Same body will fail again.

429 — slow down. Backoff. Still a budget item. Do not spin.

500 — their fault, maybe flaky. Retry with cap, then breaker.

Unknown tool — parser, not retry. Retrying launch_nukes is not recovery.

The loop should branch on a small error enum, not on a stack trace pasted into the prompt. Truncate traces. Models will “fix” HTTP if you dump it. Adapters should already have wrapped status codes.

Circuit breakers

The breaker wraps a tool (or a host). Fail N times → open. Open means later calls return circuit_open without hitting the world. After a cooldown you may half-open (one probe). This lesson’s box opens and stays open so you can see the third call never touch flaky.

Per-tool breakers beat one global breaker (search down should not block get_ticket). Per-run counters belong on state so jobs resume the count.

When open, the observation is circuit_open. Stop may handoff. The model should not invent a search snippet. Observe-before-finish still applies: the obs is an error, finish that ignores it is a guess.

Same error twice

If the error is identical (same tool, same args, same timeout), you are in thrash territory (next part). Breaker plus thrash plus budget is three backstops. You want all three. Personality “surely this time” is none of them.

Fail closed

Unknown tools, bad JSON, open circuit, budget, guard failed: the world does not change. Recovery is not “guess a nearby API.” Recovery is retry with policy or stop.

Backoff, what the assembler sees, and half-open

Backoff for 429 is wait-then-retry, not tight-loop retry. In a request, waiting might be a short sleep with a cap. In a job, waiting is wake_when plus a clock. Either way, count the attempt. Exponential backoff without a cap is a polite furnace. Cap the waits and the tries.

What the assembler sees after a failure must be the error observation, truncated: {"error": "timeout"} or circuit_open. If you hide it, the model will finish from the last happy thought. If you dump a stack trace, the model will try to patch your Python. Neither is recovery. Structured errors are recovery.

Half-open: after cooldown, allow one probe. Success closes the breaker (fails = 0). Failure opens again. Do not half-open ten tools at once on the same host if they share a quota. Per-host breakers are allowed; name them in the trace.

Repair vs retry: 400 with missing field → repair prompt once (parser). 400 with unknown enum → fail. 5xx → retry. Timeout → retry once for reads. Unknown tool → never retry as if the world were flaky. These branches belong in one function recover(error, tool, args, counts) so the loop does not grow if-else novels.

Do not recover by switching to a more dangerous executor (computer use, eval). That promotion is in the computer-use lesson as a trap. Recovery stays inside the same catalog, or handoff.

Log the error enum on the trace row every time. If operators only see a thought that says “search failed,” you will not know timeout from 400 from circuit_open. Recovery policy cannot fire on a poem.

Common mistakes

• Retry forever.
• Retry 400 with the same args.
• Hide errors in thoughts.
• Pretend success.
• One global breaker for every tool.
• Not counting retries as steps.
• exec() as a fallback.

Run to execute this in your browser. Nothing is sent to a server.

Two timeouts open the circuit. The third call never hits flaky — it returns circuit_open. The function would have succeeded on the third world call (n["i"] < 3), but the breaker does not care. That is the point: you stop paying the same failure. A half-open design could probe later. This box shows the cap.

Broad except Exception is for the demo. In production, catch timeouts and 5xx-shaped errors, not KeyboardInterrupt. Still fail closed.

How agents use this

Same retry budget as the tools track, owned here by the loop. Log error as an observation the assembler can see. Do not hide failures in a thought.

Pair with checkpoints: after two timeouts, save state, maybe wake later, do not lose phase. Pair with HITL: an open circuit on refund is a handoff, not a silent skip that claims done.