Checkpoints You Can Replay

A checkpoint is a snapshot after a successful step:

• run_id, step number
• typed state
• last observation
• plan status (if you have a plan)

On crash, load the last good checkpoint. Do not replay side-effecting tools unless they were idempotent (tools track). Prefer “resume from apply” over “search the web again.” Checkpoints exist so you do not buy the whole furnace twice.

Jobs save state; checkpoints are the versioned saves inside the run. Latest is what you resume by default. Named step numbers are what operators use: “replay from step 4 with a stubbed refund tool.” If you only have latest, you cannot go back. If you have no checkpoints, crash = start over.

What to snapshot

Copy typed state (phase, ids, facts, pending freeze). Copy last obs. Copy plan statuses. Copy steps_used so the budget survives. Do not copy a live socket. Do not copy secrets in plain sight if you can point to a vault — but do not skip the snapshot because redaction is hard; redact and save.

Shallow-copy dicts at least. Nested objects need a JSON round-trip like freeze. The live box uses dict(state) which is shallow; keep checkpoints JSON-serializable so you do not lie to yourself.

Resume vs replay

Resume — load latest, continue next_todo or next ReAct step. Do not re-run tools that already succeeded.

Replay — load a past step, maybe stub tools, run forward for debugging. Dangerous on writes. Stub refund. Do not hit production Stripe on replay.

On crash at step 5, load the last good checkpoint (4 if 5 never saved). Re-run only if the tool is safe or keyed. Missing step 9 returns None — fail closed, do not invent a state.

Side effects

Search again wastes money and can change hits. Refund again is an incident. Idempotency keys make a second refund a no-op. If you do not have keys, do not replay writes. Operators should see “already applied” from the store, not from a thought.

The assembler after resume should see the scratchpad you saved, not an empty brain. Memory belongs in the checkpoint or in a store keyed by run_id.

Fail closed on missing

resume_from(9) is None. Do not default to empty gather. Do not default to done. Empty gather plus a write tool is how you refund twice from a “helpful” resume. None → operator error → handoff.

Frameworks

LangGraph checkpointers, Temporal histories, and your JSON column are this. The brand is furniture. Operators should still say “replay from step 4.” If the framework cannot export a typed snapshot, you do not have a checkpoint you can operate — you have a log you hope to parse.

What “good” means, and how replay stubs work

A checkpoint after a successful step means: parser blessed, executor returned (even if the obs is an error you are willing to keep), state applied, then save. Do not save mid-execute with a half-refund. Do not save only after finish. If you crash between refund and save, keys save you; if you have no keys, you have a hole — prefer execute then save in a tight pair, or save a “doing” status then “done” (two checkpoints). Doing without a timeout sweeper is how jobs stuck forever.

Last good checkpoint on crash at 5: if 5 never saved, load 4. If 5 saved an error obs you accept, 5 is good. If 5 saved a corrupt blob, fail closed and handoff. Corruption is not “use gather.”

Replay for debugging: load step 4, replace refund with a stub that returns {"ok": true, "stub": true}, run forward in a non-prod worker. Label the run replay_of. Do not write replay results into the original run_id. Operators compare traces. Stubbing search is usually safe. Stubbing refund in prod is not a replay; it is an incident.

Memory and plan status belong in the snapshot or they will not resume. A checkpoint of phase apply without ticket_id is a broken object — do not save it. Validate the typed state on save the same way you validate on load.

Keep more than latest if you can afford it: last N steps, or every step until a retention policy. Storage is cheaper than re-searching. Redact secrets in snapshots the same as traces.

Resume is the default operator verb; replay is the debugger’s verb. Mixing them in prod is how stubs become real refunds. Name the entrypoint: resume(run_id) vs replay(run_id, from_step, stubs).

Common mistakes

• Only latest, no step index.
• Restart from 1 always.
• Replaying writes without keys.
• Inventing state when the id is missing.
• Checkpointing thoughts but not phase.
• Mutating the saved dict in place (alias again).

Run to execute this in your browser. Nothing is sent to a server.

Step 2 is latest (apply). Replay from step 1 still has gather. Missing steps return None — fail closed, do not invent a state. Two checkpoints, two phases. That is how you avoid searching the web again just to reconstruct T1.

The box keeps CKPT in a list. A job store would key by run_id and step. Same idea. Copy on save and copy on load so later mutations of state do not rewrite history — here we assign a new dict before checkpoint 2, which is clean. If you mutated the same dict, you would need the copy even more.

How agents use this

LangGraph checkpointers, Temporal histories, and your JSON column are this. Operators should be able to say “replay from step 4 with a stubbed refund tool.”

Error recovery next assumes you can stop calling a flaky tool without forgetting where you were. The checkpoint is that memory. The circuit breaker is the policy. Together they are how loops survive the world.