Idempotency Keys
Build the key from job_id, step_id, and tool. Same key, same result, no second side effect. Do not put the model’s thought text in the key.
An idempotency key is a string the write tool uses to say “I already did this.” Same key, same result, no second side effect. Retries are normal. Double spend is the bug.
Build it from things that do not change when the worker retries: job id, step id, tool name. Do not include the model’s thought text — that changes and would mint a new key, which is a second refund with extra steps. Do not mint a random uuid on every retry. Random is the opposite of idempotent.
Read tools can be retried without a key. Write tools cannot. If you are unsure, it is a write. Email, tickets, refunds, calendar invites, “post to Slack,” filesystem append — writes.
How the box actually works
The tool service owns a ledger keyed by the string. First call: perform the side effect, store the result. Second call: return the stored result, do not perform the side effect. The worker always sends the same key for that step.
| Include in the key | Exclude |
|---|---|
job_id | Model thought / scratchpad prose |
step_id (stable) | Random uuid per attempt |
| Tool name | Tenant display name (use the signed id if you must scope) |
| Sometimes: hashed canonical args | Wall-clock timestamps |
Job, step, and tool. Thoughts do not go in the key. Random uuids are a second refund.
Same key, one writeArgs in the key: only if two refunds in the same step could be different amounts on purpose. Usually one write per step. If the model retries the same step with a different cent amount, that is a product decision: either reject the mismatch (“key reused with different body”) or treat it as a new step. Payment APIs often reject mismatch. Copy that behavior.
Owners: domain tool owns the ledger. Runtime mints the key from job + step + tool before the call and logs it on the span. On-call needs a runbook: how to look up a key in Stripe/email/your DB after a DLQ replay.
Pass through vendor keys when they exist. Do not invent a second ledger that can disagree with Stripe.
The worker mints the key before the HTTP call and puts it on the span, even if the call times out. “We are not sure it landed” is the whole reason for a key: the next attempt sends the same string. If you drop the key on timeout, you will pay twice “to be safe.” Safe is the same key.
Step ids must be stable across retries of the same decision. If the assembler re-numbers steps because a thought was inserted, you will mint a new key. Step id comes from the job’s checkpointed counter, not from len(thoughts).
A thought-in-the-key ticket
Someone built keys as job_id + thought[:40] so they would be “unique.” Every retry had a new thought. Every retry refunded. Uniqueness was the bug. The fix was boring: job_17:3:refund. A later step 4 refund (a second, real decision) correctly created a second row. That is not a duplicate. That is another action.
On-call documentation now says: if DLQ replay is safe, the key is in the span; if the ledger already has it, replay is a no-op; if you need a new side effect, you need a new step id, not a new random suffix.
Run to execute this in your browser. Nothing is sent to a server.
first is not a duplicate. retry is a duplicate with the same 1999 cents. other step is a new row because step 4 is a different action. Ledger size 2. Same key means no second payout. New step means a second payout on purpose. If those two sentences ever blur in a design review, stop and write this test.
What goes wrong
Thoughts in the key. Attempt number in the key (retry becomes a new write). Only-job-id as the key (step 4 cannot write). Dropping the key on timeout because “we are not sure it landed” — then you double. Client-generated keys from the model. Keys stored only in worker memory.
Mismatch: first call 1999 cents, retry 2000, ledger returns 1999 without telling anyone. Surface duplicate and the stored body. If mismatch should be an error, return KEY_MISMATCH and do not pay the new amount.
A second common mess: two services each keep a ledger. Stripe says paid, your table says not, the worker “fixes” it with a new key. Pick one ledger of record (usually the vendor’s, mirrored locally). Reconciliation is a batch job, not an extra refund in the hot path.
Clock-skewed step ids from “use the timestamp” collide or never retry the same. Integers on the job row are boring and correct.
How to test it
- Same key twice: one side effect,
duplicate: True. - Different step: two rows.
- Thought text changes, key does not (do not even pass thought into
key()). - Mismatch body: defined error.
- After process restart, ledger still has the key (real DB in staging).
Contract-test the vendor: send the same Stripe idempotency key twice in test mode.
How agents use this
Document replay for on-call: which keys are safe, which ledgers to check, how to re-drive a dead-letter item after a fix. Queues without that page are a bag of delayed incidents.
Never let the model invent the key. The worker mints it. The model may choose that a write happens; the platform chooses the identity of the write.
When you add a new write tool, copy this ledger pattern the same day. A “small” Slack post without a key will page you at 2 a.m. after a deploy.
On-call replay: open the span, copy the key, query the ledger / Stripe / email provider. If the row exists, re-driving the DLQ is a no-op and that is success. If you need a genuine second send, you need a new step id and a human decision, not a new suffix on the old key.
Check your understanding