Queues and Retries

Workers crash. Deploys happen. Vendors return 503. A production runner is a queue consumer with at-least-once delivery: the same job slice may run twice. If that slice calls refund, you will double-pay unless you designed for it.

Assume at-least-once. Exactly-once is a myth at the edges. Therefore every write tool takes an idempotency key, checkpoints commit after the tool succeeds (or you use an outbox), and handlers are safe to replay. If a worker dies mid-slice, the message should reappear. Set the visibility timeout longer than the slice, shorter than forever. Heartbeat if a tool is slow.

This lesson is the queue’s behavior. The next lesson zooms in on the key itself. Dead letters and fairness come after that. You need all three; this one is why retries are normal rather than a scandal.

How the box actually works

A worker leases a message for a visibility window. During the window it runs a slice. If it finishes, it acks and the message is gone. If it dies, the lease expires and another worker gets the same slice. That is at-least-once.

Backoff without jitter is a stampede: every worker wakes on the same second and hits the tool again. Jitter is not decoration.

Checkpoint timing: if you checkpoint “I will refund” before Stripe returns, a crash retries and you might refund twice even with a key if you minted a new key. Checkpoint after success, or write an outbox row in the same database transaction as “slice started” with the key already chosen.

Owners: runtime owns the consumer, visibility, backoff. Domain tools own idempotency at the write. On-call owns “how many attempts before DLQ.”

Log attempt, job_id, and the key on every tool span. Future you will need them.

Visibility timeout is a number you can get wrong in both directions. Too short: two workers refund. Too long: a dead worker holds the slice for half an hour while the queue looks “in flight.” Heartbeats extend the lease while a slow tool is honestly working. If you cannot heartbeat, split the slice so the tool is a separate job with its own timeout.

Drain is a queue operation: stop leasing on the old SHA, wait until in-flight leases expire or checkpoint, then terminate. A rolling kill is a scheduled retry storm.

A double-refund ticket

A worker applied a 1999-cent refund, then crashed before ack (deploy killed the pod). The message reappeared. The second worker refunded again. Ledger had two rows. The model did not “remember.” The queue did its job. The write tool had no key.

After the fix, the key was job_17:step_3:refund. Two crashes after the write still left one ledger row. Backoff printed different delays because of jitter, so the payment API was not hammered in lockstep. The incident write-up ended in an idempotency test, not a prompt change.

Run to execute this in your browser. Nothing is sent to a server.

You should see three attempts (0, 1, 2). Attempts 0 and 1 crash after the write. new_write is True only once. Backoff values differ because of jitter (seeded so the demo is stable). Final ledger still has one 1999-cent refund and ok True. Two crashes after the write did not double-pay. If you deleted the if key not in ledger guard, the ledger would lie and this lesson would be a finance event.

What goes wrong

Visibility timeout of 10 seconds on a slice that calls a 30-second tool: overlapping workers, overlapping writes. Retrying 4xx semantic errors (bad invoice id) as if they were 503s: infinite nonsense. Resetting attempt counters when you re-enqueue. Checkpointing before the side effect. Backoff without a cap, then without jitter. Treating the model as the record of whether a refund happened.

Holding a database transaction open for the whole model call “to be exactly once.” You will stall the DB and still not be exactly once at the payment edge.

How to test it

• Crash injector: succeed write, crash before ack, redeliver — ledger size 1.
• Backoff: delays increase, stay ≤ cap, jitter differs across workers (do not assert exact floats beyond a seed).
• Visibility: a test that runs longer than the timeout must heartbeat or you assert duplicate detection via the key.
• Attempt counter lives on the job row.
• 503 retries; 400 does not.

If your payment API already supports idempotency keys, pass them through and test with their test clock.

How agents use this

Design every write tool for replay. Read tools can be sloppy; writes cannot. When you add a tool, ask “what happens if this slice runs twice?” If the answer is “email twice,” you need a key or you need to not send from the worker.

Deploys drain: stop leasing new messages on the old SHA, let slices checkpoint, then kill pods. A rolling kill without drain is a retry storm you scheduled.

Put queue depth and attempt histograms next to the traces. A job on attempt 4 is an incident in slow motion — DLQ is next.

Pass vendor idempotency keys through when they exist. Your ledger and Stripe’s ledger must not disagree. If you only remember locally, a crash after Stripe succeeded and before you wrote the row is a puzzle; the vendor key is the tie-break.