Dead Letters, Poison, Fairness
Max attempts then DLQ. Reject poison at the gateway. Per-tenant concurrency so one swarm cannot starve everyone. Slow enqueue when the vendor is 503.
Dead-letter jobs that exceed max attempts. A human (or a script) inspects them. Infinite retry is an outage you paid for. A payload that always crashes the worker (bad text, 50 MB JSON) must go to the dead-letter queue without taking the fleet down. Validate at enqueue in the gateway: max bytes, schema, tenant present.
One customer with a 10,000-PDF swarm should not starve everyone. Use per-tenant concurrency. Separate queues for interactive vs batch — that is an SLO choice, not hope. When the model vendor is 503, slow enqueue. Do not spawn more workers that will fail the same way.
Retries from the previous lessons are for transient failure. This lesson is for poison, fairness, and giving up.
How the box actually works
| Failure | Queue action |
|---|---|
| Transient tool 503 | Retry the slice with backoff + jitter |
| Parse error | Retry the model once, then fail the job |
| 4xx semantic | Do not retry; fail or ask a human |
| Repeated timeout on one tenant | Isolate / fair-queue so others live |
| Payload over max bytes / invalid schema | Reject at gateway → poison / DLQ with a reason |
| Max attempts exceeded | DLQ, page or ticket with job id |
| Vendor 503 across the fleet | Shed load: slow or stop enqueue, breaker |
Reject huge payloads at the door. One tenant cannot starve the rest.
Poison, fair, then DLQFairness is a scheduler property. A simple pattern: per-tenant semaphore (cap 2 running). Extra work stays queued for that tenant. Other tenants still pop. Two physical queues (interactive vs batch) keep FAQ jobs off the PDF runway.
Poison is a gateway property. If the worker can crash on parse, you already lost. Cap bytes, require tenant, require job schema, reject or DLQ at the door.
Owners: gateway owns validation. Runtime owns DLQ, attempt max, fairness. Product owns interactive vs batch SLO. On-call owns replay steps in the runbook before the first poison message.
Replay is a product: fix the crash, re-drive from DLQ with the same idempotency keys. If replay is “delete the row and ask the user to click again,” you will double or drop.
Poison vs fair vs DLQ are three different doors. Poison never runs. Fair waits its turn. DLQ already ran and lost. Mixing them — for example, treating fairness as poison — makes Beta look like a bug. Mixing them the other way — retrying poison as if it were a 503 — empties the fleet.
When the vendor is 503, a deep queue of work that will fail the same way is not resilience. It is a battery of retries you will pay for later. Slow enqueue, open the breaker, let interactive users see a structured error instead of a 20-minute wait.
A 50 MB poison ticket
A tool wrote a 50 MB observation onto the job and the next slice put it on the queue. Every worker OOM-killed. Autoscaler added workers. They died too. Interactive FAQs queued behind a corpse.
Gateway validation would have rejected the enqueue (POISON). Fairness would not have saved you if every worker loaded the same poison message — that is why poison must not be redelivered to the whole fleet. DLQ it on crash loop and prevent it at the door.
A second incident the same month: Acme’s batch swarm filled the only queue. Beta’s “how long are refunds?” sat for 20 minutes. Two queues + tenant caps. Beta ran. Acme waited. That is fairness.
Run to execute this in your browser. Nothing is sent to a server.
ok admits a small Acme payload. poison is POISON for a 500-character blob over a 200-byte cap — DLQ at the door. With Acme already at cap 2, fair is FAIR_QUEUE. Beta still gets ok True. Poison does not take the fleet. Acme cannot starve Beta. That is the whole lesson in four prints.
What goes wrong
Infinite retry. Retrying poison. One queue for all SLOs. Fairness by “please use the API kindly” in the docs. DLQ nobody reads. Replay that mints new job ids and new keys (double send). Slow consumers without a max payload. Vendor 503 + scale workers to 200.
Validation only in the worker: the crash happens before the validator.
Fairness that is global concurrency only: one tenant still owns all slots if they arrived first. Per-tenant caps exist so arrival order is not destiny. Interactive jobs on the batch queue inherit batch SLOs no matter how small they are — put the queue name on the job at enqueue and refuse to hop.
How to test it
- Oversize blob →
POISON, no worker call. - Missing tenant →
NO_TENANT. - Tenant at cap →
FAIR_QUEUE; other tenant admitted. - Crash loop fixture → DLQ after max attempts, not infinite.
- Interactive vs batch: a batch flood does not move interactive p95 in a load test.
Write replay steps and run them in staging on a fake DLQ item.
How agents use this
Interactive vs batch is an SLO choice; encode it as two queues. Write replay steps in the runbook before the first poison message. When the vendor is sick, shed load at enqueue — your queue is not a battery that stores infinite failing work.
Per-tenant caps are also a cost control: they pair with monthly dollar caps. A swarm should hit fairness before it hits the finance pager, or at least not take Beta down on the way.
Write the DLQ runbook while you are calm: how to inspect the payload without loading 50 MB into a laptop; how to confirm the key; who is allowed to re-drive. A Slack message of “just replay all of them” is how you double-email 2,000 people.
Check your understanding