Jobs, Not Requests

HTTP requests want an answer in seconds. Agent loops wait on tools, humans, vendor retries, and the next model call. Those clocks do not match. Pretending they do is how you get a 504, a spinner, and a customer who clicks send again.

So the gateway enqueues and returns job_id. The client polls or streams events: queued, running, waiting_for_human, succeeded, failed. The job’s memory lives in the job store, not in the HTTP process. A deploy, a scale-in, or a laptop sleep must not be the delete button for a half-finished loop.

This is the same shape as work you already operate: payroll batches, image transcodes, “we will email you when it is ready.” Agents are not special. They are long-running work with a model in the middle. A 15-minute serverless timeout is a hidden architecture choice — usually the wrong one for anything that might wait on a human or a slow vendor.

How the box actually works

Two clocks. The request clock starts when the client hits the gateway and ends when you return job_id (or a first event). The job clock starts at enqueue and ends at a terminal status. They must not be the same variable.

Polling is honest: GET /jobs/job_1 returns status, events, and maybe final. Streaming is the same data pushed as server-sent events. Do not stream tokens as your only state. Operators and UIs need job events (“searching the handbook”, “waiting for approval”). A silent 90-second spinner is a product bug even if the answer is later perfect.

Cancel is part of the protocol. If the user closes the tab or hits stop, that signal must reach the queue. Cancel that only aborts the browser still spends tokens and may still refund. Cancel that is not idempotent will double-pay when the user hammers the button.

Checkpoints belong on the job row: last observation, step count, dollars so far, versions. The worker is replaceable. The row is not.

A tab-close ticket

A customer started “explain my last invoice and refund the overage if policy allows.” They closed the laptop in a tunnel. The old system held the request open; the load balancer cut it at 60 seconds; the worker died; the model had already emitted a refund tool call that never got an HTTP response, so the client SDK retried the whole chat. Two refunds. The ticket said “idempotency” (next lessons) but the first bug was request lifetime = job lifetime.

After the change, close-tab did nothing to the job. The UI on the phone showed running, then waiting_for_human. The customer approved. One refund. Support’s “support code” was job_17, which opened the trace. Nobody grepped Slack screenshots.

The product lesson: stream state, not vibes. “Searching the handbook…” is an event on the job. It is also how you prove the loop is alive without holding a socket for the whole loop.

Run to execute this in your browser. Nothing is sent to a server.

ACK is a job id while status is still queued. The first poll has no final answer. After two worker slices, poll shows succeeded and 5-7 days. The gateway function never searched, never called a model, and would have refused to enqueue if now had already passed http_deadline. The loop lived on the worker. The client only polled.

What goes wrong

Teams hide the job behind a “friendly” API that blocks until final. That reintroduces the request clock. Teams store events only in the websocket server. A reconnect looks like a new job. Teams implement cancel as status=canceled in the UI store but never nack the message, so the worker keeps spending.

Serverless is a special trap. If your worker max is 15 minutes and HITL can wait an hour, you do not have a HITL product. You have a timeout. Wakeups, approvals, and vendor callbacks must re-enqueue a slice, not resume a frozen lambda.

Another failure: poll that returns the entire transcript every time. That is a cost and a PII leak. Return events since a cursor. Put blobs behind signed URLs.

How to test it

• Gateway: enqueue under deadline; refuse when now >= http_deadline; never call tools.
• Poll: queued job has no final; after slices, status and final match the store; unknown id is 404, not an empty success.
• Cancel: mark canceled, worker’s next slice exits without a write tool; a second cancel is a no-op.
• Kill the HTTP process in a test after ACK; assert the job row still exists and a new worker can finish it.
• UI contract: every non-terminal poll includes a human-readable last event so the spinner is never silent.

Chaos: restart workers between slices. The job must continue from the checkpoint, not from the user’s last HTTP body.

How agents use this

Treat the product as a job viewer. The chat widget is a client of job_id. Mobile, email, and a staff admin screen all poll the same store. That is how a human on-call takes over a waiting job without stealing a socket.

Stream state the way a build system does: queued, running with a step name, waiting, done. Users can wait a long time if they know the machine is alive. They will not wait a silent spinner.

Wire cancel all the way down: UI → gateway → job flag → worker. Measure “canceled jobs that still called a write tool.” That number should be ~0. If it is not, cancel is theater and cost still climbs.

When you add HITL, it is another status and a wakeup, not a longer request. The approval page loads the job by id. The gateway that created the job is long gone. That is the point.