Production Architecture

A production agent is a distributed system that happens to call a model. Users still see a chat box. Underneath, a request is authenticated, a job is written, a worker pulls a slice of work, tools run in their own services, and a control plane can stop the whole thing. If your architecture is “one function that loops until done,” you will learn about timeouts from your users, and about deploys from a half-finished refund.

This track is about operating that system. Other tracks already covered how the loop thinks, how retrieval ranks chunks, and how eval goldens are written. Here those ideas show up only as fixtures on a job: a prompt version, a tool name, a CI gate. The work is the boxes around the model — who owns them, what they store, how they fail, and how you stop them.

The shape is boring on purpose. Boring is what on-call can draw at 2 a.m. Fancy graphs with twelve model personas are not an architecture. Six named boxes with owners, timeouts, and budgets are.

How the box actually works

A request does not “run the agent.” It admits work. The gateway checks login, rate limits, and payload size, then writes a job row and pushes an id onto a queue. It returns job_id immediately. It does not hold the client for a 40-step loop.

The model is one box. It is untrusted. It suggests JSON. Tool services decide. If you cannot name the owner of each box, the agent becomes everyone’s weekend: the person who wrote the prompt is paged for a queue outage, and the person who owns Stripe is paged for a template typo.

Workers run a bounded slice, then save a checkpoint. A slice might be one model call plus one tool, or a short inner loop with a step cap. The job store is the truth, not the worker’s RAM. Tool families are their own APIs with their own auth — search is not imported next to refund inside the worker “to keep it simple.”

Control plane is not optional chrome. Human-approval UI, feature flags, prompt bundles, and a global agents.disabled live here. If money can move (model spend, refunds, SMS), that box needs a budget as well as an uptime target.

A Friday ticket

Acme’s support agent went live as a FastAPI handler that called the model in a while-loop until finish. p95 was fine in staging because staging jobs were two steps. On Friday a deploy restarted the pods. Forty in-flight conversations died. Three customers had been mid-refund. The ledger showed one refund applied twice (the client retried the HTTP call) and two that vanished (the process died after the model chose refund but before Stripe returned).

The ticket was titled “make the prompt more careful.” The actual fix was the diagram above. The gateway started returning job_id. Refund moved to a tool service with an idempotency key. The worker loaded the job, ran one slice, wrote a checkpoint. The Friday deploy became a drain-and-restart instead of a massacre.

That is what “the model is one box” means in a real week: you stop asking the template to survive process death.

Run to execute this in your browser. Nothing is sent to a server.

The HTTP call prints {"job_id": "job_1"}. The queue held that id. The worker ran later, set status to succeeded, and stored the handbook sentence. Nothing in handle_user called the model or a tool. That split is the architecture. If you collapse it so the gateway runs worker_once, you are back to Friday’s deploy.

What goes wrong

The usual failure is gravity toward one process. A new tool is “just a function” in the worker. A new secret is “just an env var” on the same box. A new long step is “just wait in the request.” Each shortcut is locally rational and globally an incident.

Other patterns: no owner for the queue, so poison messages retry forever. No owner for traces, so nobody can answer “why did job_17 spend this?” No owner for flags, so the kill switch is a Slack message to whoever has kubectl. Staging that skips the queue “because it is slower” trains you to ship an architecture you never ran.

Multi-region without a job store is the same bug at larger scale: memory on a box in one city is not a store.

How to test it

You do not need Kubernetes to test the shape. You need assertions on boundaries.

• Gateway tests: a call returns job_id in well under your HTTP deadline; the job row exists with queued; the queue contains the id; the handler did not invoke tools.
• Worker tests: given a job id, one slice mutates status and writes a checkpoint; a second process can load that checkpoint.
• Isolation tests: the worker dict of env vars does not contain Stripe; the refund service does.
• Owner tests (yes, really): a markdown diagram in the repo names a team per box. CI can grep that the names still exist. Folklore diagrams rot.

Run a game-day where you kill the gateway mid-request and prove the job is still in the store. If that sentence is hard, the architecture is still a function.

How agents use this

Draw the six boxes for your team and write the owners next to them. Put the drawing in the runbook, not in a slide that died after launch. Every new tool must declare which box it lives in. “The worker will just call Stripe” is a rejected design, not a shortcut.

If a box can spend money, give it a budget and a metric: model dollars per job, refunds per hour, SMS per tenant. Uptime without a budget is how a “healthy” worker burns the month.

When you add a feature — human approval, a second model, a batch channel — add it as a box or a flag on an existing box. Do not add it as a longer system prompt. The prompt is config for the model box. It is not the queue.

On-call uses this diagram as a sorting hat. Timeout at the edge → gateway. Jobs stuck in queued → queue and workers. Wrong tenant on a refund → tool service authz. “The AI was weird” is not a box. Point at one.

Note:The tryit uses dicts and a list. That is production-shaped. The brand of broker is a later choice.