Production Agents

Production Architecture

Name the boxes you operate: gateway, queue, workers, tool services, stores, and a control plane. The model is one box with an owner, a timeout, and a budget.

Jobs, Not Requests

HTTP returns job_id. The client polls or streams events. Job state lives in a store so deploys, tab closes, and 15-minute timeouts cannot erase the loop.

Isolation and Confused Deputy

Workers do not hold admin keys. Tool services enforce the signed job tenant. The model is an untrusted client of your APIs, like a browser.

Version Everything the Policy Needs

Stamp prompt, tool schema, model id, flags, and worker git SHA on every job. “What was in prod on Tuesday?” is an incident question, not a guess.

The Smallest Prod-Shaped Stack

Day one is not Kubernetes. Day one is a queue, a job table, a trace table, and a practiced way to stop the agent — even if those are a list and two dicts.

Tracing and Observability

Traces first: spans for the job, each model call, and each tool. Operators need a timeline, parsed tool calls, and dollars — not a pretty token stream.

Redact Before You Export

Traces are a new PII store. Redact secrets, cookies, and card numbers before any exporter, before the scratchpad, and in the operator UI.

Metrics That Page a Human

Page on blast radius: forbidden tools, cost per minute, queue depth, HITL past SLA. Put noisy model timeouts on a dashboard with a circuit breaker.

Cost and Latency

Agent cost is steps times growing input tokens. Cap it, trim the scratchpad, route easy jobs small, and write SLOs that admit the loop is slower than a FAQ.

Budgets Are Stop Conditions

max_steps, max_usd, max_wall_clock, max tokens in, plus a per-tenant monthly cap. When a cap hits, stop with a structured error — never a silent model downgrade.

Queues and Retries

Workers crash and vendors 503. Assume at-least-once delivery, put idempotency keys on writes, back off with jitter, and checkpoint after the tool succeeds.

Idempotency Keys

Build the key from job_id, step_id, and tool. Same key, same result, no second side effect. Do not put the model’s thought text in the key.

Dead Letters, Poison, Fairness

Max attempts then DLQ. Reject poison at the gateway. Per-tenant concurrency so one swarm cannot starve everyone. Slow enqueue when the vendor is 503.

Secrets and Security

Secrets live in a manager or the tool service, never in prompts, repos, or traces. Treat the model as an untrusted client; redact before the scratchpad; pin dependencies.

Tenant From the Gateway

Tenancy is a filter on every read and write, using the signed job tenant. Include it in CI. A shared index without tenant filters is a breach waiting for a query.

Deploy

Shipping an agent ships code plus policy. Version workers, canary a slice, health-check without spending, and treat a prompt edit as a rollbackable release.

Canaries and Feature Flags

Send 5% of jobs or a volunteer tenant to the new worker. Watch goldens, forbidden tools, cost, HITL, latency — then promote or roll back. Flags are versioned config.

Kill Switches You Have Practiced

Global agents.disabled, per-tool flags, per-tenant pause. Practice flipping them on a game day. A switch nobody has ever flipped is decorative.

Evals in CI

CI is the enforcement point: tool unit tests and golden agent fixtures with fake models on every PR. Paid model evals are capped and less frequent.

What Blocks Merge

Block on forbidden tools, authz regressions, parser failures, and a drop on a critical tag. Do not block on flaky live search or uncalibrated judge scores. Do not skip the gate.

Incident Playbooks

Write the playbook before the afternoon: contain first (kill switch, pause queue, trace ids), then see, then rollback. Agents fail as distributed systems.

Contain, Then See

After the flag flips: traces for the blast radius, classify policy vs tool vs vendor vs injection vs HITL bypass, then rollback and rotate if keys leaked. Do not delete traces.

Every Incident Ends in a Golden

If you cannot write a test that would have failed before the deploy, you do not understand the incident yet. Then game-day: old SHA fails, new SHA passes.

Boring Ops Is the Goal

Production is a queue, traces, caps, tenancy, CI that can say no, and a practiced kill switch. A public URL is just DNS. Stop the loop, see the trace, prevent the encore.