Projects/Autonomous Ops Agent/Part 1
Overview and Architecture
Define an ops agent as observe-metrics → diagnose → propose → gated act → report, with never-destructive-by-default as a hard invariant.
Ops is where agent autonomy meets blast radius. A weather agent that hallucinates rain is embarrassing. An ops agent that runs rollback production on a parse error is a headline. This project builds an agent that monitors, diagnoses, proposes, and does not touch destructive tools until a human (simulated) sets HUMAN_APPROVAL.
The product is a loop around a metrics snapshot (dicts), a runbook (dicts of diagnoses → candidate actions), and an action executor with two classes of tools: read/diagnose (always allowed) and mutate (gated). After the incident, it writes a report that a manager could read.
Autonomy slider for ops
| Level | What the agent may do |
|---|---|
| 0 | Dashboard only (this is a workflow) |
| 1 | Diagnose and draft a ticket |
| 2 | Run read-only queries (get_metrics, get_logs) |
| 3 | Propose mutate actions with diffs |
| 4 | Execute approved mutate actions |
| 5 | Execute mutate without approval — out of scope forever on Joeven |
This project implements levels 2–4. Level 5 is a failed eval if it happens.
Architecture boxes
- World — services with metrics: error_rate, p95_ms, cpu, deploy_id, replica_count.
- Observe —
get_metrics(service),get_logs(service, n)simulated. - Diagnose — rules or a fake model mapping symptoms →
{incident, severity, evidence}. - Propose — list of actions with
risk: low|high,destructive: bool. - Gate —
execute(action_id)returnsneeds_approvalunlessHUMAN_APPROVALcontains that id (or a signed bundle). - Report — markdown/JSON timeline after stop.
Destructive vs not
| Action | Destructive? | Default |
|---|---|---|
page_oncall | no (but noisy) | allow or rate-limit |
scale_replicas | maybe | approve if delta large |
restart_service | yes (drops in-flight) | approve |
rollback_deploy | yes | approve |
drop_traffic_pct | yes | approve |
delete_database | yes | not in registry |
If it is not in the registry, it does not exist. Do not add delete_database even as a joke in production catalogs.
Run to execute this in your browser. Nothing is sent to a server.
Stop conditions
- Resolved: after an approved action, metrics in the simulated world improve below SLO (the executor updates the dict).
- Waiting: proposal emitted, no approval yet — stop with status
awaiting_approval(do not busy-loop). - Rejected: human sets approval to
deny. - Budget: max observe/diagnose steps.
Busy-looping execute while waiting for approval is how you spam on-call. Stop and report.
Fake human
HUMAN_APPROVAL is a dict {action_id: "allow"|"deny"}. In a real app this is a Slack button that writes to your API. Here you set it in the Try it box to see both branches.
Tip:Write the invariant as a unit test: every mutate tool checks the gate. If someone adds reboot_all and forgets the decorator, the test fails.Why ops is not ReAct with extra tools
You could dump rollback_deploy into the weather-style JSON loop and let the model call it when it feels like it. That is how people page themselves into a rollback storm. Ops needs a workflow around the model: observe until you have evidence, diagnose into a closed set of hypotheses, propose from a runbook, then stop for a human on anything with blast radius. The LLM may fill a schema. It may not own the execute button.
Think of the agent as an on-call junior who can read dashboards and draft a change ticket, not as the person with production SSH. Junior-plus-ticket is already valuable. Junior-plus-root is a compliance finding.
Two clocks
Incidents have an error-budget clock (SLO burn) and an approval clock (human lag). The agent must not confuse them. Fast diagnosis does not authorize fast mutate. Your traces should show both: time-to-propose and time-to-apply. In this project the second clock is you typing an approval dict.
Exercise
On paper, write the timeline for checkout 12% errors: observe, diagnose "error budget burn, likely bad deploy d44", propose rollback to d43 (destructive), pause for approval, execute, metrics drop to 0.4%, report. That is the golden incident.
Check your understanding