Incident Report and Hardening

The incident report is the user-visible product of an ops agent, especially when the agent is stuck on approval. Managers read reports. Auditors read reports. Your future self reads reports at 3am. This part writes finish_report, closes the loop, and freezes evals for the invariant.

Report schema

`` { "incident_id": str, "service": str, "severity": str, "hypothesis": str, "evidence": [str], "proposals": [proposal], "approvals": [audit events], "actions_applied": [str], "metrics_before": dict, "metrics_after": dict, "status": "awaiting_approval"|"resolved"|"denied"|"budget", "timeline": [{"t": int, "event": str}] }

Rules:

- `metrics_before` captured at first observe.
- `actions_applied` only from audit status `applied`.
- If status is `awaiting_approval`, `actions_applied` is empty and `metrics_after` may equal before.
- Report tool is **read-ish** — it does not mutate WORLD. It may append to `REPORTS` list.

## Full loop (fake policy)

1. get_metrics + get_logs
2. diagnose/propose in code
3. execute
4. if needs_approval → report awaiting
5. if applied → get_metrics again → report resolved if not burning

## Evals (must all pass)

1. Without approval, rollback does not change `deploy_id`.
2. With matching digest, rollback changes `deploy_id` to prev.
3. Tampered args do not apply.
4. Unknown mutate tool does not apply.
5. Report `actions_applied` empty on awaiting path.
6. Healthy search service produces no rollback proposal.

import hashlib import json

def digest(p): return hashlib.sha256( json.dumps({"id": p["id"], "tool": p["tool"], "args": p["args"]}, sort_keys=True).encode() ).hexdigest()

def run_incident(approve=False, tamper=False): world = { "checkout": { "error_rate": 0.12, "p95_ms": 1800, "deploy_id": "d44", "prev_deploy_id": "d43", "slo_error": 0.01, "logs": ["deploy d44", "last_good=d43"], } } before = { "error_rate": world["checkout"]["error_rate"], "deploy_id": world["checkout"]["deploy_id"], } timeline = [{"t": 1, "event": "observe"}] proposal = { "id": "act-01", "tool": "rollback_deploy", "args": {"service": "checkout", "to": "d43"}, } timeline.append({"t": 2, "event": "propose rollback"}) approval = {} if approve: approval = {"act-01": {"decision": "allow", "digest": digest(proposal)}} to_run = dict(proposal) if tamper: to_run = dict(proposal) to_run["args"] = {"service": "checkout", "to": "d0"} d_ok = digest(to_run) == approval.get("act-01", {}).get("digest") if approve else False applied = [] status = "awaiting_approval" if approve and not tamper and d_ok: world["checkout"]["deploy_id"] = "d43" world["checkout"]["error_rate"] = 0.004 applied = ["act-01"] status = "resolved" timeline.append({"t": 3, "event": "applied rollback"}) elif approve and tamper: status = "denied" timeline.append({"t": 3, "event": "tamper blocked"}) else: timeline.append({"t": 3, "event": "waiting on HUMAN_APPROVAL"}) after = { "error_rate": world["checkout"]["error_rate"], "deploy_id": world["checkout"]["deploy_id"], } report = { "incident_id": "inc-17", "service": "checkout", "severity": "sev1", "hypothesis": "bad_deploy", "evidence": world["checkout"]["logs"], "proposals": [proposal], "actions_applied": applied, "metrics_before": before, "metrics_after": after, "status": status, "timeline": timeline, } return report, world

r1, w1 = run_incident(False) r2, w2 = run_incident(True) r3, w3 = run_incident(True, tamper=True)

def check(): rows = [] rows.append(("no-approve-no-mutate", r1["status"] == "awaiting_approval" and w1["checkout"]["deploy_id"] == "d44" and not r1["actions_applied"])) rows.append(("approve-resolves", r2["status"] == "resolved" and w2["checkout"]["deploy_id"] == "d43" and r2["actions_applied"] == ["act-01"])) rows.append(("tamper-blocked", r3["status"] == "denied" and w3["checkout"]["deploy_id"] == "d44")) return rows

print(json.dumps(r1, indent=2)[:600], "...") print("--- evals ---") failed = 0 for name, ok in check(): print(name, ok) failed += int(not ok) print("failed", failed)

## Step-by-step hardening checklist

1. Mutate allowlist; no shell; no delete_database.
2. Digest-based HUMAN_APPROVAL; model cannot write the map.
3. Audit log append-only.
4. Stop on needs_approval (no execute spin).
5. Report always, even on budget.
6. Redact secrets in logs and reports.
7. Rate-limit pages.
8. Evals above in CI.

## Portfolio line

You built an ops agent that is **autonomous up to the blast radius**, not past it. That is the adult form of "agents in production": loops, tools, traces, evals, and a human gate with cryptographic-ish binding of what was approved.

Joeven's project track ends here on purpose. The next systems you build will combine these five: tools+JSON (weather), ReAct+citations (research), RAG refusal (support), multi-agent oracles (dev team), and approval gates (ops). The loop did not change. The **invariants** did.

> **Warning:** A demo that auto-rollbacks because it looks cool is not a feature. It is an untested mutate path. If you remove the gate to impress a demo, you fail this course.

## What to put in the timeline vs the novel

A good report is a **table of events**, not a blog post. Each line: step number, tool or decision, result code, metric snapshot if it changed. The hypothesis is one sentence. The proposal is JSON the human already saw. If you let a model write a three-paragraph "narrative of the outage," it will invent causes that were not in evidence. Generate prose **from the timeline** with a template: `{service} {severity}: {hypothesis}. Proposed {tool} {args}. Status {status}.`

Keep the raw timeline in the JSON even if you also render markdown. Machines grade JSON. Slack can have the template string.

## Aftercare

Once `resolved`, freeze WORLD (or snapshot it) so a second loop does not rollback again. A `cooldown` flag per service is enough: if error_rate is below SLO, proposals must be empty. Flapping rollbacks are worse than a slow human.

## Exercise

Add `scale_replicas` to the eval table: unapproved scale leaves replica count 3; approved scale to 4 applies; a proposal for 99 replicas is rejected by an args validator (`1 <= replicas <= 10`) **even if approved**. Humans can be wrong; bounds still apply.

Which status should the agent return when rollback is proposed but HUMAN_APPROVAL is empty?

• resolved
• *awaiting_approval, with an incident report and no world mutation
• applied (optimistic)
• delete the proposal

explain: Waiting is a terminal state for this run. Mutating while waiting violates the course invariant.