Supervisor Loop Until Tests Pass

The supervisor is a state machine. It is not ReAct. It is the workflow that makes multi-agent safe. This part implements the loop end-to-end with the fake roles and oracle from parts 2–3.

State machine

`` START → PLAN → CODE → TEST → (PASS → STOP_OK) ↘ FAIL → REVIEW → CODE → ... BUDGET → STOP_FAIL

Details:

- After PLAN, always CODE (even if planner no-op).
- After CODE, always TEST (even if patch rejected — still test old repo).
- After FAIL, REVIEW then CODE. Skip review if you want a faster TDD-only team; we include review to practice the slot.
- After PASS, optional REVIEW. If reviewer requests changes on green tests, **ignore** and STOP_OK (supervisor override). Document this; it is a product decision against nit hell.
- `max_rounds` counts **coder turns** (the expensive ones).

## Trace events

`{type, round, role?, patch?, tests?, error?}`. Print them. This is your demo.

import json import re

CASES = [(1, "1"), (3, "Fizz"), (5, "Buzz"), (15, "FizzBuzz")] ALLOW = {"planner": {"PLAN.md"}, "coder": {"src/fizzbuzz.py"}, "reviewer": {"REVIEW.json"}} PATH_OK = re.compile(r"^[a-zA-Z0-9_./-]+$") SAFE = {"range": range, "str": str, "int": int}

def apply_patch(repo, role, path, content): if path not in ALLOW.get(role, set()) or not PATH_OK.match(path) or ".." in path: return {"ok": False, "error": "forbidden", "repo": repo} if not isinstance(content, str) or len(content) > 4000: return {"ok": False, "error": "bad_content", "repo": repo} nxt = dict(repo) nxt[path] = content return {"ok": True, "error": None, "repo": nxt}

def run_tests(repo): ns = {"__builtins__": SAFE} try: exec(repo["src/fizzbuzz.py"], ns, ns) fn = ns["fizzbuzz"] except Exception as exc: return {"ok": False, "errors": [str(exc)], "passed": 0, "failed": len(CASES)} errors = [] passed = 0 for n, want in CASES: try: got = fn(n) except Exception as exc: errors.append(f"{n}: {exc}") continue if got == want: passed += 1 else: errors.append(f"{n}: {got!r}!={want!r}") failed = len(CASES) - passed return {"ok": failed == 0 and not errors, "errors": errors, "passed": passed, "failed": failed}

def planner(repo, trace): if repo["PLAN.md"].strip(): return [] return [{"path": "PLAN.md", "content": "- modulo 15/3/5\n- else str(n)\n"}]

def coder(repo, trace): failed = any(t.get("type") == "tests" and not t["tests"]["ok"] for t in trace) if not failed: src = "def fizzbuzz(n):\n if n % 3 == 0: return 'Fizz'\n return str(n)\n" else: src = ( "def fizzbuzz(n):\n" " if n % 15 == 0: return 'FizzBuzz'\n" " if n % 3 == 0: return 'Fizz'\n" " if n % 5 == 0: return 'Buzz'\n" " return str(n)\n" ) return [{"path": "src/fizzbuzz.py", "content": src}]

def reviewer(repo, trace): last = next(t["tests"] for t in reversed(trace) if t.get("type") == "tests") verdict = {"verdict": "approve" if last["ok"] else "request_changes", "notes": last["errors"][:1]} return [{"path": "REVIEW.json", "content": json.dumps(verdict)}]

def apply_all(repo, role, patches): for p in patches: out = apply_patch(repo, role, p["path"], p["content"]) repo = out["repo"] if not out["ok"]: return repo, out["error"] return repo, None

def run_team(max_rounds=4): repo = { "src/fizzbuzz.py": "def fizzbuzz(n):\n return str(n)\n", "PLAN.md": "", "REVIEW.json": "[]", } trace = [] repo, err = apply_all(repo, "planner", planner(repo, trace)) trace.append({"type": "role", "role": "planner", "error": err}) for rnd in range(1, max_rounds + 1): repo, err = apply_all(repo, "coder", coder(repo, trace)) tests = run_tests(repo) trace.append({"type": "tests", "round": rnd, "tests": tests, "patch_error": err}) print(f"round {rnd} tests ok={tests['ok']} passed={tests['passed']} errors={tests['errors'][:2]}") if tests["ok"]: repo, _ = apply_all(repo, "reviewer", reviewer(repo, trace)) return {"status": "pass", "rounds": rnd, "repo": repo, "trace": trace} repo, _ = apply_all(repo, "reviewer", reviewer(repo, trace)) trace.append({"type": "role", "role": "reviewer", "review": repo["REVIEW.json"]}) return {"status": "budget", "rounds": max_rounds, "repo": repo, "trace": trace}

out = run_team() print("STATUS", out["status"], "ROUNDS", out["rounds"]) print("REVIEW", out["repo"]["REVIEW.json"]) print("SRC\n", out["repo"]["src/fizzbuzz.py"])

## Step-by-step: what to notice when you run it

Round 1: coder's Fizz-only code fails cases 5 and 15. Reviewer requests changes. Round 2: coder sees a failed test in the trace, writes the full function, tests pass, stop. If you set `max_rounds=1`, you get `budget` with failing tests — an eval row.

## Do not let the coder call run_tests internally

If the coder "runs tests" inside `exec` you lose the supervisor's monopoly. Keep `run_tests` out of the allowlisted files. The fake coder only **reads** trace events the supervisor appended.

## Parallelism

Do not parallelize planner and coder. They will write conflicting worlds. Multi-agent ≠ concurrent. Turn-taking is the feature.

> **Note:** When roles become LLMs, wrap each call with JSON parse retries like the weather agent. The supervisor stays sequential.

## Failure injection you should run once

After the happy two-round path works, break things on purpose:

1. Set the coder to always emit the Fizz-only version. Confirm `budget` and a failing oracle.
2. Let the planner try to patch `src/fizzbuzz.py`. Confirm the allowlist error in the trace and that tests still see the stub.
3. Empty `PLAN.md` after planning (supervisor bug). Confirm the coder still can pass if the spec in tests is enough — then decide whether you **require** a plan (product call).

These are not extra features. They are how you know the state machine is real.

## Exercise

Add an eval: `max_rounds=1` expects `status=='budget'` and `not run_tests(repo)['ok']`. `max_rounds=4` expects `pass`. Then forbid a coder patch that contains `exec` or `__import__` as a hardening preview.

When tests are green, why may the supervisor ignore request_changes?

• Reviewers are always wrong
• *To prevent style-nit infinite loops; tests are the stop condition unless you explicitly grade style
• JSON cannot store nits
• The planner already approved

explain: Unbounded review is a budget attack. Green tests stop the team unless style is in the oracle.