Projects/Multi-Agent Software Team/Part 5

Evals and Hardening the Team

Test forbidden paths, budget stops, two-round success, and block dangerous source before exec.

A multi-agent coding team is a privilege amplifier. The coder writes code that exec runs. Hardening is: narrow writes, narrow exec, evals for cheating, and budgets. This part locks those in.

Eval table

id	setup	expect
green-path	default team, max_rounds=4	status pass, fizzbuzz(15)==FizzBuzz
budget	max_rounds=1	status budget
cheat-tests	coder tries to patch tests	apply_patch error forbidden
cheat-planner	planner tries to patch src	forbidden
dangerous	coder content contains `__import__`	reject before exec
idempotent	run team twice	both pass, same src

Dangerous patterns

Before exec, scan source for __import__, import , open(, eval(, exec(. This is a blocklist, not a sandbox. Blocklists fail open against creativity. They still catch the obvious fake-model accidents and some LLM mischief. Combine with tiny builtins.

If rejected, treat like a failed patch: append error, do not exec, tests fail, coder may retry.

Reviewer storm

If request_changes 3 times, next supervisor step is CODE anyway (already true) but do not require reviewer before test. You already test after every code. Good.

Trace size

Do not put full source in every trace event in production. Store hashes (hashlib.sha256) plus last path. In the Try it box, printing src at the end is enough.

Live Pythonpython

import hashlib
import json
import re

DENY_SRC = ("__import__", "import ", "open(", "eval(", "exec(")

def dangerous(src):
    low = src.lower()
    return [p for p in DENY_SRC if p.lower() in low]

def apply_patch(repo, role, path, content, allow):
    if path not in allow.get(role, set()):
        return {"ok": False, "error": "forbidden_path"}
    if role == "coder" and dangerous(content):
        return {"ok": False, "error": "dangerous_src", "hits": dangerous(content)}
    nxt = dict(repo)
    nxt[path] = content
    return {"ok": True, "repo": nxt}

allow = {"coder": {"src/fizzbuzz.py"}, "planner": {"PLAN.md"}}
repo = {"src/fizzbuzz.py": "def fizzbuzz(n):\n    return str(n)\n", "PLAN.md": ""}

def digest(src):
    return hashlib.sha256(src.encode()).hexdigest()[:12]

evals = []

# cheat tests
r = apply_patch(repo, "coder", "tests/test_fizzbuzz.py", "pass", allow)
evals.append(("cheat-tests", r.get("error") == "forbidden_path"))

# planner to src
r = apply_patch(repo, "planner", "src/fizzbuzz.py", "x=1", allow)
evals.append(("cheat-planner", r.get("error") == "forbidden_path"))

# dangerous
r = apply_patch(repo, "coder", "src/fizzbuzz.py", "def fizzbuzz(n):\n    eval('1')\n    return str(n)\n", allow)
evals.append(("dangerous", r.get("error") == "dangerous_src"))

# good hash changes
good = "def fizzbuzz(n):\n    return 'Fizz' if n % 3 == 0 else str(n)\n"
r = apply_patch(repo, "coder", "src/fizzbuzz.py", good, allow)
evals.append(("good-write", r.get("ok") is True))
print("digest", digest(r["repo"]["src/fizzbuzz.py"]))

print(json.dumps([{"id": i, "pass": p} for i, p in evals], indent=2))
print("all", all(p for _, p in evals))

Output

Run to execute this in your browser. Nothing is sent to a server.

Step-by-step hardening checklist

Allowlists per role (already).
Dangerous-source scan before exec.
Tiny builtins.
Tests path not writable.
max_rounds eval.
Supervisor override on green+nits.
Hash traces; do not log secrets (none here).
When swapping in LLMs: JSON schema per role, parse retries, separate rate limits per role.

What you should not claim

This is not a production coding product. exec in the browser, fizzbuzz, and fake models are a coordination laboratory. The transferable skill is: shared state, role allowlists, unforgeable oracle, supervisor workflow. That maps to real PR bots (planner comments, coder patches, CI is the oracle, CODEOWNERS is the reviewer).

Watch out:Never point this pattern at a real filesystem with a live model until you have a VM/sandbox, a diff review gate, and human approval for merges — the next project exists for that instinct.

Mapping to a real GitHub bot

This project	Production
`repo` dict	git worktree or PR branch
`apply_patch` allowlist	CODEOWNERS + path filters
`run_tests`	CI job (pytest, typecheck)
reviewer role	CODEOWNERS human or bot comment
`max_rounds`	job timeout + token budget
dangerous scan	secret scan + AST policy
supervisor	GitHub Action / Temporal workflow

The names change. The invariants do not: unforgeable tests, narrow writes, sequential turns, stop on green or budget. If your swarm library hides the supervisor, you still have to write these checks — just somewhere worse.

Keep a single-process demo in the Try it box and a pytest file on disk with the same assertions. Browser evals catch logic. CI evals catch regressions after you swap fake roles for API calls. If they diverge, the browser version is the lesson and CI is the product.

Exercise

Add max_file_bytes and an eval that a 5000-character source patch is rejected. Then add a trace event {"type": "hash", "path": ..., "sha": ...} after every successful coder write. Confirm the hash changes between round 1 and round 2 of the team loop.

Check your understanding

What is the unforgeable success signal for the software team?

PLAN.md contains the word done
Reviewer thought
Supervisor run_tests returning ok
Number of patches

Supervisor Loop Until Tests Pass

Done

All projects