Projects/Multi-Agent Software Team/Part 2

Shared Repo and Test Oracle

Implement the repo dict, path allowlists, full-file writes, and a fake test runner that execs fizzbuzz in an isolated namespace.

The shared repo is a versioned dict. Every write produces a new snapshot you can put in the trace (or a diff of paths). This part builds apply_patch, run_tests, and the read-only test spec. No git. Git is a later adapter.

Patch API

apply_patch(repo, role, path, content) -> {ok, error, repo}

path in ALLOW[role]
content is str, len ≤ 4000
path matches ^[a-zA-Z0-9_./-]+$ and does not contain ..
Copy repo (dict(repo) is shallow; values are strings so OK) and set the path
Return error dicts, do not raise, so the supervisor can tell the agent "patch rejected"

Test oracle

Read src/fizzbuzz.py, exec into ns = {}, get ns["fizzbuzz"], run cases. Catch SyntaxError, KeyError, TypeError. Result:

{"passed": int, "failed": int, "errors": [str], "ok": bool}

ok is failed == 0 and errors empty and passed == len(cases).

Do not exec the tests file. Agents might try to rewrite it. The oracle is code you wrote in the supervisor module.

Isolated exec

Use a fresh dict. Do not pass __builtins__ unrestricted if you can avoid it — in CPython you can set {"__builtins__": {"range": range, "str": str, ...}}. In Pyodide this still is not a security boundary against a determined model, but it documents intent. Part 5: never exec model code with real builtins on a machine with secrets.

Live Pythonpython

import json
import re

CASES = [(1, "1"), (3, "Fizz"), (5, "Buzz"), (15, "FizzBuzz"), (9, "Fizz"), (10, "Buzz")]
ALLOW = {
    "planner": {"PLAN.md"},
    "coder": {"src/fizzbuzz.py"},
    "reviewer": {"REVIEW.json"},
}
PATH_OK = re.compile(r"^[a-zA-Z0-9_./-]+$")

def new_repo():
    return {
        "src/fizzbuzz.py": "def fizzbuzz(n):\n    return str(n)\n",
        "tests/test_fizzbuzz.py": "spec: 3 Fizz, 5 Buzz, 15 FizzBuzz else str(n)",
        "PLAN.md": "",
        "REVIEW.json": "[]",
    }

def apply_patch(repo, role, path, content):
    if role not in ALLOW:
        return {"ok": False, "error": "bad_role", "repo": repo}
    if path not in ALLOW[role]:
        return {"ok": False, "error": "forbidden_path", "path": path, "repo": repo}
    if not isinstance(content, str) or len(content) > 4000:
        return {"ok": False, "error": "bad_content", "repo": repo}
    if not PATH_OK.match(path) or ".." in path:
        return {"ok": False, "error": "bad_path", "repo": repo}
    nxt = dict(repo)
    nxt[path] = content
    return {"ok": True, "error": None, "repo": nxt}

SAFE_BUILTINS = {"range": range, "str": str, "int": int, "len": len}

def run_tests(repo):
    src = repo.get("src/fizzbuzz.py", "")
    ns = {"__builtins__": SAFE_BUILTINS}
    try:
        exec(src, ns, ns)
    except Exception as exc:
        return {"ok": False, "passed": 0, "failed": 0, "errors": [f"exec: {exc}"]}
    fn = ns.get("fizzbuzz")
    if not callable(fn):
        return {"ok": False, "passed": 0, "failed": 0, "errors": ["missing fizzbuzz"]}
    passed = failed = 0
    errors = []
    for n, want in CASES:
        try:
            got = fn(n)
        except Exception as exc:
            failed += 1
            errors.append(f"case {n}: {exc}")
            continue
        if got == want:
            passed += 1
        else:
            failed += 1
            errors.append(f"case {n}: got {got!r} want {want!r}")
    return {"ok": failed == 0 and not errors and passed == len(CASES), "passed": passed, "failed": failed, "errors": errors}

repo = new_repo()
print("stub", run_tests(repo))
denied = apply_patch(repo, "coder", "tests/test_fizzbuzz.py", "pass")
print("deny tests", denied["error"])
good_src = (
    "def fizzbuzz(n):\n"
    "    if n % 15 == 0: return 'FizzBuzz'\n"
    "    if n % 3 == 0: return 'Fizz'\n"
    "    if n % 5 == 0: return 'Buzz'\n"
    "    return str(n)\n"
)
patched = apply_patch(repo, "coder", "src/fizzbuzz.py", good_src)
print("patch", patched["ok"])
print("green", run_tests(patched["repo"]))
print(json.dumps({"allow_coder": sorted(ALLOW["coder"])}))

Output

Run to execute this in your browser. Nothing is sent to a server.

Step-by-step environment rules

One repo object passed into every role. Roles return patches, they do not close over a global if you can help it (a global is OK in Try it; prefer arguments).
Tests are code in the supervisor, spec text in the repo for the planner to read.
Forbidden writes return an observation the agent can see: forbidden_path. A fake coder that tries to cheat should fail the eval.
Deterministic fizzbuzz. No random. Same repo always same test result.

Why full-file writes

Unified diffs are painful to parse when the model miscounts lines. Full-file replacement needs a size cap. For a one-function repo it is the right trade. If you add more files later, the coder patches one path per turn (supervisor rule) to avoid nuking PLAN.md.

Copying, identity, and traces

dict(repo) is a shallow copy. That is correct because values are strings. If you later store nested dicts (a parsed AST cache), a shallow copy will leak mutations across "snapshots." Prefer copying the values you mutate. After each successful patch, append {path, n_bytes, sha256[:12]} to the trace — not the full file. Full files belong in the repo; traces belong in logs.

When a patch is rejected, do not keep a half-applied repo. apply_all should apply sequentially and stop on first error, returning the last good snapshot. Partial applies are how PLAN.md updates while src stays stale and you debug ghosts.

Exercise

Add a second allowed file src/util.py that the coder does not need. Write a test that a planner patch to src/fizzbuzz.py is forbidden. Print both errors.

Check your understanding

Why is the test file not the oracle?

Tests cannot live in dicts
Agents might edit it; the supervisor's run_tests is the unforgeable grader
exec is required by ReAct
Fizzbuzz cannot be specified in Python

Overview and Architecture

Up next

Planner, Coder, and Reviewer Policies