Curriculum/Multi-Agent Systems
Keep a Single-Agent Baseline
The split has to beat one loop on evals. If it does not, delete a persona. Multi-agent is not an identity. Score quality, cost, latency, and incidents.
Before you ship a team, freeze a baseline: one agent, the same tools that are safe together, the same golden tickets. A baseline is not “the intern we used in January.” It is a runnable policy you can still run in CI next to the team. If you cannot run it, you cannot beat it. You can only tell stories.
Score both the baseline and the team on the same fixtures:
- Task success — the job check: tests green, citation present, refund not double-paid
- Cost — tokens, steps, child jobs, dollars
- Latency — wall time to done or to legal stop
- Incidents — double refund, ping-pong, forbidden tool, leaked context
If quality is the same and cost doubles, keep the one loop unless safety won.
Beat the single-agent baselineIf the team wins on vibes and loses on the numbers, keep the baseline. Multi-agent is an optimization. Optimizations that lose on the scoreboard are regressions. Deleting a persona is a valid ship.
This lesson is the last brake in the “why” part. You now know honest splits, merge rules, the coordination tax, and typed interfaces. None of those entitle you to keep a team that is worse.
What “beat” means
A team beats the baseline when it wins on a named axis without a silent loss on another that you care about.
| Team result vs baseline | Verdict | Note |
|---|---|---|
| Better quality | Win | Even if a bit slower, if latency still under SLO |
| Same quality, cheaper or equal cost | Win | Isolation might still be the real win — record it |
| Same quality, more than 2× cost | Loss unless a named safety or incident win | |
| Worse quality | Loss | Pretty traces do not count |
| Same quality, more incidents | Loss | Cost and pass rate can hide a double refund |
The 2× cost line in the toy below is a policy, not physics. Your company might allow 3× for a hard isolation win (refund tools off the coder). Write the policy down. “We like teams” is not a policy.
Safety can be the win: the team never lets the planner patch src, and the baseline cannot separate those tools without becoming two agents anyway. Then say “incident class planner-writes-src is now zero” and keep the team even if dollars ticked up. Do not hide a 5× bill inside that sentence if incidents did not change.
Latency can be the win for a swarm of cheap scorers versus one fat sequential pass. Latency can also be the loss: two serial LLM calls for a ticket one call used to close. Measure.
Walkthrough: find why job 17 failed and open a PR
A single agent can search logs and edit files if both tools are safe together and the context fits. That is the baseline. Put it on the golden: same job 17 fixture, same tests.
Split when log search returns megabytes you must compress, or when opening a PR needs a different approval path than reading logs. Then the team must beat the baseline on the scoreboard, not on a whiteboard drawing of four boxes.
Three runs:
Baseline. Pass. $0.04. No incidents.
Fancy team. Pass. $0.20. Same quality. Five times the money. No new safety property. Loss.
Good team. Pass. $0.03. Cheaper. Win.
A fourth run you will see in reviews: team fails, baseline passes, but the team “found a nicer architecture.” Ship the baseline. Fix the team offline until it wins, or merge.
Run to execute this in your browser. Nothing is sent to a server.
What printed: fancy versus base is False and more than 2x cost, same quality. Good versus base is True and cheaper or equal. score(base) reports the baseline is ok at 0.04. Same-quality at 5× cost is a loss. Cheaper and still green is a win.
The toy does not yet encode incidents or latency. Add them as extra fields in your real beats: if team["incidents"] > base["incidents"], return false. If you only compare dollars, a team that double-refunds cheaper will “win.”
Keep the baseline alive
Baselines rot. Someone deletes the one-agent config because “we are a multi-agent shop now.” Then you cannot answer “did the team get worse?” except by arguing. Pin the baseline next to the team in CI. When the team loses for a week, the build should hurt.
Personas accrete. A director appears after an incident. An intern-2 appears after a demo. Check monthly: run same_contract, run beats, delete a name. A merge that restores the baseline cost with the same pass rate is a ship, not a demotion.
Do not A/B vibe on live customers as your only comparison. Use goldens first (evals track). Online incidents still matter: if the team’s ping-pong rate is new, that is a loss even when goldens are green — your goldens missed a bounce. Add the bounce as a fixture. Do not add a fourth persona to “manage” it.
How agents use this
Check this monthly. Personas accrete. Deleting one is a valid ship. Publish a tiny table: baseline pass, team pass, cost ratio, incident count, hop count. If product wants the team for marketing, they can have a diagram. Production runs the winner.
When you later add debate or a swarm, the baseline is still the one-agent policy, not last week’s four-agent mesh. Each extra pattern must beat the cheapest machine that already works. The last lesson of this track will say the same in a ladder: workflow, one agent, sequential roles, supervisor, swarm, debate.
Wire beats to the job record: estimated cost before launch, actual cost after, pass boolean, incident tags. The failure-modes lesson will refuse a swarm of 50 when the price exceeds the cap. That is this lesson’s cost line with a hard stop instead of a review comment.
If the team matches the single agent on quality and costs 5×, keep the baseline unless you can prove a different win you are willing to pay for: isolation that removes a class of incidents, latency that hits an SLO, a legal approval path you cannot fold into one allow-list. Write that win. Then keep measuring, because wins decay when prompts drift.
Check your understanding