Evals in CI
CI is the enforcement point: tool unit tests and golden agent fixtures with fake models on every PR. Paid model evals are capped and less frequent.
If evals are a notebook someone runs after a scare, they are folklore. CI is the enforcement point: a prompt or worker change that fails the gate does not merge. This track does not re-teach how to design golden properties. It treats goldens as ops fixtures that a pipeline can fail closed on: forbidden tools, billing strings you already chose, authz denies.
Layers, fast to slow:
- Unit tests for tools and parsers — every PR, seconds
- Golden agent suite with fake models/tools — every PR, a minute or two
- LLM-backed goldens — nightly or on prompt changes, with a spend cap
- Canary in prod — after merge, not instead of CI
PRs that cannot run (1) and (2) without network are too coupled to vendors. Fake the model in CI like Joeven lessons do. Seed fake models. Freeze fixtures. Record prompt version in the test report. If CI is non-deterministic, people will ignore it — correctly.
If you do call a real model, budget the job, cache completions keyed by a hash of prompt + fixtures + model, and do not run it on every typo in the README.
How the box actually works
| Layer | Network | Blocks merge? | Owner |
|---|---|---|---|
| Tool / parser unit tests | No | Yes | Domain + runtime |
| Fake-model goldens | No | Yes | Product + ops |
| Paid LLM goldens | Yes, capped | Nightly; promote only when stable | ML |
| Canary | Prod | After merge | Ops |
No network on the PR job. Empty billing is 0, not 100%.
CI is the enforcement pointA gate function runs items: if a forbidden tool appears, fail; if a billing-tagged item lacks the required fact, fail the rate. Empty billing is 0, not 1 — same trap as eval dashboards. Attach the report to the PR. Reviewers should see which id failed, not a red X with a 4,000-line log of tokens.
Owners: platform owns the CI job. Product owns fixtures. Security owns the forbidden list. Nobody owns “retry until green.”
Quarantine flaky rows with an owner. Do not hide them in a retry loop.
PR jobs have no model network. That is a policy, not a preference. If a developer “just needs one live call,” that belongs on nightly with a cap, a cache key of prompt+fixtures+model, and a budget alarm. README typos must not spend.
The report that attaches to the PR lists failing ids, tags, and the prompt version under test. A 4,000-line token dump is how reviewers skip the gate. Distill a stable nightly into a fake-model script when it has been green for a week — that is how the PR gate grows without getting slower and noisier.
A notebook-folklore ticket
Billing broke for two days after a worker SHA that “only refactored parsers.” The notebook of goldens was last run in March. CI had lint. After the gate: fake-model suite on every PR, c1 forbids wire, c2 requires 5-7. The broken candidate failed both. Merge blocked. The notebook became a nightly extra with a spend cap, not the enforcement point.
Run to execute this in your browser. Nothing is sent to a server.
The good candidate prints pass: True, billing_rate 1.0. The broken one fails on wire and on billing (final is soon, not 5-7). Empty billing would be 0, not 1 — if you skipped billing items, billing_n is 0 and the rate is 0.0, which fails min_billing. That is the same rule as evals, used here as a merge gate, not as a lecture on judges.
What goes wrong
Gate only at night. Gate that needs a paid API on every PR. Flaky live search in the PR suite. Retry until green. Report that is a token dump. Skipping “just this once.” Distilling nothing: nightlies never become deterministic PR tests.
A suite that takes 40 minutes will be skipped in spirit even if it is required in YAML. Keep PR under a couple of minutes. Move the long paid calls to nightly. If people learn the incantation to ignore CI, you do not have a gate. You have a suggestion.
Fixtures that import production keys “to be realistic” will leak and will flake. Fake the model. Freeze the handbook snippet. Record the prompt version.
How to test it
The tryit is the test. Also: CI config has no model network on PR jobs; nightly has a spend cap; artifacts include failing ids. Break candidate_agent on a branch and watch merge stay red.
If a golden is flaky, quarantine with an owner — do not retry until green.
Prove isolation from vendors: turn the network off in the PR job and watch the fake-model suite still pass. If it cannot, you have not faked the model; you have hidden a live call. Cache keys for nightly must include prompt version, fixture hash, and model id so a template edit cannot reuse a stale completion.
How agents use this
Attach the gate report to the PR. Fake models in PR CI. Paid evals capped and rarer. Canary after merge, not instead of CI. When a prompt bundle changes, the report records the version stamp so you can replay.
Promote nightlies that are stable into the PR gate without the live model: distill into a fake-model script or a property check. The PR gate stays deterministic. The nightly hunts surprises.
Record the bundle stamp in the CI report so a red gate names p12 + sha-abc, not “the tests.” When a prompt PR fails c2, the reviewer should see the billing fixture, not a philosophy of evals.
Check your understanding