Offline vs Online

Offline evals run on a dataset you own: frozen tickets, fake tools, recorded pages, expected properties. They belong in the suite you rerun on every prompt and model change. No live users. No live Stripe. If you cannot rerun it, it is not an offline eval — it is a memory of a demo.

Online evals sample live traces: did a human override, did the user reopen the ticket, did the payment bounce, did a reviewer deny a HITL refund, did cost explode on one tenant. The world is not frozen. Labels are delayed and noisy. You still need them, because fixtures lie as soon as finance changes the window and nobody updates the JSON.

The point of this lesson is not “pick one.” The point is that each one covers the other’s blind spot, and the weekly review should put both numbers on the same page.

What each one can and cannot see

Offline without online is overfitting a museum of tickets. Your pass rate is 94% on last quarter’s goldens. Production users now paste screenshots of a new refund portal. The suite never saw it. The agent invents a policy. You learn from a viral complaint.

Online without offline is flying by dashboard anecdotes. Reopen rate ticked up. Nobody can replay. Nobody knows whether the model skipped search_kb or the KB was empty. You will “fix” it by lengthening the prompt and wait to see if the line goes down. That is not measurement. That is weather.

Fixtures are the offline contract

A fixture is a fake world the tools see: a tiny invoice table, a tiny KB document that says 5-7 days, a PDF that contains an injection, a user id that is not Ada’s. The agent under test gets the same tool names as production and a smaller, known implementation. Live Stripe in the suite is how CI becomes flaky and how you accidentally refund a real customer from a unit test. Keep money moving in fakes until a dedicated production track says otherwise.

Online sampling is not “run the golden set against production.” It is events: override, reopen, bounce, HITL deny, safety tag, write-tool used. You join those to trace ids. You do not need a full property check on 100% of traffic on day one. You need enough to disagree with the museum.

Walkthrough: 2/3 offline, reopen plus override online

Acme’s offline slice this week is three goldens: two FAQ passes, one injection fail that the current prompt still misses. Offline pass rate is 2/3.

The online slice of live billing tickets shows: one reopen (user came back “you said 5-7 but the portal says 10”), one human override (agent tried to refund, reviewer stopped it), no payment bounce.

If you only publish 0.667, someone will say “pretty good” and ship a prompt tweak. If you only publish “we had an override,” someone will say “rare.” Together: the suite already knows injection is weak, and production is telling you the fixture fact 5-7 may be stale. The move is not “add adjectives.” The move is: update the golden or the product in a reviewed commit, and keep both numbers.

Run to execute this in your browser. Nothing is sent to a server.

What printed: offline is about 0.667 (two of three). online is ['override', 'reopened']. Neither number replaces the other. Empty gold would have printed 0.0 — the same empty-suite rule you will see again on pass rate. A live list with no flags would print [], which is information only if you actually sampled.

What goes wrong if you skip this

Skip offline: every change is validated by whoever is on Slack. Injection fixtures never exist. You cannot bisect a model upgrade. Skip online: you worship a JSON file while customers reopen tickets because the real policy moved. Skip both: you are not doing evals; you are doing vibes with extra steps.

A common fake compromise is “we look at production once a month.” A month of unmeasured refunds is not a slice. Name a cadence: offline on every relevant change, online flags on a weekly review, and a rule for disagreement.

Disagreement is the interesting case. Offline says the FAQ still has 5-7; online reopens say customers were told 10 days by the portal. That is not “the model is random.” That is a stale fixture or a stale product. The rule is: one reviewed commit that updates the KB fixture and the expected property, or a product fix that makes the portal match the KB. Quietly adding “sometimes 10 days” to the prompt so both numbers feel true is how you get a third number in the wild.

Online flags need names you can count without a novel: override, reopened, bounce, hitl_deny, hitl_timeout, write_tool. A free-text “looked weird” column will not aggregate. Sample enough traffic that a week without flags is surprising, not because you forgot to look. Offline still carries injection PDFs and refusals that live traffic may not volunteer this week.

How agents use this

Name an owner for each suite. If production disagrees with the golden, you change the product or the golden in a reviewed commit — never in a quiet prompt tweak. The production track will teach CI, queues, and kill switches that consume these numbers. This lesson only insists that you have both kinds, that offline is fixtures, and that online is live events with names you can count.

When you add a new billing worker or a new retrieval corpus, add offline cases first, then watch online reopens for that tag. Multi-agent ping-pong that looks fine on a frozen script can still produce overrides in production; the online flag is how you notice.