Write the Eval First

Joeven’s rule for this track: write the eval before you grow the prompt. The scarce resource is not tokens. It is labels: the correct tool, the correct refusal, the correct citation id, the forbidden names, the fact that must appear. Ten well-labeled cases with hidden checks beat a thousand unlabeled chats that nobody can score twice.

An unlabeled transcript is a ticket, not an eval. You can read it and form an opinion. You cannot rerun a property. You cannot fail a PR. You cannot tell whether a new model is better or just different. Piling transcripts into a folder named “evals” is how teams fake maturity.

Growing the prompt first feels faster. You add a sentence. The demo works. Then you add another sentence for the incident. Then the first behavior dies and there is no case to resurrect it. The eval-first order is: specify the behavior as a row, watch it fail, then change prompt or code until it passes without killing the other rows.

What a label actually contains

You do not need every field on every row. You need the fields that make this behavior fail closed. A billing FAQ without must_call is how invented policy ships. A wire probe without forbid is how a careful paragraph plus a tool call ships.

Hidden tests matter. If the expected answer is pasted into the prompt as an example, you are measuring memorization of the suite. Hold out cases the prompt authors do not get to see. The next part of this track (goldens and properties) is that discipline in table form. This lesson is the habit: labels before adjectives.

Walkthrough: thin vs full on the same question

User input: “How long do refunds take?”

Thin. id: g1, input only. missing_labels reports must_call. You cannot fail a hallucinated FAQ. You also cannot forbid refund. Anyone can claim the ticket is “covered.”

Full. Same id and input, plus must_call: search_kb, must_contain: 5-7, forbid: ['refund']. Now the invented pretty answer fails. The sneak refund fails. The grounded answer passes. The row is a spec.

A third row should exist before you “just add refunds as a tool”: must_refuse or HITL, forbid: ['wire'], tag safety. If you add the tool first and the row later, production is the test suite until someone gets to it.

Quality beats volume. Fifty paraphrases of “how long do refunds take?” without a single injection document is not coverage. Two paraphrases, one stale-policy case, one injected PDF, one cross-tenant probe, one “must still help” FAQ — that is a start. Labeling is work. Do it on purpose. Double-label a slice if humans disagree; disagreement means the spec is vague, not that you need a bigger model.

Hidden tests are part of writing first. The prompt author should not see every must_contain and every injected PDF. If they do, the “improvement” will be pasting the answers into the system prompt, and the suite will measure memorization. Holdout rows live in the same schema with a flag. Product PRs that only touch the visible slice should still run holdout. When holdout drops and the visible slice does not, you learned something true.

A label is also a weight. The wire probe is not one vote among fifty FAQs. Write that down in the row or in the runner: forbid hits fail the suite. Eval-first means you decide that before someone argues that 98% is “basically fine.” The golden-set lessons will store this as tags and gates. The habit starts when the first row is written, not when the dashboard exists.

Run to execute this in your browser. Nothing is sent to a server.

What printed: thin ['must_call']. full []. The last print line is the moral. This checker is intentionally small — it does not even require forbid. Your real schema should. The toy shows the shape: missing keys are missing spec.

What goes wrong if you skip this

You will grow the prompt until it is a novella of special cases. Incidents get a new paragraph. Unlabeled behaviors die in the shuffle. Leadership will demand features that break the unlabeled 12%, and you will not be able to show which 12%. You will then try to “add evals later” by dumping production logs into a spreadsheet. Logs without labels are more tickets.

Skip eval-first and golden sets become a graveyard of screenshots. Skip it and judges get asked to invent the spec at grade time. Skip it and safety probes never get must_refuse. The rest of this track assumes the row exists before the cleverness.

How agents use this

Publish a weekly slice: success by tag, forbidden-tool count, cost per passing case. If leadership only sees a demo GIF, they will demand features that break the unlabeled remainder. Make the suite visible the way you make uptime visible.

In product code, a case is a dict (or a JSON line) checked into the repo, reviewed like any other contract. Prompt PRs that do not touch goldens should make you suspicious. New tools ship with rows: one happy path, one forbid, one authz miss. Multi-agent roles each get rows, or the supervisor will be evaluated as if it did the work.

You now know why evals exist, why side effects are the score, why offline and online both count, why the unit is a trace, and why labels come first. Next: golden sets — versioning those rows, pass rate as a fraction, properties instead of essay equality, quarantine, and coverage by tag.