The Output Contract
Decide the shape of a valid reply before you write adjectives. The parser is the API.
The output contract is what your code will accept after the model speaks. It is not a style guide. It is not “be concise.” It is the API between a next-token machine and the rest of your program.
If you do not decide the shape, the model will pick a new one every Tuesday: a paragraph, then a list, then JSON with a joke key, then a markdown table. Your parser will grow ifs until it is a second model. That is how agents quietly stop being software.
Decide the contract before you write adjectives. The grader is the product. The prompt is how you ask the model to hit the grader.
Pretty prose that misses the shape still fails. Fail closed, then retry.
The parser is the APIShapes that actually parse
Common contracts, from loosest to tightest:
| Shape | When it is enough | How you grade it |
|---|---|---|
| Free text for a human | Chat UI, no downstream code | Length cap, banned substrings, maybe a second check |
| “answer, then Sources:” | Support bot that must cite | Split on a sentinel; fail if no sources line |
Tagged block (<final>, FINAL:) | Agent loop with scratch above | Extract between sentinels |
| JSON object with required keys | Anything a program will consume | json.loads, key check, enums |
The LLM track taught json.loads and retries. Here the job is asking for less, in words a grader can check. A contract that demands twelve optional fields will be filled with twelve hallucinations. A contract that demands two required fields will fail closed when the model wanders.
JSON mode (vendor constrained decoding) is machinery that makes JSON more likely. It is not the contract. The contract is still your keys, enums, and “no extra fields.” Models in JSON mode still invent keys. Reject them.
Make the contract small on purpose
A good contract is small:
- Enums beat free text.
status=ok|need_clarification|refusedis a finite set. “a status reflecting your vibe” is infinite. - Required keys beat optional novels. If
debugis optional, it will someday contain a secret. - One final object beats “thought then maybe JSON.” Scratch can exist (later lessons). The product is still one object.
- Empty is illegal.
answer: ""is not a reply. Fail it.
Extra keys are a side channel. An attacker who cannot change answer can still stick debug: "send to attacker" if you accept unknown fields. Strip them in code if you must parse a vendor blob, but prefer reject and retry so the model learns the contract. Logging extra keys is fine. Forwarding them to the user or to another tool is not.
Write illegal replies first
If you cannot name three illegal replies, you do not have a contract. You have a preference. Put those three in the eval:
- Prose with no JSON
- JSON with extra keys
- JSON with empty
answeror a status not in the enum
If they pass, you do not have a contract. You have a demo.
A fourth illegal reply shows up in agents: a well-formed object that claims a tool you did not enable. That is still a contract fail even if JSON is pretty. The parser should only accept tool names from the allowlist. This lesson’s toy parser checks status and answer. The agent format lesson will check tool.
Run to execute this in your browser. Nothing is sent to a server.
What printed: the first sample passes. Extra debug fails. The prose line is not json. Empty answer fails. maybe is bad status. That table is the contract. Adjectives would not have produced it.
Put the contract in two places, then repeat it
Put the contract in the spec and in the validator. If they drift, the validator wins — and you have a bug in the spec. Repeat the two lines that must survive (schema + forbidden actions) after untrusted blocks, because models overweight the end of the prompt. Recency is the next lesson. The contract is why recency matters.
Do not ask for “JSON or a short paragraph.” That is two contracts. The model will pick the easier one on the hard ticket.
Walkthrough: the Tuesday shape
Monday the billing bot returns {"status": "ok", "answer": "INV-17 is open."}. Tuesday a “friendlier” spec says “you may add a short note for the user.” Wednesday the object has note, debug, and confidence. Your parser takes answer and ignores the rest. Thursday debug contains a tool error with a key. You forwarded the object to a client “for transparency.” That is a leak with extra keys.
The contract that would have stopped this is the one you already wrote: two keys, enum status, extra keys fail, empty answer fails. Friendliness belongs in the string inside answer, not in new fields. If you need a user-facing note, name it, require it, grade it. Optional novels are how APIs rot.
What goes wrong if you skip this
Every downstream if is a second parser. Retries become “try to find a brace.” Evals cannot name a fail (test_extra_keys does not exist). Agents will emit tool names in prose because prose was allowed. You will then claim JSON mode will save you. JSON mode still invents keys. The contract is the keys you reject.
How agents use this
The parser is the API. Fail closed. Retry once with “return only the object, no prose.” Then hand off. Do not write a second parser for “almost JSON.” Optional fields become hallucinations. The tools track will add JSON Schema for arguments — same habit, different layer. This layer is the reply.
Tip:Write three illegal replies in your eval: prose, extra keys, empty answer. If they pass, you do not have a contract.
Check your understanding