Structured Output

Agents need actions, not vibes. The reliable way is structured output: the model must emit data that your code can parse and check against a schema. A schema is a list of fields, types, and allowed values. Free prose is for the user. Tool choice, arguments, and “final vs continue” are for parsers.

If you parse English with a pile of regexes, you will eventually execute a sentence you did not intend. If you eval the string as Python because it “looks like code,” you handed the model a shell. This site’s live boxes, and your production executor, should never exec model text.

The loop

1. Ask for JSON (prompt + vendor response_format / tool-calling if available).
2. Parse (json.loads, after stripping a markdown fence if you must).
3. Validate types, enums, ranges, extra keys, missing keys.
4. On failure: retry once with the validator’s error, or hand off.
5. Never execute a string the model labeled python.

Vendor-native tool calling is structured output with extra ceremony: the model emits a function name and argument object the API already parsed. Prefer that when you can. Still validate. Vendors are not your type checker. job_id: -1 can be perfect JSON.

Constrained decoding (JSON schema, vendor response_format) makes invalid tokens unlikely. Use it. It does not validate business rules. Your validator still runs.

Ask for less. Two fields parse more often than twelve optional ones. Optional fields become hallucinations: the model fills them because the schema made them legal, not because the user provided them.

Repair vs refuse

Models wrap JSON in markdown fences, add trailing commas, or use single quotes. You can strip fences. You should not write a heroic repair parser that guesses keys. Repair is how you execute a poisoned id. Two retries max, then a deterministic fallback (handoff). Send the schema error, not “be better.”

Typed fields: ids are ints or uuid strings, not “the latest one.” Enums beat free text for actions. If the action is not in {"get_job", "finish", "abstain"}, it is not an action.

Run to execute this in your browser. Nothing is sent to a server.

Attempt 1 is prose; you get a json error. Attempt 2 is a dict with get_job and 17. result is that dict. If both failed, you would get handoff: True instead of a guessed tool. That fallback is the product. Guessing get_job from the English of attempt 1 would have worked this time and failed on “Sure, let's delete_job(17).”

Log parse failures as their own metric. If 8% of calls fail JSON, you have a prompt, a model-size, or a temperature problem — the user did not type the JSON, the policy did.

Native tools vs JSON in prose

If the vendor returns tool_calls, do not also regex the assistant prose for JSON. Two channels will disagree. Prefer native. If you only have prose, require one object, no chatter, temperature 0 (next lesson). Strip a single markdown fence around the object if present; if you see two objects, fail.

Fences: models like to wrap JSON in a markdown code block. Strip a leading fence line and a trailing fence line, then parse. If after stripping you still have English before the first brace, fail — do not search for the first { in a paragraph that also contains an example object. Heroic brace-slicing is how you execute the example instead of the action.

Enums and ranges: action must be in a frozen set. job_id must be an int >= 0 (or a UUID string with a regex you wrote). Coercing "17" to 17 is a product choice; coercing "seventeen" is a guess. Document the choice. Default: refuse strings for int fields.

Retry body: second call messages include a user or tool-style note with the exact validator string (missing keys: job_id). Do not say “please output valid JSON.” The model already tried to be valid. The error is the missing key.

A walkthrough: extra keys

The schema has action and job_id. The model adds comment": "user seemed angry". Extra keys fail validation. You could strip extras. Stripping is how a sneaky confirm: true survives if you later add that field carelessly. Reject extras unless you have a documented bag for notes that code never executes.

What goes wrong

• eval or exec on model text.
• Infinite retry with the same prompt.
• Optional fields that the model fills with invented amounts.
• Accepting job_id as a string "seventeen".
• Repair parsers that swap digits to “make JSON work.”
• Using an LLM-as-judge to “fix” JSON instead of a schema error string. Costly and circular.

How agents use this

Every tool argument hits the real world. Schema validation is an eval you run in the hot path. Two retries max, then hand off. Store validator errors on the trace. Temperature 0 for JSON steps. Native tool-calling when you can.

Unit-test validate with extras, missing keys, negative ids, and a fence-wrapped object. Unit-test run_with_retry with the prose-then-JSON script. You do not need a live model to prove the parser is strict.

Log parse_ok on the span. Alert if the fail rate leaves your baseline. The fix is usually temperature, a tighter schema, native tools, or a smaller set of fields — not a larger model that essays more confidently.

Tip:Send the schema error, not “be better.” Never execute a string the model labeled python. Ask for fewer fields.