Tool Fixtures and When Tools Fail

You do not need an LLM to test tools. You need fixtures:

1. A valid argument object → ok
2. Missing required field → missing
3. Extra key → unknown field
4. A write called twice → same receipt
5. A name not on the allowlist → denied
6. Path or host escape → denied before IO
7. Actor over limit → user_limit
8. Huge result → truncated true

Re-run this set in CI when someone “just adds a parameter.” Schema drift is how agents break on Tuesday. The loop is a client. If the catalog is wrong, the loop cannot save you with a better thought. Stop prompting. Fix the functions.

Tools are functions. Unit-test them. Models are for choosing when to call them. Grade agents on whether they called a legal tool, not on whether the prose sounded sure. If prompting cannot save a god tool, delete the god tool.

Keep fixtures next to the schema

Same directory, same review, same deploy. When the schema gains a field, the valid fixture gains it and a new missing-field case exists. When a tool is disabled, its fixtures still run against deny. Golden lists of enabled names per surface belong here too.

Do not harvest fixtures only from production traffic. Production is late and messy. You still want a live shadow log of real arg shapes — to add cases, not to be the only suite.

When this track is not enough

If the model picks the wrong tool, first check descriptions and names. If it picks the right tool with extra keys, check the validator. If it never stops calling tools, you need stop rules (prompting track) and caps (this track). If it refunds the neighbor, you have a confused-deputy bug, not a temperature bug.

The next tracks are RAG (search as a tool with citations) and agents (loops, memory, when not to agent). Tools stay the hands. Those tracks stay the brain and the library. Do not drag loop design into the dispatcher. Do not drag retrieval ranking into JSON Schema. Stay in lane: schema, dispatch, idempotency, sandboxes, MCP, permissions the runtime enforces.

Classroom fixture runner

Three rows: Paris ok, empty object missing city, extra drop unknown. failures must print 0. Change the validator to ignore extra keys and watch FAIL. That FAIL is the CI you wanted before shipping.

Run to execute this in your browser. Nothing is sent to a server.

What printed: three PASS lines and failures 0. When failures is not 0, do not ship the tool. Do not “see how the model does.” The model is not the schema test.

What goes wrong

Only production traffic. Only happy-path demos. Fixtures in a notebook nobody runs. Grading on prose. Adding parameters without adding extra-key cases. Skipping write-twice. Skipping deny. These green-light catalogs that injection already owns.

How to test the suite itself

Count fixtures per tool. Fail CI if a tool has fewer than the minimum three. Fail if a write tool lacks a double-call case. Fail if the enabled-set golden drifted from the registry keys without a review marker. This is boring. Boring is operable.

When the catalog is wrong, stop prompting

Wrong-tool rates that survive a rename and a clearer description are often a god tool, a rhyming pair, or a missing getter. Extra keys in traces mean the validator is not actually additionalProperties: false. Double emails mean the write has no key. Neighbor refunds mean actor is not from the session. Huge next-prompts mean the packer is not on the path. None of those are temperature. None of those are “add please to the system prompt.” Add a fixture that would have failed CI, then change the function.

Minimum suite per tool: valid, missing required, extra key. Writes add double-call. Allowlists add deny. Path and host tools add escape cases. Identity adds actor-over-limit. Packing adds a huge result. CI fails if a tool has fewer than the minimum, or if the enabled-set golden drifted from registry keys without review. Harvest production shapes to add cases. Do not wait for production to be the suite.

Stay in lane when you debug. Retrieval ranking is not JSON Schema. Loop stop rules are not the dispatcher. This track owns schema, dispatch, idempotency, sandboxes, MCP filters, and permissions the runtime enforces. The agent loop is a client of that contract. If the contract is untested, you have hope with JSON. When a fixture fails, the owner of the tool — not the prompt author — gets the ticket. That ownership line is how catalogs stay small. Tools nobody will fixture should not ship. A green demo without the eight cases above is not a catalog. It is a risk with a name. The cheapest honest run is still three dicts and a loop with no model: valid, missing, extra key. Everything else in this track — keys, jails, actors, packed observations — is more of the same idea. If you cannot run it in CI, you cannot claim the runtime enforces it.

How agents use this

Keep fixtures next to the schema in git. The loop’s evals should import the same validators. When a new MCP server version lands, run host-policy goldens before you advertise a name. When tools fail in production, add a fixture that would have caught it, then change code. Do not add a sentence to the system prompt and call it a fix.

This track’s contract: the runtime enforces schema, dispatch, idempotency, sandboxes, MCP filters, and permissions. The agent loop consumes that contract. If the contract is untested, you do not have tools. You have hope with JSON.

Note:Three fixtures, zero LLM calls. That is the cheapest honest test.