Designing Tools

Tool design is API design with a chaotic client (the model). The client is fluent, overconfident, and will call you in a loop. Design for that. The loop is a client of whatever you ship. If you ship a shell, you shipped a shell. If you ship get_user(user_id), you shipped a gate.

Small

One tool, one job. get_user(user_id) and list_orders(user_id) beat crm(natural_language_query). Small tools give you:

• Schemas you can validate
• Logs you can read
• Permissions you can split (read vs write)
• Evals per function
• Idempotency keys that mean one thing

A “do anything” tool is a shell. If you wanted a shell, say so and put a human in front of it, a sandbox around it, and a session cap of one. Do not name it helper and hope.

Saving round trips is the usual excuse for god tools. Extra turns are cheaper than untestable writes. If two tools always go together, the loop can call both. You can also add a third tool that does a fixed composition with no extra arguments (get_user_with_orders(user_id)) — still typed, still no natural language bag.

Typed

Arguments should be ids, enums, numbers, booleans. If you must take free text, take one field that is clearly a search query, not a field that is “the rest of the plan.” Search queries are still capped, still not SQL.

Return types should be boring too: JSON objects with ok or error, never a poem, never a Python object dump. Errors are codes. Lists have max length. Money is integer cents.

Names and versions

Version tools like APIs (get_job stays v1, or get_job_v2 with a migration). Renaming a parameter is a breaking change for every prompt and every eval. Prefer add-and-deprecate. Do not silently change job_id from string to int without a version and a coerce window.

Names should say read or write. sync_ is a smell. get_or_create_ is two tools. run_ is a shell unless the rest of the name is a tiny closed job (run_report with an enum of report ids is fine; run_code needs a sandbox lesson).

Errors, auth, and size

Every tool gets the structured error catalog, the observation cap, and a place in the allowlist. Design those with the signature, not after the first incident. If you cannot write the error list, the tool is too vague.

Authz is part of design: whose id is in the args? The end user or a free-form victim id? get_my_invoice versus get_invoice(any_id) is a product choice. The confused-deputy lesson will force the first for writes. Start now for reads that can exfil.

God tool versus two getters

The live box is the argument. god_tool concatenates the command into a lie. It cannot be made safe. get_user and list_orders look up dicts, miss with hints, and never take a shell string. Email as user_id misses. That miss is the design working.

Run to execute this in your browser. Nothing is sent to a server.

What printed: the god tool claims it refunded and read passwd. The typed getters return Ada, her order, and a miss for email. You can write pytest for the getters without a model. You cannot write a meaningful pytest for the god tool except “it returns a string.” That is the standard for every tool you add.

What goes wrong

Natural language parameters. Optional bags. Tools named after vendors. Tools that return different shapes on success (list vs object vs string). Hidden writes. Hidden network. Designing for the happy demo and adding errors later. Later never comes.

How to test design

If you cannot write a pytest without mocking an LLM, the tool is too vague. Fix the tool, not the test. Fixtures: valid, missing, extra key, not_found, and for writes a second call. If the fixture file is harder to write than the handler, listen to that.

Design the catalog like an API you will still own in a year

Every name needs an owner, a risk label, a timeout, a size cap, and a fixture file. If a proposed tool cannot fill that row, it is not ready. “We will add errors later” means you will add them in an incident. Later never comes.

Composition is allowed when it is typed. get_user_with_orders(user_id) is still one id, still two lookups you could have done in the loop, still no natural-language bag. It is a convenience with a schema. crm(command) is not a convenience. It is a shell. If two tools always travel together, you may add the composition or you may let the loop call twice. You may not invent a string argument that means “do the rest.”

Version in the name or in a header you control, not in silent field renames. Evals, traces, and stored keys refer to names. Renaming job_id to id breaks all three. Add get_job_v2, migrate, deny the old name with a hint. Deprecation is a runtime observation, not a wiki page.

Return shapes should be boring across the catalog: ok plus payload, or error plus code. Mixing lists, strings, and poems trains fragile parsers in the loop. The loop is a client. Treat it like a mobile app you cannot update every hour.

How agents use this

Catalog review is a product meeting: each name, schema, risk, cap, owner. The loop does not get a vote. When someone wants “just one meta-tool,” require a human, a sandbox, and a reason the small tools failed. Usually they did not fail. Usually nobody wrote them.

Keep the client (the loop) boring. Put intelligence in choosing among small tools, not in parsing a novel argument.

Watch out:A god tool cannot be permissioned, keyed, or eval’d per job. Split it.