Model vs Runtime

Keep three jobs separate, or you will debug them as one blob at 3 a.m.

• Model: choose a tool name and arguments, or a final answer
• Your runtime: validate args, check permissions, execute, truncate the result
• The world: whether the goal is actually done

The model never “has” your database password. It has a string that looks like arguments. If your code does not run, nothing in the world changes. That is the security boundary and the honesty boundary at the same time.

This split is the product. Frameworks add glue. Vendors add a “tools” flag. None of that moves execution into the GPU. The GPU proposes. Your process disposes.

Why the split exists

Models are untrusted. They are trained to be helpful, not to be your IAM. They will emit refund with a guessed invoice id. They will claim they already refunded. They will mix a thought with a tool call. If the runtime treats the assistant’s last sentence as a ledger, you have no product. You have a narrator.

The runtime is trusted in the opposite direction: it holds keys, talks to Stripe, opens files, hits MCP servers. That trust is dangerous if it believes the model. So the runtime must not believe the model. It believes a parsed name, a validated argument object, a policy table, and a function return.

The world is a third party. Your function can return ok: true and still be wrong if Stripe later reverses, or if the ticket was already closed. Traces should record what your function returned. Reconciliation with the world is a later job (webhooks, polls). Do not pretend the model can see Stripe directly.

Trace is truth

Your UI must not show “Refund sent” because the assistant wrote a polite paragraph. Only the dispatcher’s log is truth. If the model claims it called a tool without emitting a tool call, treat the claim as fiction. Store the claim if you want to grade honesty. Do not store it as an event in the ledger.

A useful log line is boring: tool name, args (redacted), result size, duration, policy version, who approved. You cannot replay 3 a.m. refunds from a chat screenshot. You can replay them from that line.

Redact. Invoice ids may stay. Card numbers, tokens, emails, and raw SQL must not. Truncation is a later lesson; redaction starts now: if you would not paste it into Slack, do not paste it into the trace UI the model will see on the next turn.

The model is a client, not a kernel

The agent loop — assemble context, call the model, parse, execute, stop — is a client of the tool runtime. It should look like any other client: it sends a name and a dict, it gets a JSON blob, it does not hold the Stripe key. If the loop inlines requests.post next to the prompt, you have collapsed client and kernel. Then a parser bug is a payment bug.

Treat the dispatcher as an internal API. Unit-test it with a fake model: a list of {name, args} dicts. You do not need an LLM to prove unknown names fail and known names return JSON. The next lessons build that dispatcher. This lesson only demands that you see the three jobs.

Claims versus calls

People write “the model refunded the user.” No. The model emitted text. Maybe that text was a structured tool call. Maybe it was a paragraph. Only the second job — runtime — can refund. Speak that way in tickets or you will assign the incident to “the AI” instead of to a missing validator.

The live box makes the difference visible. ui_status looks at a flag ran, not at the politeness of assistant_text. The first print is a story with an empty log. After refund("INV-17"), the log has a row, and the UI is allowed to say the refund happened.

Run to execute this in your browser. Nothing is sent to a server.

What printed: the first status line is a claim with an empty log. refund appends a row and returns ok. The second status is allowed because ran is true. The log finally names INV-17. That is the only sequence a product should trust.

What goes wrong

A chat UI that streams tokens and paints “Done” when the model says Done. A support agent that posts internal notes from the assistant message instead of from tool results. A test suite that asserts on the final sentence and never opens LOG. A wrapper that “helpfully” executes whatever JSON appears inside a thought. All of these collapse model and runtime.

The opposite bug is also real: a runtime so chatty that it executes tools the model did not ask for “because the user probably wants a refund.” That is not a runtime. That is a second unlogged agent. The model proposes. The policy may deny. It should not invent a call.

How to test the split

Write two fixtures. One: assistant text mentions refund, no tool call in the parsed decision, log stays empty, UI must not show success. Two: parsed call refund with valid args, function returns, log has one row, UI may show success. Neither fixture needs a vendor model. If you cannot write them, you do not have a split. You have a script.

How agents use this

Every tool call writes an event: name, args (redacted), result size, duration, who approved. If you cannot replay the log, you cannot debug 3 a.m. refunds. Put the event in the same store the loop already uses for memory — as data, not as new policy.

When you add a tool, you add a runtime path, not a prompt sentence. The prompt may mention the name. The name must exist in the registry or the mention is a lie. Generate the prompt snippet from the registry so those two lists cannot drift. That generation is runtime work, not poetry.

Watch out:A demo that prints “I emailed the user” with no mail server is a lie. Treat it as a failed test, not as a UX flourish.