Code Interpreters

Let the model write Python for math — then run it in a tiny sandbox, not on your laptop as root.

advanced22 min16 / 24Mark complete

Models are messy calculators and decent code authors. A code interpreter tool takes a string of Python (or a tiny expression language), runs it, and returns stdout or a small table. It is the right tool for:

Arithmetic you refuse to trust from tokens
Filters and stats over a file you provided
Charts for a report, in a locked-down runtime

It is the wrong tool for “just run whatever so the agent can fix the server.” That is unattended remote code execution. If you need to change production, that is a write tool with approval, not a notebook.

Joeven’s Try-it boxes are a cousin of this idea: Pyodide stdlib, no network, no pip. Production interpreters need the same kind of ceiling, named and enforced in the worker.

Sandbox, not your laptop

Untrusted Python never shares a process with Stripe keys.

Threat model

Model-written code is untrusted, same as a user paste. It may try to import os, read secrets, loop forever, or network out. It may look like homework and still open a socket. The runtime is a sandbox: no network, no host filesystem (or a sealed folder), CPU/memory/time limits, no package installer, stdout size cap.

The dispatcher still exists. The interpreter is one registered tool, not a hole next to the API process. Running model text inside the same process that holds Stripe keys is how a sandbox becomes a kernel.

Whitelist, do not blacklist

Blacklists lose. You forget a builtin, or a dunder, or an object’s method that reaches the filesystem. A classroom interpreter allows a tiny AST: numbers and + - * /. A production data interpreter allows a data library in a container with no credentials. Both are allowlists.

If you need general Python, you need a real jail (container, gVisor, a dedicated service), not a filter that bans the word os. Filters are not sandboxes. They are delay tactics.

The classroom evaluator walks the AST. Allowed nodes: expression, constants that are ints or floats, unary minus, a closed set of binary ops. Import, attribute, call, name, subscript — forbidden. That is the whole security idea, scaled down to arithmetic. Scale up with isolation, not with a longer ban list.

What the tool returns

Stdout text, a small table, or an error type and message. Cap bytes. Do not return container paths. Do not return the list of env vars that failed to load. Log the code for humans (redact if it contains pasted secrets). Give the interpreter only the files it needs, mounted read-only if they must exist.

Disable networking even “just this once” for installing packages. The model will try. If the task needs a library, bake it into the image ahead of time. Session installs are a supply-chain incident.

Timeouts apply. Infinite loops are why you have CPU and time limits. Size caps apply to stdout. A print in a loop is a 2 MB observation.

Classroom arithmetic jail

Four samples: a product, a power, an import attack, an open attack. The first two return numbers. The last two raise because those AST nodes are not allowed. There is no hidden escape in the walker. If you add ast.Call to the allow set, you have left the jail. Do not.

Live PythonOpen full playgroundpython

import ast
import operator

ALLOWED_OPS = {
    ast.Add: operator.add,
    ast.Sub: operator.sub,
    ast.Mult: operator.mul,
    ast.Div: operator.truediv,
    ast.Mod: operator.mod,
    ast.Pow: operator.pow,
    ast.USub: operator.neg,
}

class SandboxError(ValueError):
    pass

def safe_arith(expr):
    tree = ast.parse(expr, mode="eval")

def ev(node):
        if isinstance(node, ast.Expression):
            return ev(node.body)
        if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
            return float(node.value)
        if isinstance(node, ast.UnaryOp) and type(node.op) in ALLOWED_OPS:
            return ALLOWED_OPS[type(node.op)](ev(node.operand))
        if isinstance(node, ast.BinOp) and type(node.op) in ALLOWED_OPS:
            return ALLOWED_OPS[type(node.op)](ev(node.left), ev(node.right))
        raise SandboxError("forbidden node: " + type(node).__name__)

return ev(tree)

samples = [
    "(3 + 4) * 10",
    "2 ** 8",
    "__import__('os').system('echo pwned')",
    "open('/etc/passwd').read()",
]
for s in samples:
    try:
        print(s, "->", safe_arith(s))
    except Exception as e:
        print(s, "->", type(e).__name__ + ":", e)

Output

Run to execute this in your browser. Nothing is sent to a server.

What printed: 70.0, 256.0, then two forbidden-node errors (or parse/type errors) for import and open. The whitelist rejects imports and calls because those AST nodes are not allowed. That is sandboxing as a allowlist, not as hope.

What goes wrong

Running the interpreter in the API process. Allowing network “for pip.” Mounting /. Returning full exception objects. Using a blacklist of function names. Letting the model choose the image. Sharing one container across tenants. All of these skip the runtime’s job.

How to test the interpreter

A golden list of allowed expressions and expected numbers. A golden list of forbidden snippets that must raise. A timeout test if you have a loop snippet and a timer. A size test for a long print. Never test by “trying to pwn it in a notebook once.”

Isolation is the product, not the expression language

The classroom AST is a teaching jail. Production still needs a process boundary: a container or a dedicated interpreter service with no production credentials, no network, a sealed filesystem, CPU and memory limits, a wall timer, and a stdout cap. Running model text in the API process that holds Stripe is how a clever constant becomes a kernel. The dispatcher calls the interpreter service. It does not import a runner next to the webhook handler.

Files the model may see are files you mounted on purpose, read-only, the minimum set, with the same path jail as read_file. Do not mount home directories. Do not allow session installs of packages. Bake libraries into the image. Session installs are a supply chain. Multi-tenant: one jail per job, no leftover disk, no shared /tmp.

Log the code for humans. Return exception type and a short message for the model. Never return host paths, env dumps, or the list of failed imports that reveals the image. Timeouts on infinite loops are mandatory. Size caps on print loops are mandatory. This tool is still on an allowlist: research sessions maybe, billing sessions no.

If you only need arithmetic, keep the tiny AST. Smaller jail, smaller eval, fewer surprises. General Python is a bigger product. Do not grow into it because a demo looked cool.

How agents use this

Expose calculate(expr) or run_sandboxed(code) as a named tool with a schema. The loop calls it like any other tool. It does not paste code into your shell. Log the code. Return exception type and message, not host paths. Permissions: this tool is still on an allowlist; research agents may have it, billing agents may not.

If arithmetic is all you need, do not ship general Python. Ship the tiny AST. Smaller jail, smaller eval.

Watch out:Model code is untrusted. Allowlists and isolation, not hope.

Check your understanding

How should an agent run model-written Python?

In the same process that holds production keys
In a sandbox: allowlisted ops or a locked-down runtime, no network, limits, logged code
By pasting it into production bash
Only if the thought said please

Timeouts, Retries, and Size Caps

Up next

Browser and Computer Use