Projects/Weather Tool Agent/Part 4

Evals and Tests

Test tools, the parser, goal predicates, and golden transcripts with a tiny stdlib runner — no pytest required in the browser.

If you cannot test the agent, you do not have an agent. You have a vibe. This part writes checks that fail in CI (or in the Try it box) when the loop regresses. Joeven's browser runtime has no pytest; a 30-line runner is enough. On your machine, paste the same assertions into pytest.

What to test (the pyramid)

Layer	Example	Speed	Catches
Tools	`geocode("Paris")` has lat	Instant	Catalog and error codes
Parser	extra keys rejected	Instant	Protocol drift
Policy-in-the-loop	golden goals → answers	Fast	Sequencing bugs
Evals	a table of goals and expected substrings	Fast	Product regressions

You do not need a live LLM to test the loop if the policy is injected. That is dependency injection: run_agent(goal, policy=fake_model). When you add a real model, keep the fake tests and add a smaller live eval you run nightly, not on every save.

Goal predicates

Write goal_satisfied before you love the sentence the model wrote. For weather:

Status is ok.
Answer contains a number (temperature) or starts with cannot:.
If the city is in the catalog, the answer must not start with cannot:.
If the city is unknown, the answer must start with cannot:.
Transcript must include a geocode tool event.

A predicate that only checks status == "ok" will bless empty poetry.

Golden transcripts

A golden test freezes the sequence of tool names for a known goal. Paris should be geocode → weather → finish. Atlantis should be geocode → finish. If a "smart" refactor calls weather for Atlantis, the golden fails. That is the point.

Do not freeze temperatures in goldens unless you want catalog edits to break tests. Freeze shape: tool order, error codes, presence of °C or C.

Live Pythonpython

import json

CITIES = {"paris": {"name": "Paris", "lat": 48.86, "lon": 2.35}}
FORECASTS = {(48.86, 2.35): {"temp_c": 18, "conditions": "cloudy"}}

def geocode(city):
    row = CITIES.get(str(city).strip().lower())
    return dict(row) if row else {"error": "unknown_city"}

def weather(lat, lon):
    row = FORECASTS.get((round(float(lat), 2), round(float(lon), 2)))
    return dict(row) if row else {"error": "no_forecast"}

TOOLS = {
    "geocode": lambda **kw: geocode(kw["city"]),
    "weather": lambda **kw: weather(kw["lat"], kw["lon"]),
    "finish": lambda **kw: {"final": str(kw["answer"]).strip()},
}

def parse_action(obj):
    if set(obj.keys()) != {"tool", "args"}:
        raise ValueError("shape")
    return obj

def policy(transcript):
    geo = next((e["content"] for e in reversed(transcript) if e.get("name") == "geocode"), None)
    wx = next((e["content"] for e in reversed(transcript) if e.get("name") == "weather"), None)
    goal = transcript[0]["content"].lower()
    city = "paris" if "paris" in goal else "atlantis"
    if geo is None:
        return {"tool": "geocode", "args": {"city": city}}
    if "error" in geo:
        return {"tool": "finish", "args": {"answer": "cannot: unknown city"}}
    if wx is None:
        return {"tool": "weather", "args": {"lat": geo["lat"], "lon": geo["lon"]}}
    return {"tool": "finish", "args": {"answer": f"Paris is {wx['temp_c']}C and {wx['conditions']}."}}

def run_agent(goal, max_steps=6):
    transcript = [{"role": "user", "content": goal}]
    for _ in range(max_steps):
        action = parse_action(policy(transcript))
        result = TOOLS[action["tool"]](**action["args"])
        transcript.append({"role": "tool", "name": action["tool"], "content": result, "args": action["args"]})
        if action["tool"] == "finish" and "final" in result:
            return {"status": "ok", "answer": result["final"], "transcript": transcript}
    return {"status": "max_steps", "answer": "", "transcript": transcript}

def tool_names(out):
    return [e["name"] for e in out["transcript"] if e.get("role") == "tool"]

def goal_satisfied(goal, out):
    if out["status"] != "ok":
        return False
    ans = out["answer"]
    known = "paris" in goal.lower()
    if known:
        return any(ch.isdigit() for ch in ans) and not ans.startswith("cannot:")
    return ans.startswith("cannot:")

def assert_true(cond, msg):
    if not cond:
        raise AssertionError(msg)

# Tool tests
assert_true(geocode("Paris")["lat"] == 48.86, "paris geocode")
assert_true(geocode("nope")["error"] == "unknown_city", "unknown city")
assert_true("error" in weather(0, 0), "missing forecast")

# Parser tests
try:
    parse_action({"tool": "geocode", "args": {}, "extra": 1})
    raise AssertionError("extra keys should fail")
except ValueError:
    pass

# Loop evals
cases = [
    ("What is the weather in Paris?", ["geocode", "weather", "finish"], True),
    ("Weather in Atlantis?", ["geocode", "finish"], True),
]
failed = 0
for goal, golden, expect_ok in cases:
    out = run_agent(goal)
    names = tool_names(out)
    ok = names == golden and goal_satisfied(goal, out) is expect_ok
    print(json.dumps({"goal": goal, "names": names, "answer": out["answer"], "pass": ok}))
    if not ok:
        failed += 1

print("failed:", failed)
assert_true(failed == 0, "eval table")
print("all tests passed")

Output

Run to execute this in your browser. Nothing is sent to a server.

Step-by-step: a test you can extend

Isolate tools. If geocode("Paris") breaks, you do not debug the loop.
Isolate parse_action. Feed dicts, strings, extra keys, missing args.
Run a table. Each row is a goal, expected tool sequence, and a boolean for goal_satisfied.
Count failures, print the table, then assert zero. Printing first makes the Try it box educational when it fails.
Keep tests deterministic. No random in default tools.

Eval vs unit test

A unit test pins a function. An eval pins a behavior the product cares about: "unknown places refuse." You will later score a real LLM the same way: same table, flaky rows marked quarantine, pass rate vs a threshold. Start with 100% on the fake model. That is your oracle. When the LLM scores 80%, you know the drop is the model, not the loop.

What not to assert

Do not assert exact English except for the cannot: prefix you control. Models paraphrase. Even fake models get refactored. Assert structure, codes, and a temperature digit.

Tip:Golden tool-name sequences are the highest value per line of test code in beginner agents.

Exercise

Add a case Weather in paris (lowercase). It should still pass. Add a case with no city (How is the weather?). Decide the product: refuse with cannot: no city, or default to a configured hometown. Write the predicate before you change the policy. Either choice is valid; untested defaults are not.

Check your understanding

Why inject fake_model instead of calling a paid API in unit tests?

APIs cannot return JSON
So loop tests are deterministic, free, and pin the control plane
Fake models are more accurate
Tests are illegal with APIs

JSON Action Loop and Retries

Up next

Hardening the Weather Agent