Projects/ReAct Research Agent/Part 4

Citations, Max Steps, and Cannot-Answer

Validate citations against opened URLs, require evidence spans, and implement a first-class cannot-answer finish for missing facts.

This part is the difference between a search demo and a research product. Three rules, all enforced in code:

Citations ⊆ opened URLs.
Positive answers need at least one citation and a verbatim evidence span from an opened body (optional but we will implement a simple evidence in body check).
Cannot-answer is a legal, tested outcome — including max-steps and empty search hits.

If you skip these, the agent will quote https://wiki.example/acme because it is in the prompt examples, without opening it.

Finish schema

text

{
  "answer": str,
  "citations": list[str],
  "cannot_answer": bool,
  "evidence": str   # substring copied from a page body; empty if cannot_answer
}

Validation function validate_finish(payload, opened, pages):

Types and keys.
If cannot_answer: citations must be empty; evidence empty; answer starts with cannot:.
If not cannot_answer: citations non-empty, each in opened, evidence non-empty, evidence is a substring of at least one opened page body (normalize whitespace).
Reject answers that mention years or names not in the concatenated opened bodies (lightweight overlap check). This is imperfect; it still kills the worst hallucinations.

Cannot-answer triggers

Trigger	Agent behavior
Search hits empty	Open nothing; finish cannot-answer
Opens succeed but fact missing	Do not stitch a story; cannot-answer
`open` not_found for all attempts	cannot-answer
`max_steps`	cannot-answer with reason `budget`
Finish fails validation	Retry finish once, then cannot-answer

The last row matters. If you only raise, the user sees a 500. Convert invalid finish into a protocol error observation: finish rejected: citation not opened.

Live Pythonpython

import json

def validate_finish(payload, opened, bodies):
    need = {"answer", "citations", "cannot_answer", "evidence"}
    if set(payload) != need:
        return "keys"
    if not isinstance(payload["answer"], str) or not isinstance(payload["citations"], list):
        return "types"
    if not isinstance(payload["cannot_answer"], bool) or not isinstance(payload["evidence"], str):
        return "types"
    if payload["cannot_answer"]:
        if payload["citations"] or payload["evidence"]:
            return "abstain_shape"
        if not payload["answer"].startswith("cannot:"):
            return "abstain_prefix"
        return None
    if not payload["citations"]:
        return "need_citation"
    if any(url not in opened for url in payload["citations"]):
        return "cite_not_opened"
    blob = " ".join(bodies)
    ev = " ".join(payload["evidence"].split())
    blob_n = " ".join(blob.split())
    if not ev or ev not in blob_n:
        return "evidence"
    return None

opened = {"https://wiki.example/acme"}
bodies = ["Acme Robotics was founded by Ada Ortiz in 2014 in Lisbon."]

good = {
    "answer": "Ada Ortiz founded Acme Robotics in 2014.",
    "citations": ["https://wiki.example/acme"],
    "cannot_answer": False,
    "evidence": "founded by Ada Ortiz in 2014",
}
bad_cite = dict(good)
bad_cite["citations"] = ["https://en.wikipedia.org/wiki/Acme"]
missing = {
    "answer": "cannot: not in corpus",
    "citations": [],
    "cannot_answer": True,
    "evidence": "",
}
hallucinated_ev = dict(good)
hallucinated_ev["evidence"] = "founded by Marie Curie in 1898"

for name, payload in [
    ("good", good),
    ("bad_cite", bad_cite),
    ("abstain", missing),
    ("bad_evidence", hallucinated_ev),
]:
    print(name, validate_finish(payload, opened, bodies))

# max-steps product mapping
def budget_finish():
    return {
        "answer": "cannot: step budget",
        "citations": [],
        "cannot_answer": True,
        "evidence": "",
    }

print("budget valid:", validate_finish(budget_finish(), set(), []))

Output

Run to execute this in your browser. Nothing is sent to a server.

Step-by-step: wiring validation into the loop

When the policy calls finish, run validate_finish.
If it returns a string error, do not return to the user yet. Append {"role": "tool", "name": "finish", "content": {"error": code}}.
Allow one repair turn. If the second finish is still invalid, return budget_finish()-style abstention with reason invalid_finish.
Track opened only on successful opens.
Pass bodies of opened pages, not the whole WEB, into the validator. Otherwise the agent could "cite" an opened URL while copying evidence from an unopened page that leaked into the prompt. Keep page bodies out of the system prompt except as observations.

Eval table (use in the next part too)

Question	Expected
Who founded Acme Robotics?	Ada Ortiz, cite wiki/acme
Who is the current CEO?	Kenji Ito, must open news page (if snippet is short)
What is Ada's favorite food?	cannot_answer
Invent a URL in citations	validate_finish fails

Max steps vs infinite curiosity

Set max_steps=6. A policy that searches the same query twice is wasting. In part 5 you can detect duplicate queries and inject you already searched that. For now, the cap is enough.

Tip:Store cannot_answer as a boolean in JSON, not as English "I cannot answer" only. Machines grade booleans.

Exercise

Write validate_finish tests for: empty citations on a positive answer; citation to an opened URL but evidence from a different unopened string; cannot-answer with a leftover citation. All three must fail. Then implement the one-repair-turn in your loop from part 3.

Check your understanding

When is a citation valid in this project?

If the URL looks like Wikipedia
If the URL was successfully opened in this run and evidence is a substring of an opened body
If the model thought about it
If search returned a snippet

The ReAct Loop

Up next

Evals and Hardening