Observe Before You Finish

ReAct’s most common bug is skipping the observation. The model calls a tool, then finishes with a pre-written answer. The thought already “knew” it would rain. The search was theater. Your loop must refuse that.

This is architecture, not a pep talk. Code it. A prompt that says “use the observation” is a wish. A gate that says finish is illegal until the last event is an observation is a system.

The six-part loop already appends observations in memory. This lesson is the rule on top: finish is not legal until the world has spoken, unless the goal is already solved in the prompt (rare: the user pasted the only fact you needed, and even then you should be honest that no tool ran).

The skip looks confident

Traces of the skip:

1. Thought: I know Paris weather.
2. Action: search (or get_weather)
3. Observation: {rain, 12C} — or the model never waits
4. Finish: “Bring shorts, it is hot.”

Sometimes step 3 is missing entirely: the model emits search and finish in one breath, or the assembler dropped the observation. Sometimes step 3 exists and finish ignores it. Both are guesses. The second is worse because you paid for a tool and then ignored it.

A verifier that only checks “did we call finish?” will pass both. A verifier that checks “does the answer mention retrieved facts?” or “was the last event an obs?” will fail them. Fail them.

Rules you can implement

• After a tool that is not finish, the next model call must include that observation
• finish is only legal after at least one observation or when the goal is already in the prompt (rare, and you should log no_tool_finish)
• If the model emits finish with an answer that does not mention retrieved facts, the verifier can reject it
• The executor always appends an observation after a non-finish tool, even if the observation is {"error": "timeout"}. Empty skip is how the gate is fooled.

The legal-finish gate in the live box is the blunt version: last event must be kind obs. That does not prove the answer used the obs. Grounded critics (later) score that. Start with the blunt gate. You cannot use evidence you never stored.

Finish-first is illegal

Finish as the first action means the model never looked. That is a chatbot with a finish sticker. For this track’s default (tools exist, the world might disagree), refuse it. Return illegal finish as an observation or as a stop. Do not run finish. Do not show the user a final answer.

If your product is “answer from weights, tools optional,” you still want a flag: used_tools: false. Do not pretend a tool was used.

The assembler can cause the skip

If the window dropped the latest observation to save tokens, the model will finish from the thought. Truncate old events, not the last obs. If you must truncate observations, keep the last one whole. The assembler budget lesson said never drop the goal. Here: never drop the latest observation either.

If you summarize, the summary must include the last tool JSON, not only “we searched.”

Parallel calls do not skip observe

Two reads in one turn still produce two observations before finish is legal. Do not finish in the same turn as a write. Do not finish in the same turn as a read unless you already stuffed those observations into the model’s next call — which means it is the next turn.

Classic ReAct: one action, then observe, then maybe finish. Keep that until you have a reason.

Illegal finish is a trace event

Log illegal_finish with the events you had. That is a better eval later than reading thoughts. This track: make the event exist so operators can search for it. If half your runs are illegal finish, the prompt is wrong or the assembler is dropping obs. If a few are, the model is rushing. The gate catches both.

What this is not

This is not multi-agent debate (“another agent checks the answer”). It is one loop that will not claim success before the executor has written. A critic can sit on top later. The gate is cheaper than a second model call.

Error observations count. A timeout is still an observation: the world spoke, it said fail. Finish that ignores the timeout is the skip in a different costume. The blunt gate (last kind is obs) would allow finish after a timeout; the grounded critic should not. Use both: legality of the name, then quality of the answer.

Common mistakes

• Prompt-only “please use the tool result.”
• Finish in the same breath as the first tool.
• Truncating the last observation.
• Treating a thought as evidence.
• Verifying only that finish was called.
• Swallowing tool errors so no obs is appended.

Run to execute this in your browser. Nothing is sent to a server.

Finish-first is illegal. After search, the executor always appends an observation. Finish is then legal. The gate is: last event must be an observation. The event list never gains a finish act on the first call. After search, last is obs. Then finish is ok.

This gate does not read the answer text. It only checks that the world spoke. Pair it with a grounded critic when the answer must cite the obs. Do not skip the gate because you plan to add a critic. The gate is free.

How agents use this

If traces show finish with an answer that does not match the last tool JSON, fail the run in review. That is cheaper than hoping the prompt said “use the observation.”

Put finish_ok next to should_stop. Stop on success only if finish was legal. An illegal finish should not count as success even if the model used the finish name.