Reflection and Self-Critique

Reflection is a second model call (or a cheap program) that looks at a draft and says: ship, retry, or hand off.

It is not a second personality. It is not “another agent” in the multi-agent sense. It is a function over a draft: input dict, output score and errors. You can implement it with code when the checklist is mechanical (tests, arithmetic, required keys). You can implement it with a model call when the checklist is linguistic (tone, citations present). Either way, cap retries. Either way, never exec the draft.

The original Joeven demo used exec on model code. Never do that. A critic that runs model Python is a shell as a service. The critic here is a function over a dict. The live box keeps that rule. Do not “improve” it by executing strings.

When reflection helps

Use it when:

• The first pass is often almost-right (code, citations, tone)
• You have a checklist the critic can score
• A retry is cheaper than a human
• You can stop if the same error repeats

Do not use it when the first pass is already a guess with no evidence. Reflection cannot invent a source. Observe-before-finish still applies: if there is no observation, a critic that says “looks good” is a vibe rater. Ground the critic in the next lesson.

Do not use it as a substitute for stop. “Try again” without a cap is a furnace. “Are you sure?” with no rubric is a coin flip that costs tokens.

Checklist, not vibes

A checklist is pass/fail items:

• Output equals n * n
• JSON has keys name and args
• Answer cites snippet ids that exist
• Tests green

A vibe is “be better” or “are you sure?” The critic must return ok plus errors you can log. If it only returns a paragraph of praise, you cannot branch.

Code critics are cheap and strict. Model critics are expensive and fuzzy. Prefer code when the check is mechanical. This lesson’s square critic is code. You do not need a model to know 4 squared is 16.

Retry cap and the same error

Cap at two or three tries. If try 2 has the same error as try 1, stop — that is a tool or spec bug, not a missing pep talk. Changing the draft function (or the prompt) after a structured error is allowed. Blindly calling the same model again is thrash.

The live box simulates a fix by swapping make_draft after a failed critic. In production you might send the error string back to the model once. You would still cap. You would still not exec.

Reflection sits on the six parts

The draft comes from the main loop (model + maybe tools). The critic is another model call or a program. The parser still fail-closes the draft if it is supposed to be JSON. Stop still owns the cap. Memory should store critic errors as observations so the next draft sees them — as data, not as a new personality.

Do not spin a second infinite loop. One critic call per try. Max tries in the reflect loop. Then ship, cannot, or handoff.

What you do not reflect

Do not reflect a pending write into existence. If the draft is “refund 400,” the critic should check policy, not cheer. HITL still freezes args. A critic is not an approver of money unless your checklist is the policy table — and even then, irreversible tools may still need a human.

Do not exec code to “see if it works.” Run tests in a sandbox you own, or check outputs as data. This lesson checks out == n * n. That is enough to learn the shape.

What the critic returns, and who retries

A useful critic return is small: ok, score if you want a number, errors as a list of strings the next draft can see. Do not return a three-page essay. The assembler will stuff errors into the next context as an observation. If that observation is huge, you recreated the assembler-budget bug inside reflection. Truncate errors. One line per checklist miss is enough: “expected 16 got 8.”

Who retries? The same specialist loop, with a cap. Not a new agent with a new personality. Not a crew. You swap the draft function (as the box does) or you send the error list back to the same model once. If try 2 fails the same error string, stop. That is how you tell a spec bug from a sloppy first draft. Spec bugs belong in your tests and your tool, not in try 6.

Reflection does not replace stop, HITL, or observe-before-finish. Order of gates:

1. Parser blessed the draft shape.
2. Executor (or a dry run) produced evidence if tools were needed.
3. Critic scored the checklist.
4. HITL if the tool is irreversible.
5. Then the world changes.

If you criticise after the refund, you wrote a postmortem. If you criticise without evidence, you wrote a vibe. If you exec the draft to “run the tests,” you gave the model a shell. The square example is deliberately not code execution: it compares two numbers. When you later test real code, run your tests on your runner with an allowlist, never exec of model text. That rule does not relax because the critic “looked careful.”

Common mistakes

• exec / eval on model text.
• “Are you sure?” with no rubric.
• No retry cap.
• Critic with no evidence (next lesson).
• Treating reflection as a second agent society.
• Using reflection to skip observe-before-finish.

Run to execute this in your browser. Nothing is sent to a server.

Try 1 fails the checklist (8 is not 16). Try 2 uses the corrected draft function and passes. No exec. No second soul. The already-good path passes on try 1. The critic is arithmetic. That is the point: a function over a dict.

If both drafts were wrong, the loop would hit cannot. That is a stop, not a fourth try. Keep it that way.

How agents use this

Ground the critic (next lesson). Cap retries. If the same error repeats, stop — that is a tool or spec bug, not a missing pep talk.

Put critic results in the trace: score, errors, try index. Operators should see “critic rejected expected 16 got 8,” not “the agent reflected.” Names of parts, again.