Computer Use in the Loop

Computer use means the agent sees pixels (or a DOM dump) and emits clicks. It is the worst executor you should ship:

• Slow and expensive
• Easy to click the wrong thing
• Hard to audit
• Breaks when the UI moves

Prefer an API or a structured tool (tools track). Use computer use only when no API exists, and then treat it as a bounded grid with a ban list, a click budget, and observe-before-next-click. Never “almost Pay.”

This lesson is loop policy, not a GUI driver. The tools track designed the GUI tool. Here: observe the new screenshot before the next click, budget the clicks, stop on banned labels, stop on unexpected copy. HITL still owns money. A screenshot of a Pay button is not an approval.

Last resort

If refund(order_id, amount) exists, the agent must not click through the billing UI. Computer use is for the leftover: a vendor with no API and a rare flow. If the flow is common, build a tool. If the flow is money, still do not click Pay — build a typed money tool with HITL, or handoff.

“After three failed API calls, click Pay” is in the quiz for a reason. Failure of an API is a breaker/handoff, not a promotion to pixels.

Bound the grid

A known grid of cells with labels is something you can log: clicked Save at [2,0]. A raw x,y on a 4k screenshot is not an audit. Bound the screen to a region. Map cells to labels. Empty cells error. Off-grid errors. The parser should emit a cell, not “click the green thing.”

Banned labels: Pay, Transfer, Delete, and anything your policy table would have marked needs_approval or irreversible. The loop treats banned as a stop, not as a retry with a nearby cell. Nearby is how you hit Pay after missing Save.

Observe after every action

Computer use is still ReAct. Click is an action. The new screenshot (or a label diff) is the observation. Do not click twice from one thought. Observe-before-finish applies: do not claim “saved” until the obs shows Saved. Do not finish because the thought was confident.

Unexpected copy (“Are you sure you want to pay?”) is a stop. The model will click Yes. Code should not allow Yes on that dialog.

Budget the clicks

Clicks are steps. Cap them. Thrash: same cell three times is a stop. A furnace that clicks File, File, File is still a furnace. Wall time is worse than API loops because each step is a screenshot token dump. Assembler budget: send a cropped grid, not a 20MB image every turn, if you can.

Never click Pay

Pay is banned. Use a typed money tool with HITL. Pixel color is not a contract. The thought “the invoice is done” is not a contract. Three failed API calls are not a contract. If the only way to pay is the button and you have no typed tool, handoff. A human clicks Pay. That is HITL with a UI, not an agent with a banned cell.

Dialogs, UI drift, and when this costume is allowed

Unexpected copy is a stop class: confirm-pay dialogs, 2FA prompts, “transfer to,” cookie walls. Maintain a list of banned substrings in the observation text (or OCR). Hitting one returns banned or unexpected_copy and stop/handoff. The model will always have a thought that says Continue is fine. Code disagrees.

UI drift: a button moved, the grid is wrong, Save is now at [2,1] where Pay was. That is why pixels are a last resort. If labels are parsed from a structured accessibility tree, prefer that over raw pixels — still bound, still ban Pay. When the tree lies, stop. Do not “find Pay by icon.”

Allowed computer-use jobs are narrow: download a CSV from a vendor with no API, click Save on a form you cannot POST, read a label. Even then: click budget, observe after, no writes that match the irreversible policy table. If the job is daily, build the API adapter. Computer use as a daily driver is a cost and audit hole.

Assembler: send the grid of labels, not a 20MB PNG, unless you have no alternative — and then crop. Dumping full screenshots every turn wrecks the budget and hides the ban list in noise. Trace: log label clicked and at [r,c], not a screenshot in the customer UI.

Parser: cell coordinates plus optional label check. If the model says “click Pay” as a name, that is unknown_tool or banned before the driver runs. Fail closed at the name, not at the pixel.

Common mistakes

• Computer use as the default executor.
• Unbounded x,y clicks.
• Retrying a banned cell nearby.
• No screenshot observation.
• Treating green pixels as approve.
• Dumping full screenshots into every assemble.

Run to execute this in your browser. Nothing is sent to a server.

Save is allowed. Pay is banned. Empty cells error. File is a harmless click. The loop should treat banned as a stop, not as a retry with a nearby cell. There is no “click [2,1] but nicer.” There is handoff.

A real driver would take a screenshot after Save and parse whether the name stuck. This box is the policy: labels plus a ban set. Keep the ban in code, not only in the prompt (“please do not click Pay”).

How agents use this

The tools track designed the GUI tool. This lesson is the loop policy: observe the new screenshot before the next click, budget the clicks, never “almost Pay.”

Router/specialist/verifier next is how you keep computer use (if you must) inside a small specialist with a tight allowlist — not inside a god-loop that can also refund.