Traces You Can Debug

If you cannot answer “why did it refund?”, you do not have an agent — you have a demo.

A trace is the movie of the loop: one row per step, enough to replay, redacted enough to store. Prompting wrote the text. Here you store the movie. Operators debug the loop. Customers see an answer. Those two UIs must not be the same document.

Redact secrets. Keep thoughts out of the customer UI. Keep them in the operator UI. Traces are not “a privacy bug by definition.” They are how you operate. Privacy is redaction and access control, not deleting the only record of a refund.

This track is not the eval course. You still log structured rows now so later you can replay with a stubbed model. If you skip the movie, you will not have something to replay.

A row per step

Each row should have:

• run_id, step
• tools advertised (assembler)
• raw model text (truncated)
• parse ok / error
• tool name and args (redacted)
• observation (truncated)
• stop reason or null
• phase / specialist role if you have them
• freeze hash if HITL

If you only store the final sentence the user saw, you cannot answer “why.” If you store the full production database, you built a data leak, not a trace. Truncate raw and obs. Cap is a budget, same as the assembler.

Truncate on purpose

Raw text is capped so traces do not become a second furnace. Observations too. Full blobs live in object storage keyed by run_id if you truly need them, with a tighter ACL. The operator table should be scannable: step 1 search, step 2 finish success.

The live box caps at 40 characters so you see the policy. Production might cap at 2k. Infinite is how your log system bills more than the model.

Redact

API keys, bearer tokens, raw card numbers, passwords: never in traces. Tool args may contain PII — hash or drop those fields. Policy: a denylist of keys (authorization, password, ssn). If you cannot store it, you also cannot paste it into the next assemble. The assembler and the tracer share the redaction function.

Thoughts: operator yes, customer no. Some vendors forbid showing chain-of-thought. Store if allowed; still hide in product UI.

Operator vs customer

Customer: answer, maybe a short “I looked up ticket T1.” Operator: advertised tools, parse_ok, name, obs stub, stop. When a refund happens, the operator row must show the blessed args and the freeze hash, not only “Refund sent.” Runtime log is truth (tools track). Trace is that log in loop order.

Replay

Replay means: load rows, stub the model to return the same raw (or skip to parse), stub tools, assert the next decision. You need raw or the parsed decision. You need advertised tools to know the assembler was in play. You do not need the full DB.

A stubbed refund on replay is a checkpoint lesson. The trace tells you which step. Without step-indexed rows, replay is fan fiction.

Count reasons later

Stop reasons, parse errors, illegal finish, thrash, circuit_open: enums you can count. This track only needs them on the row. Do not wait for an eval harness to start logging. If the field is missing, you cannot count it.

What to redact, what to keep, and how rows chain

Keep: run_id, step, advertised tool names (not 40 full schemas), parse error codes, tool name, redacted args (drop secrets, keep order_id and amount if policy allows), obs kind and short body, stop enum, phase, role, freeze hash. Drop: bearer tokens, passwords, raw cards, full HTML, untruncated thoughts in the customer export.

Args redaction is a denylist of keys plus a max string length. If you drop amount, you cannot debug HITL. If you keep Authorization, you shipped a secret. Choose field by field. The same function should run before assemble when obs go back to the model.

Rows chain by run_id + step. Missing step 3 with 1,2,4 present is a bug in log_step, not in the model. Operators should notice holes. Parallel tool calls (if you add them) need ids on the row so obs can land out of order. Classic ReAct: one row, one action.

Customer export is a different document: final answer, maybe citations, maybe “we looked up T1.” It is not a truncated operator trace. Mixing them is how thoughts leak. Operator UI is gated. Retention: traces live long enough to debug and to replay; they are not eternal chat. Prod policy comes later; this track needs you to have a row.

When a run refunds, you should find one row with name refund, parse_ok true, freeze hash matching the HITL ticket, obs ok. If you cannot, the movie is incomplete and “why did it refund?” has no answer.

Common mistakes

• Final sentence only.
• Infinite raw dumps.
• Secrets in rows.
• Thoughts in the customer UI.
• No run_id / step.
• Trace = production DB dump.
• Skipping traces for “privacy” without redaction.

Run to execute this in your browser. Nothing is sent to a server.

Two rows. Success is on step 2. Raw text is capped so traces do not become a second furnace. Step 1 has parse_ok and name search. Advertised tools are stored as a list copy. That copy is how you debug “why did it think refund existed?” — look at tools on that row, not at today’s code.

Add a parse-fail row in your head: parse_ok false, name None, stop maybe null, obs the error. Still a row. Missing rows are how “the model was weird” tickets start.

How agents use this

This is how you debug and, later, how you eval: replay traces with a stubbed model. Prompting wrote the text. Here you store the movie.

Tool thrash next is a stop reason you will want on this row. Handoff too. If the movie cannot say cannot: tool thrash, you will argue about vibes in the queue.