Tracing and Observability

Observability for agents is traces first, metrics second, logs third. A counter that says “errors up” does not tell you which tool, which tenant, which prompt version. A pretty token stream is for demos. At 2 a.m. a human needs a table: what happened, in order, with durations and estimated dollars.

You can start with a JSON table and still be production-shaped. OpenTelemetry names are useful even if the first exporter is “insert into traces.” The parent span is the job. Children are model calls, tools, parses, and human waits. If you only log the final paragraph, you cannot operate the loop.

This lesson is how you see a job. Redaction, paging, cost caps, and CI gates come next. Without spans, those controls are blind.

How the box actually works

Every job gets a trace_id (often equal to job_id). Every interesting wait becomes a span with a parent. Attributes are how you slice later.

Roll-ups an operator wants in one screen:

1. The timeline — model vs tool vs queue wait vs HITL. A job that “feels slow” is often queue time, not genius thinking.
2. Parsed tool calls — name and arguments after schema check, not only raw tokens.
3. Whether a human gate was skipped.
4. Cost so far vs cap, from usd_est on spans.

Owners: runtime instruments the worker (start/stop spans). Domain tools add their own child spans. Data owns retention and the trace store. On-call owns the saved views: “last 20 failed jobs,” “cost by tenant this hour.”

Put trace_id on user-facing error pages as a support code. Attach a trace link to every alert. “p95 is high” without a job id is a riddle.

Operator procedure for a slow or expensive job:

1. Open job.run. Read tenant, versions, status.
2. Sort children by start time. Note queue wait if you recorded it as a span or as queued_ms on the job.
3. Sum usd_est. Compare to the cap on the job row.
4. Click the first failing tool or the fattest llm.complete. Read parsed arguments, not the token stream.
5. If HITL exists, confirm a hitl.wait span with a decision. Missing span plus a write tool is a bypass.

If that procedure requires grep, the product is not operable yet. Build the screen that makes those five steps clicks.

A cost-spike ticket

At 01:40 a finance bot posted “model spend 8× weekday baseline.” Metrics said tokens up. Nobody knew why. Slack filled with screenshots of the chat UI. Three engineers grepped different log piles for the word “refund.”

The missing object was a trace for job_17: two llm.complete spans totaling ~930 ms and a handful of cents, plus a tool.search_kb at 80 ms. Once spans existed, the spike was obvious the next time: a new prompt version stuffed the whole handbook into every step, input tokens climbed, usd_est rolled up by prompt=p13. The page became “p13 cost per job,” not “the AI is hungry.”

Support stopped asking customers to paste the conversation. The error page already had job_17.

Run to execute this in your browser. Nothing is sent to a server.

The JSON dump is four rows: one job, two model calls, one search. SUMMARY should show about 930 ms in the model, 80 ms in tools, two LLM calls, and about 0.005 dollars. Most of the wall time was the two llm.complete spans, not the handbook search. Dollars roll up from spans. That roll-up is what you chart per tenant and per prompt version. If usd_est is missing, your cost dashboard is fiction.

What goes wrong

Logging only completions. Logging only stdout. Spans without tenant or prompt version, so you cannot group. Spans with full prompts and secrets (next lesson). Tracing sampled at 1% on a product that has 40-step jobs — you will miss the expensive one. Dashboards of token totals with no exemplars. Alerts that cannot open a trace.

Another failure: treating the vendor playground as observability. When the vendor is the incident, you have no copy of the spans.

Pretty UIs that show tokens streaming but do not persist parsed tool names: demos look alive, incidents look empty.

How to test it

• After a fake job, assert spans exist for job.run, each llm.complete, and each tool.
• summarize math: llm_ms, tool_ms, usd, llm_calls match fixtures.
• Attributes required: tenant, job_id, model, template version (from the stamp).
• Unknown job id on the trace UI is a 404, not another tenant’s trace.
• An alert fixture includes a trace_id field. CI fails if the pager payload lacks it.

Load a recorded job in the UI and time how long until an on-call engineer can say “two model calls, search ok, 0.005 dollars.” If that takes grep, the product is not operable.

How agents use this

Instrument the worker, not the prompt. Wrapping complete() and call_tool() is the whole trick. Every new tool gets a tool.<name> span with ok and code. Every cap event gets a span or an event: MAX_USD.

Put the support code on the error page and in the staff admin. Paste culture should die. Saved query: last 20 failed jobs with tool names and dollars. Exemplars beat averages.

When you page a human, the first link is the trace. The second is the versions stamp. The third is the ledger if money moved. If you cannot produce those three, you are not paging, you are starting a scavenger hunt.

Export is not observability. If the only copy of spans lives in a vendor UI, a vendor incident makes you blind. Keep a store you can query when they are down. Sampling is allowed on huge successful FAQ jobs; do not sample away failed jobs, forbidden tools, or anything over a dollar.

Tip:Queue wait is a span too. A “slow model” that sat in queued for 12 minutes is a capacity bug.