Log the Call

If it is not on the trace, it did not happen. LLM calls need the same habit as tools. Finance, a user who says “it ignored me,” and your future self replaying a failure all need the same JSON line.

Log (redacted):

• request id / vendor id
• model name and knobs (temperature, max_tokens, top_p, stops)
• spec id (spec@2026-03-01)
• usage: prompt tokens, completion tokens, cached tokens if any
• finish reason
• message roles and lengths, not necessarily full secrets
• parse ok / validator error
• spent so far vs cap
• router decision and whether you escalated

You need this when finance asks, when a user says “it ignored me,” and when you replay a failure into a fake client. Without this, you will argue from screenshots.

Full-prompt logs are a privacy incident waiting. Default to roles + lengths + redacted tails. Full dumps in a locked debug bucket, not in the product warehouse.

Redact both sides

If the user pasted a key, it sits in a user message. If the model echoed it, it sits in the assistant message. Redact both. Prefer storing hashes of secrets your scanner knows. Do not print sk-live in a “preview” field. The tryit shows the idea with a tiny replace loop. Production uses a list of patterns and a vault scanner, and still fails on novel secret shapes — so minimize what you store.

Version the spec id on the span. When someone “just added a sentence,” you will know which traces used which poem. Group evals by model + spec id.

What a replay needs

Replay is the reason you log messages at all. A failing ticket should become a unit test: load the stored message list (redacted), feed FakeChatClient a scripted next turn, assert the tool name. If you only stored roles and lengths, you can still see that a tool row was missing. If you stored redacted contents in a debug bucket, you can reproduce the parser bug. If you stored nothing, you will re-ask the user to “send it again.”

Same id as the agent step: trace_id, span_id, step=4. When a tool ran, the tool span and the LLM span share the trace. “The model ignored the tool” is a join query: was there a tool span, and was there a tool message on the next LLM call? Logging only the assistant preview cannot answer that.

Usage fields answer finance. prompt_tokens climbing across steps of one trace is the cost lesson in a chart. Cached tokens (if the vendor sends them) tell you whether prefix cache is fiction. finish_reason histograms catch a packing regression (length spike) or a prompt change (filter spike) before anyone names a vendor.

Knobs belong on the span because “JSON broke on Tuesday” is often “someone set temperature to 1 on the tool step.” You will not remember the playground default you copied. The log will.

PII and retention

Default warehouse: model, knobs, spec id, usage, finish reason, roles, lengths, parse ok, spent, router role, escalate flag, request id. No raw ticket body. No raw tool HTML. Debug bucket: full redacted messages, 14-day retention, locked ACL, sampled 1% plus all errors. Legal will thank you for the split. A warehouse that holds every prompt is a second CRM you did not mean to build.

Redact before persist. The tryit redacts the preview; production must redact every content field. If a secret appears in a validator error (“bad key sk-live…”), redact errors too. Prefer ValueError("missing API_KEY") in your code so you do not have to redact your own exceptions.

Streaming: log the assembled message, not each delta. Cost and PII both explode with deltas.

Run to execute this in your browser. Nothing is sent to a server.

The printed span has roles, usage, finish reason, and a preview with SECRET redacted. The user message still contains sk-live-abc inside call if you logged the raw object — so you must redact messages before persist, not only the preview. The toy redacts the assistant tail. Homework in your head: run the same redact on every m["content"] before you write the warehouse row. If you skip that, the span is a second copy of the key.

One JSON line per LLM call, same id as the agent step. Replay: feed messages into FakeChatClient. Eval: group by model and spec version.

What goes wrong

• Screenshots as the source of truth.
• Full prompts in the product warehouse.
• No usage, so the bill is a mystery.
• No spec id, so a prompt PR cannot be blamed or credited.
• Logging deltas (streaming lesson) and multiplying PII.

How agents use this

A span() function next to the HTTP client. Tests: redact removes sk-live; a missing usage field fails the schema of the log itself. Dashboards: call count, tokens, finish reasons, JSON fail rate, escalate rate, spent vs cap.

When finance asks why yesterday’s LLM bill jumped, answer with call count, model, and usage tokens per trace — not with adjectives from the system prompt, not with the user’s email body verbatim.

A useful finance view is three numbers per day: calls, input tokens, output tokens, broken down by model and by step kind. A useful eng view is JSON fail rate, length rate, filter rate, escalate rate. If those sit in different systems that cannot join on trace_id, you will still argue from screenshots. Put them on the same span.

Retention: usage and ids can live as long as the ticket. Raw prompts should not. When a customer asks you to delete their data, the debug bucket must be in the deletion path. If it is not, you do not have a log policy. You have a leak with timestamps.

Watch out:Full-prompt logs are a privacy incident waiting. Default to roles + lengths + redacted tails. Full dumps in a locked debug bucket.