Redact Before You Export

Traces are a new place you keep private data. They look like logs, so teams treat them casually. They are closer to a database of conversations, tool dumps, and error pages — including secrets that vendors stuffed into a 500 HTML body. Redact secrets, cookies, health data, and raw card numbers before export to a SaaS vendor. Retention: 30 days may be plenty. “Keep forever for training” is how you get a second incident.

If you export traces, that vendor is a subprocessor. Legal should know. Engineers should know what is redacted. Logging full prompts to a third party may break a contract you never read. Read the agreement before you enable “prompt telemetry.”

Scan for sk- and Bearer (imperfect, still worth it). Redact in the UI too — curious employees are a threat model. The exporter is not the only reader.

How the box actually works

Redaction is a pipeline stage, not a dashboard filter. Data flows: tool result → redact → scratchpad / spans → store → exporters. If you redact only in the UI, the vendor already indexed the key. If you redact only at export, the model already saw the key and may echo it on the next step.

Patterns to scan (none are proofs): sk-, Bearer , api_key=, long base64 cookies, eyJ JWT prefixes, PAN-shaped digit groups. Prefer structured tool results so you are not regexing HTML. A tool that returns {"code": "UPSTREAM_500"} is safer than one that returns the vendor’s error page.

Retention and access: traces are as sensitive as tickets. SSO groups. Audit “view trace.” Do not put the trace bucket on the public internet because the URLs look random.

Owners: security sets the nets and the subprocessor list. Runtime applies them. Legal keeps the inventory. On-call rotates keys when a leak is suspected — redaction is not encryption of a key that already left.

Allow-list fields on spans: job_id, tenant, tool, code, ok, token counts, usd_est, versions. Deny-list or drop: raw HTTP bodies, cookies, Authorization, environment dumps, full handbook pages. If a debug field is tempting, put it behind an audited “reveal” in the UI, not in the exporter.

Retention clocks: job rows might live 90 days; traces 30; blobs until the ticket closes. “Keep forever for training” needs a legal review and a redaction pass that is stricter than ops traces, not looser. Training data is another store, not a reason to skip the net.

A Datadog ticket

A worker logged upstream 500, key=sk-live-... because the billing SDK put the key in the exception. The trace exporter shipped it. A security scanner in the vendor’s UI found it 40 minutes later. That is a leaked live key, not a “log hygiene” nit.

Contain: rotate the key, disable the exporter, patch the tool to return a code, add the sk- net before the scratchpad. Legal added the vendor to the subprocessor list that should have existed first. The postmortem golden was “tool error bodies matching sk- never appear in stored spans.” The model did not need the real key to debug. It needed UPSTREAM_500.

Run to execute this in your browser. Nothing is sent to a server.

RAW would leak is a reminder: do not print the live string in real workers. SAFE should show [REDACTED_KEY] and Bearer [REDACTED]. The third line shows the same net on a prompt-injection style instruction. The exporter never sees the live key if this function runs first. The scan is a net, not a proof — a key with a weird prefix still needs structured errors and rotation.

What goes wrong

Redact after export. Redact only sk-live and miss sk-test and Bearer. Store full HTTP responses as spans. Train a fine-tune on raw traces. Leave reveal-secret on the trace UI without audit logs. Ship prompt telemetry because the vendor checkbox was default-on.

Workers that dump os.environ into an observation “for debugging.” Computer-use screenshots of a password manager. Support pasting traces into a ticket system with looser access than the trace store.

Regex that is so greedy it destroys job ids and you cannot debug. Pair nets with allow-listed fields.

How to test it

• Fixtures containing sk- and Bearer never appear in serialized spans after redact.
• A tool adapter test: SDK exception in, code out, no key.
• Exporter integration test with a fake sink: sink must not contain the fixture key.
• UI test: default view masked; reveal emits an audit row.
• Contract test: new span attributes go through an allow-list.

Assume the net misses. Practice rotation. Time it.

How agents use this

Redact before the scratchpad if a tool error might contain a key. The model is another exporter — one that repeats what it saw. Rotate after any suspected leak. A leaked key in a trace vendor is still a leaked key.

Security reviews should read this pipeline, not only the system prompt’s manners. Ban dumping the environment into observations. Pin tool dependencies so a package named by the model cannot become a secret vacuum.

When legal asks “where do prompts live?”, you should have a diagram: job store, trace store, exporter, model provider. Each has retention. Each has a redaction stage. “In the cloud” is not an answer.

Run the nets in one function that tools, assembler, and the trace writer all call. Three copies of regex will drift. Add new patterns in one place. Assume they miss: rotation drills still happen quarterly. A net that has never caught a fixture in CI is not wired.