Safety Filters

Hosted APIs often run safety filters on the prompt, the completion, or both. You may get:

• A refusal in assistant text (“I can’t help with that”)
• finish_reason like content_filter
• Empty or redacted content
• An HTTP error

This is not your schema. If you json.loads("") you will crash or invent an action. Branch on finish reason and emptiness first. A filter is a stop. Empty content is not structured output.

Your own policy still matters. Vendor filters are generic (weapons, scams, sometimes medical). Your agent must not refund without approval even if the filter said the text was fine. Filters are extra, not the whole policy. Code allowlists beat a poem in the user message. Jailbreak text from the user is still user text.

Handle the response before the parser

Order of operations:

1. HTTP error path (401, 429, …) — previous lessons
2. finish_reason and empty content
3. Native tool_calls if present
4. json.loads on text
5. Business validation

Skipping to step 4 is the bug. Filtered → blocked, do not parse. Length → truncated, maybe retry with more room or a smaller prompt (finish-reason lesson). Stop with JSON → ok.

Do not retry a filter with a sneakier prompt. That is how you get banned. If your product must discuss a sensitive but legal topic (security incidents, abuse reports), use a vendor and a spec that allow it, plus human review — do not fight the filter in a loop.

Run to execute this in your browser. Nothing is sent to a server.

Prints ok, blocked, truncated. Only ok should go to json.loads in a later function. The truncated row is incomplete JSON; treating it as blocked would also be safer than parsing. Here we label it so the next lesson can grow max_tokens or shrink the prompt.

Empty content with finish_reason=stop is also suspicious. The toy treats any empty text as blocked. That is a reasonable default for an agent that must emit JSON. A prose chatbot might allow empty less often, but agents should not.

Logging without storing abuse

Logging the raw blocked prompt can store the abuse. Redact. Keep a reason code (content_filter, empty_completion) and hashes if you need to count repeats. Full dumps belong in a locked review bucket with access control, not in the product warehouse.

Map blocked to a user-visible refusal and a trace event. Support should see “vendor filter,” not a stack trace from json.loads.

Your policy vs theirs

A vendor may allow “how do I reset MFA?” and still blank a payload that looks like credential stuffing. Your spec may forbid refunds the vendor would happily write a paragraph about. Implement both. Never use “the filter allowed it” as authorization for a tool.

If a filter is too hot for your domain (security research assistant), change vendor or product tier with a written review. Do not disable safety by stuffing “ignore all filters” into system. That is not a reliable API, and it is not a professional posture.

Vendor names for the same idea differ: content_filter, content_management, HTTP 400 with a policy code. Normalize to blocked in your wrapper so the loop does not grow a switch per SDK. Log the raw vendor code next to the normalized kind so you can debug a spike.

False positives: a billing FAQ that mentions “kill the process” may filter. Your move is not a sneakier prompt. Your move is a spec that stays in-policy, a vendor that allows ops language, or a non-LLM workflow for that intent (last lesson of this track). Retrying at temperature 2 is how you look like an attacker to the vendor.

Empty assistant text with stop is still unusable JSON. Treat it as blocked or truncated depending on whether you expected prose. Agents that must emit an object should fail closed.

Your policy can be stricter than the vendor. A completion that passes the filter and says “sure, refund 40” still must hit approval=true in your executor. Filters do not know your refund rules. Do not skip allowlists because the text looked “safe.”

What goes wrong

• Default tool on parse failure, including empty string.
• Retry with temperature 2 to “get around” a filter.
• Moving user text into system so the filter sees a spec. That also grants the user spec privileges.
• Showing the filter’s internal category to attackers as a debug string.
• Ignoring HTTP 400 that is actually a policy block with a different name per vendor.
• Showing filter categories to the user as debug. That is a map for attackers.

How agents use this

handle_response sits in the HTTP wrapper. The agent loop receives ok | blocked | truncated | http_error, never a raw empty string. Tests: the three samples above, plus a refusal paragraph with finish_reason=stop (you may still parse, then refuse in your policy if it asked for a forbidden tool).

Normalize vendor-specific filter names in the wrapper. Log reason codes, not abuse text, in the warehouse. Do not retry blocked with a rewritten user message.

If the product must handle security incidents, pick a vendor and a review path that allow that topic. Write it down. Fighting the filter in a loop is not a path. A non-LLM form for “report abuse” is often the right skip.

Branch on blocked before the parser. Empty string is not {}. Your refund allowlist still runs when the filter said the text was fine.

Watch out:A filter is a stop. Empty content is not structured output. Do not retry with a sneakier prompt.