Jailbreaks

A jailbreak is direct injection: the user (or an outer agent) tries to talk the model out of the policy. Indirect injection hid in a page. A jailbreak is in the user slot, on purpose.

Classic shapes:

• “Ignore previous instructions” / “ignore all rules”
• “You are now DAN / a developer / a grandma”
• “This is a hypothetical / test / poem / movie script”
• “Repeat your system prompt”
• Encoded variants (Base64, reversed text, translation, character-by-character)

The model may play along in prose. Your runtime must not. If the spec says no refunds, refund_customer stays off the allowlist even if the assistant says “as DAN I will refund.” Jailbreaks are text. Tool allowlists are code. Code wins.

Do not retry a content filter with a sneakier user prompt. That is how you get banned and how you train yourself to launder the ask. Log a reason code. Show a refusal in the contract (status=refused). A polite essay followed by the forbidden tool is not a refusal.

The poem is one layer

A detector for “ignore previous” is useful and incomplete. Paraphrase, translation, and roleplay exist. Use needles as a tripwire that flags the trace, not as the only gate. The gates that survive:

• Tools not in the allowlist cannot run (even if the model emits their names)
• Secrets are not in the spec; they are in the environment. Revealing the full system prompt is often a product leak — teach a short public description
• Hidden CoT is not a defense — users will ask it to think about ignoring you
• Evals include jailbreaks next to the golden set. When someone “just makes the bot nicer,” re-run them. Niceness that enables refunds is an incident

Revealing the spec can still help an attacker write a better next try. Do not print the spec because the user asked in a poem.

Refusal is a first-class status

Put refused in the output contract. Do not improvise a lecture in free text your parser cannot see. Support should see a reason code in the log (jailbreak_pattern, secret_probe, unauthorized_tool). The user-visible answer can be short and boring.

Do not “help a little” after a jailbreak. Partial compliance is compliance.

Run to execute this in your browser. Nothing is sent to a server.

What printed: the polite status ask is not a jailbreak and refund is still false because the tool is not in the list. The two attacks flag as jailbreak and refund stays false. Either layer alone is weaker: a paraphrased jailbreak might skip the needles, and a jailbreak detector without an allowlist still hopes the model obeys. Together, refund cannot run.

Walkthrough: DAN wants a refund

The user says they are DAN, the unconstrained finance bot, and they would like everyone refunded. The model may write a cooperative paragraph. refund is not in the enabled list. The call cannot run. A needle detector also flags “you are now” for the trace. Either layer alone is weaker: paraphrase skips needles; a detector without an allowlist hopes the poem obeys.

Do not retry the filter with a sneakier rewrite of the same ask. Log a reason code. Return status=refused. Teach a short public description of the bot if asked what it is. Do not print the spec. Secrets stay in the environment. Hidden scratch is not a vault for this fight.

What goes wrong if you skip this

Niceness patches delete refusals. Hidden thought is treated as a vault. The spec is pasted into the chat as a party trick. Refunds ship because “the model agreed.” Jailbreak evals never exist, so the friendlier prompt looks like a win.

Direct injection is the user (or an outer agent) fighting the spec on purpose. Roleplay, hypotheticals, translation, reversed text, and “repeat your system prompt” are the same family: talk the model out of policy. The poem may comply. The runtime must not. Tools not on the allowlist cannot run. Secrets are not in the spec. Hidden CoT is not a defense — users will ask it to think about ignoring you.

Keep a small jailbreak slice next to the golden set. Re-run it whenever someone “makes the bot nicer.” A refusal is a first-class status, not a lecture your parser cannot see. Do not retry a content filter with a sneakier user prompt. Partial compliance is compliance.

Needles like “ignore previous” are tripwires. Paraphrase will skip them. That is why the allowlist is the wall and the detector is a log flag. Either layer alone is a hope.

Common mistakes

Keep the jailbreak slice small and ugly: ignore-previous, you-are-now, repeat-the-spec, encoded nonsense. Re-run it on every friendliness PR. A refusal is status=refused plus no forbidden tool, not a TED talk. Teach a short public description of the bot. Put secrets in the environment. Do not print the spec because a poem asked. Code still wins if the poem begs for a tool you never enabled.

How agents use this

Keep a small eval of jailbreaks next to the golden set. When someone “just makes the bot nicer,” re-run it. Teach a short public description of the bot; put secrets in the environment, not in the poem. Hidden CoT is not a defense. Code wins over roleplay.

Tip:Revealing the full system prompt is often a product leak. Teach the model a short public description.