When Prompting Is Not Enough
If the model needs a fact, a side effect, or a guarantee, stop decorating the poem. Use a tool, retrieval, or a workflow.
Prompting is how you talk to a next-token machine. It is the OS, the examples, the tags, the injection-as-text, the eval of the poem. It cannot:
- Know whether a fact is true today (need retrieval or a tool — later tracks)
- Charge a card, send an email, merge a PR (need a tool + permissions — later)
- Guarantee a checksum or a total (need a program)
- Remember last month’s ticket if it is not in the window (need memory/RAG, designed — later)
When evals fail after you added the fifth adjective, you are done prompting. Pick the next layer. Do not replace a working workflow with a frontier model because it is fashionable. Do not skip the habits in this track and hope a tool loop will save you. A tool loop injects faster if you still mash data into policy.
Missing facts need a fetch. Side effects need an allowlist.
Stop decorating the poemA map from failure to next layer
| Failure | Next layer | Still a prompt job? |
|---|---|---|
| Wrong format | Smaller contract, JSON mode, constrained decode | Yes — shrink the ask |
| Wrong house label | Few-shot or a cheap classifier | Yes — modes, then stop |
| Missing fact | Tool / retrieval | No — adjectives will not update the world |
| Unsafe side effect | Allowlist + human gate — not a nicer spec | Prompt restates; code enforces |
| Same error every time | Workflow / if-statement | No — “when not to use an LLM” |
| Invented id / amount | Self-check against context; then tools for live data | Check first, fetch second |
| Ignores spec after a page | Encode, dual-channel, repeat contract, evals | Yes, then allowlists |
The next tracks are tools, retrieval, and agents in the runtime sense. They assume you already know how to write a contract, wrap data, and score a prompt. If you skip those habits, extra machinery just moves the blob faster.
Every new skill starts as: can this be a template + if + tool? If yes, ship that. If no, add a prompt with a version, an eval, and a cap.
What “stop decorating” looks like in a week
Monday: model invents today’s CEO. You add “be truthful.” Tuesday: still invents. The move is a search or CRM tool and a citation check, not a sixth adjective. This track taught the check. Later tracks teach the fetch.
Monday: model writes prose instead of JSON. You add two few-shots. Tuesday: parse rate is fine, cost tripled. The move might be JSON mode and a smaller contract, or a template for that route with no model.
Monday: model refunds because a PDF said so. You add “never refund.” Tuesday: it still asks for refund. The move is removing refund from the allowlist, not a warmer spec.
Run to execute this in your browser. Nothing is sent to a server.
What printed: each bug maps to a layer. Unknown maps to measure first. That last line is the real default. If you cannot name the failing cluster, you are not ready for a tool or a longer poem.
The best prompt is often shorter than the one you are proud of. The best agent prompt names tools that exist.
Walkthrough: the fifth adjective
The model invents today’s CEO. You add “be truthful” three times. Eval still fails. The next layer is a search or CRM tool plus a citation check — not a sixth adjective. This track taught the check and the contract. Later tracks teach the fetch.
Prose instead of JSON? Shrink the contract and turn on JSON mode. Refunds because a PDF said so? Remove refund from the allowlist. Same paragraph every ticket? A template plus if may not need a model. Unknown cluster? Measure first.
A tool loop without these habits just injects faster.
What goes wrong if you skip this
You decorate forever. You skip tools that would have been an if-statement. You skip evals because the new model is fashionable. You build retrieval on a mashed prompt and call it progress. Stop decorating when the failure is a missing fact, a side effect, or a guarantee.
Prompting cannot know today’s CEO, charge a card, guarantee a checksum, or remember a ticket that is not in the window. Those jobs need tools, programs, or designed memory — later tracks. This track’s job is to make the OS honest until then: contract, wrap data, score the poem.
Map the failing cluster before you add a layer. Wrong format? Smaller contract. Wrong house label? One shot of that mode. Missing fact? Fetch, do not adjective. Unsafe side effect? Allowlist, not a warmer spec. Same error every time? An if may beat a model.
Every new skill starts as: can this be a template plus a branch plus a tool name that exists? If yes, ship that. If no, add a versioned prompt with an eval and a cap.
Common mistakes
| Failure | Bad next step | Better next step |
|---|---|---|
| Stale CEO | “be truthful” x3 | Search tool + citation check |
| Prose not JSON | More few-shot jokes | Shrink contract, JSON mode |
| PDF ordered refund | Nicer spec | Remove refund from allowlist |
| Always same paragraph | Frontier model | Template workflow |
| Unknown cluster | Random layer | Measure first |
How agents use this
Every new skill starts as: can this be a template + if + tool? If yes, ship that. If no, add a prompt with a version, an eval, and a cap. Facts that change need retrieval or tools — later. Prompting cannot refresh the web. Prompting can keep the OS honest until those layers exist.
Tip:The best prompt is often shorter than the one you are proud of. The best agent prompt names tools that exist.
Check your understanding