Which Examples to Keep
Cover the modes of the task. Keep the weird cases. Drop near-duplicates. One bad shot teaches a bad Tuesday.
You do not need more shots. You need representative shots.
A mode is a kind of input the model must handle: happy path, empty, huge, other language, “I do not know,” hostile, mixed intent, the ticket that paged you last month. If a mode never appears in the examples, instructions alone often fail it. The model copies the distribution it was shown. Five polite refunds teach polite refunds.
Count modes, not examples. Six shots and two modes is a short story, not a suite.
Four shots, four modes. Five polite refunds would still be one bar.
Count modes, not copiesKeep this set (redacted)
- One clean happy path per label or format — not five paraphrases of the same refund
- Empty string / “n/a” / whitespace — so the contract still emits
need_clarificationinstead of inventing a job - A long dump — so the model still emits the contract, not a summary essay
- One refusal in the legal output shape
- One mixed-language or broken-English ticket if that is production
- The incident that paged you last month, redacted, if it is a real mode and not a one-off circus
That list is a ceiling, not a homework assignment. If zero-shot already passes the frozen eval, you may keep zero shots (next lesson). Shots you cannot name a mode for are decoration.
Drop this set
- Five paraphrases of the same happy refund
- Examples that invent citations or invoice ids
- Examples that call tools you deleted (
shellafter you removedshell) - Jokes as the last shot (recency copies the last shape)
- Full incident channels pasted as “context examples”
- Anything with a real customer name, card, or secret
- An example whose output would fail your current parser
Leftover few-shots that call shell after you removed shell are how you get hallucinated tools. The model is not being creative. It is completing the pattern you left in the window.
Order is selection
Put a legal JSON example last if JSON is the product. The model copies the last shape. If the last shot is a poem “for fun,” Tuesday’s production reply will try to be a poem.
If you include a wrong output (negative shots, next lesson), it must not be last, and it must be labeled WRONG:. Recency should end on the legal object.
When evals fail one cluster — empty input, injection, huge HTML — add one shot of that mode and re-run. Do not paste the whole incident channel. One mode, one pair, one eval case. If the score does not move, the shot was the wrong mode, or the failure is not a prompting problem.
Run to execute this in your browser. Nothing is sent to a server.
What printed: weak is missing empty, long, and refuse — three shots, one mode. Strong is missing nothing. Weak count 3 modes ['happy']. Strong count 4 modes covering the required set. Production is not three happy refunds.
A selection checklist you can put in the PR
Before you add a shot, answer in the PR body:
- Which mode is this?
- Which eval case did it fail?
- Is the output legal under today’s contract?
- Is the last shot still a legal object?
- Did you redact ids?
- Did you remove tools that no longer exist?
If you cannot answer those, you are collecting folklore.
Walkthrough: five refunds, zero empties
The suite starts failing on empty input: the model invents job 17. You add two more refund paraphrases because “more examples should help.” The empty cluster does not move. You needed one empty shot that still emits need_clarification, and you needed it not last if the last shot must be a legal happy-path object. You also needed to drop the paraphrases: they were one mode counted five times.
A second incident: leftover few-shot still calls shell after billing disabled shell. The model emits shell on a timeout. The parser errors. The retry still wants shell. The PR that removed the tool did not grep the shot file. Selection includes deletion.
What goes wrong if you skip this
The gallery grows, cost grows, modes stay two. Recency copies a joke. Deleted tools haunt the loop. Eval clusters do not match shots, so you never learn which pair mattered. Count modes in the PR or you are not selecting. You are hoarding.
Selection is a checklist, not a feeling. Name the mode. Name the failing eval. Confirm the output is legal under today’s contract. Confirm the last shot is still a legal object. Confirm ids are redacted. Confirm deleted tools are gone. If you cannot answer those, you are collecting folklore.
Order is selection too. Legal JSON last if JSON is the product. A WRONG line, if present, is never last. When a cluster fails, add one shot of that mode, not the incident channel.
Common mistakes
| Keep? | Example | Why |
|---|---|---|
| Yes | One empty → need_clarification | Mode |
| Yes | One redacted outage | Production |
| No | Five refund paraphrases | One mode |
| No | Shot that calls deleted shell | Hallucinated tool |
| No | Joke last | Recency copies it |
Count modes in the PR body. If you cannot name six modes, you do not need six shots. If you have six shots and two modes, delete four. After you add a shot, re-run the whole suite, not only the cluster you care about. Recency and token cost mean a new pair can break JSON on the happy path. Selection is finished when the missing-mode list is empty and the last shot is still legal.
How agents use this
When evals fail one cluster, add one shot of that mode and re-run. Leftover shots that mention deleted tools are bugs in the prompt, not in the model. Count modes, not examples. Put the legal shape last.
Tip:Six shots and two modes is a short story, not a suite.
Check your understanding