Curriculum/Neural Nets & Transformers
Special Tokens
Pad, end, and chat-role markers are extra rows in the table. Agents live and die by them.
Most tokens stand for pieces of English or code. Special tokens stand for structure: start, stop, pad, “this is the user,” “this is a tool result.” They are extra rows in the embedding table, with ids like every other tile. The model does not know they are “special” except that training kept showing them in the same jobs.
If production injects a token the model never trained on, that row is random noise wearing a hat. If production drops a token the model always saw between roles, you are speaking a dialect the fine-tune never saw.
Chat models were trained on a chat template: a wrapping of special tokens plus punctuation around each turn. The wrapping is part of the language. Skipping it is not a style choice. It is a different language.
A wrong picture
A wrong picture is: “the model reads the words user and assistant in English, so I can write them however I like.” Role markers are usually reserved ids, not the English words. Writing User: as plain text is not the same as the <user> id the template uses (the real strings differ by model).
Another wrong picture is: “pad is just empty space, so attending to it is harmless.” Pad is a real id with a real row. If you forget the mask, the model mixes that row into every token. Generations get dumber in a way that looks like a random seed bug.
A third wrong picture is: “I can strip special tokens from logs to make them pretty, then replay the call from the pretty log.” Replay needs the ids (or a faithful detokenize that puts specials back). Pretty logs that drop role markers cannot rebuild the prompt.
The usual cast
| Token | Job |
|---|---|
| PAD | Fill a batch so rows have equal length; attention should ignore it |
| BOS / start | “A sequence begins” |
| EOS / stop | “I am done generating” |
| UNK | Unknown piece (byte-level BPE often skips this) |
| Role markers | user / assistant / system / tool |
Some stacks add begin-tool and end-tool, or a reserved name for each function. Those ids are part of the contract. A prompt that says “call tools as JSON” in English, without the tokens the model was trained on, is a different language. The model might still emit JSON because pretraining saw JSON. It will not match the tool protocol it was aligned to.
Never type the letters <pad> into a prompt to “save space.” That is English (or a lookalike string), not the pad id. The pad id is for batching. It is not a compression trick.
A tiny example in words
Toy vocab: pad=0, eos=1, user=2, assistant=3, refund=4, now=5.
A user turn “refund now” becomes ids like [2, 4, 5, 1]: role, two content tiles, end. An assistant turn “refund” becomes [3, 4, 1]. Concatenate them and you have a tiny chat.
The model does not “know” roles as English. It knows ids. Swap the user id and the assistant id and you have asked it to continue as the customer.
If you forget eos, the model may keep going into the next role. If you put eos in the middle of a tool argument, you cut the call off. Stop rules and specials are the same family of bug.
Encode two turns as ids
This toy uses a tiny lookup both ways: string to id, id to string. Lists of integers are what a real model would see.
Role markers are extra rows, not the English words “user” and “assistant.” Swap them and you asked it to continue as the customer.
Two chat turns as idsRun to execute this in your browser. Nothing is sent to a server.
The first id list should start with 2 (user) and later include 3 (assistant). The swapped list flips those role ids. Same English words, different structure. That is enough to change the next-token scores in a real model.
Print the tokens back with itos whenever you debug. Humans read strings. The stack reads ids. You need both views.
Templates are not decoration
A chat template decides:
- which specials wrap the system spec
- whether a newline sits before the assistant id
- whether a trailing assistant marker is added so the model continues in the right role
- how tool results are wrapped
If you build a prompt by concatenating "user: " + text and skip the template, you are off-distribution. If you mix two templates across turns, you are also off-distribution. Pin the template to the model version.
When you add a stop sequence, make sure it cannot appear inside a legal tool argument. A stop on } will kill JSON early. A stop on a role marker is common and safer if that marker cannot appear in arguments.
Padding ids in the prompt without a mask means the model attends to pad. That looks like a random bug. It is a mask bug. The padding-and-masks lesson later in this track shows the picture.
How agents use this
Never strip special tokens from logs if you need to replay a call. Store the wrapped prompt or enough structure to rebuild it. Pin the chat template to the model version. When the model updates, re-check the template; silent wrap changes look like “the model got worse.”
If you build a stop sequence, test it against a legal tool call. If the stop fires inside a string field, you will ship truncated JSON and then blame the model.
Do not invent extra specials at runtime. You cannot add a <customer_vip> id and expect a trained row. Put VIP in the text (or in a tool result) using tokens the model already has.
- Replay: keep role markers.
- Pin: template plus tokenizer plus weights.
- Stop: cannot fire inside legal JSON.
- Roles: swapping user and assistant in the wrap is a different request.
- Pad: never generate it; always mask it.
Watch out:Padding ids in the prompt without a mask means the model attends to pad. That looks like a random bug. It is a mask bug.
Check your understanding