Tokens

A language model does not see letters the way you do. It does not see words the way a dictionary does either. It sees tokens: small pieces of text, each stuck to an integer id. The rest of this track — attention, decoding, adapters — all starts from that list of ids. Cost, context limits, copy bugs, and many “the model cannot spell” complaints start here too.

Think of a token as a tile. Some tiles are whole common words. Some tiles are pieces like ing or ##tion. Some tiles are a single character or even a single byte. The model only knows the tile number. It does not “read English.” It looks up row 17 in a table, then row 4, then row 881, and mixes those rows.

If you skip this lesson, later lessons will sound like magic. They are not. A transformer is a machine that maps a sequence of token ids to scores for the next token id. Everything you type, every tool name, every JSON key, first becomes ids.

A wrong picture

A wrong picture is: “the model reads characters, so length in letters is the budget.” Letters are not the budget. Token count is the budget. A short-looking JSON blob can be many tokens. A long-looking English sentence can be fewer.

Another wrong picture is: “the model reads words, so refunds and refund are the same idea.” In a word table they would be different ids with no shared pieces. Subword tokens are the compromise that lets refund and refunds share a stem tile.

A third wrong picture is: “any tokenizer is fine; they all split text the same way.” They do not. Two models with two tokenizers cannot share embedding rows. Count cost with tokenizer B while you embed with tokenizer A, and the number will lie.

Characters vs words vs tokens

English prose is often about four characters per token. That is a rumor people repeat, not a law. Code, JSON, URLs, and other languages can be much worse. {"job_id": 17} is not “four words.” Quotes, braces, colons, and digits often become their own tiles.

Leading spaces matter. refund (space plus word) and refund (no space) are often different ids. So are Refund and refund. The tokenizer is picky about the exact bytes you send, not the “meaning” you had in mind.

A tiny example in words

Take the string please refund invoice 17.

• As characters, you have every letter and the spaces. That is a long list.
• As words split on spaces, you have four pieces: please, refund, invoice, 17.
• As a toy subword rule “keep words, split digits,” you keep the three English words and break 17 into 1 and 7. That is five tiles.

A real tokenizer is closer to the third story, with a trained list of tiles instead of our toy rule. The number 17 might stay one tile if it was common in training, or split if it was not. You cannot know without that tokenizer.

JSON is worse than English. The characters {, ", _, : each tend to cost. A tool result that dumps a whole table is not “a short message.” It is a pile of punctuation tiles.

Count three ways on the same string

This box does not run a real tokenizer. It shows why the unit you count changes the number. Lists of pieces are the point.

Run to execute this in your browser. Nothing is sent to a server.

You should see more characters than words. The toy pieces sit in the middle: four words, but 17 splits, so five pieces. The JSON string is short to a human and long as a character list because almost every symbol is a separate tile in this toy. Real tokenizers sit between characters and words — and JSON still hurts.

Change 17 to invoice. The toy piece count drops because nothing is digits. Change the text to a UUID-like string such as a1b2c3d4. If you treat the whole thing as one “word,” the toy stays one piece; a real byte-level tokenizer will smash it into many tiles. That is the production surprise.

What an id actually is

After tokenization you do not keep the strings. You keep integers. Token refund might be id 4481 in one vocab and id 902 in another. The model’s first layer is “return row number 4481.” There is no extra English sitting beside the number.

A vocabulary is the finite list of tiles the tokenizer knows. Typical sizes are tens of thousands to a few hundred thousand. Every id you send must be in that range. If you invent an id, you are pointing at a random row or crashing the lookup.

Two models cannot share those rows unless they share the tokenizer (and usually the whole embedding table). Mixing is not “close enough.”

Why the count is never just English

• Whitespace: two spaces, a tab, a newline — different bytes, often different tiles.
• Case: SQL and sql may be different ids.
• Code: names like get_job can split on the underscore into several tiles. getJob camel-case can split at the case change.
• Numbers: long integers and timestamps often split digit by digit or in small groups.
• Other languages: a “short” sentence in a language that was rarer in training can use many more tiles than English of the same meaning.

That last point is fairness and cost together. A user who writes in a language that tokenizes “badly” burns the window faster. The model is not “worse at their language” only because of training data. The budget is also tighter.

How agents use this

Log token counts, not character counts. When you trim a transcript, trim tokens. When you design a tool name or a JSON key, short names are runway, not style. get_job that splits into five tiles is harder to copy faithfully than a name that stays one or two tiles.

Filters that look at “words” miss jailbreaks that split across tokens. A banned string can be one word to you and three tiles to the model, or the other way around. Safety checks that only search the raw string still matter, but they are not the same as “what the model saw.”

When you compare two prompts, compare tokenized length with the same tokenizer the model uses. A “shorter” prompt in characters can be longer in tiles.

Count tool results before you stuff them back into the next prompt. Paste one real tool payload into a tokenizer once. The number will hurt. Then you will stop returning whole SQL tables.

• Budget: every step of an agent loop pays tokens in plus tokens out.
• Window: context is a token suitcase, not a page count.
• Names: tool names and enum values that tokenize cleanly are easier to emit.
• Trim: drop stale tool dumps by token budget, not by “it looks short.”
• Debug: if the model “forgets” a policy, first check whether that policy is still in the token window.

Tip:Paste one real tool result into a tokenizer playground once. The number will hurt. Then you will stop returning whole SQL tables.