Byte Pair Encoding

Byte Pair Encoding (BPE) is the usual recipe behind subword tokens. It is not a neural net. It is a compression-style loop that builds a tile list from data, then a fixed procedure that splits new text using that list.

Start from characters (or from raw bytes). Count which two neighbors sit next to each other most often. Glue that pair into a new token. Repeat. After enough merges, ing might be one token, attention might be two, and your company name might be twelve because it never appeared in the training corpus.

The trained tokenizer is a vocab plus a merge list. Encoding a new string means: split to bytes (or characters), then apply the merges in the order they were learned. Decoding joins the pieces back to text. There is no “understanding” in this step. There is only “which pair was glued first.”

If two products use different merge lists, the same sentence becomes different id sequences. Few-shot examples copied from one model onto another are then off-distribution, even if the English looks identical to you.

A wrong picture

A wrong picture is: “BPE reads English and picks words.” It never sees a dictionary. It only sees frequent neighbor pairs in the training bytes. the is a token because those letters sat together often, not because someone labeled it a word.

Another wrong picture is: “once trained, BPE still counts pairs on every new sentence and invents new tiles.” At train time you count and glue. At use time you only apply the saved merge list. A brand-new company name does not get a new row. It gets split into old pieces.

A third wrong picture is: “unknown words crash the model.” Byte-level BPE can always fall back to raw bytes, so there is no true unknown tile. The cost is that ugly strings become many tokens. That is a feature for coverage and a tax for UUIDs, hashes, and base64.

The loop in words

1. Start with a sequence of tiny tiles (characters or bytes).
2. Count adjacent pairs. Find the pair that appears most often.
3. Replace every run of that pair with a new tile name (the two pieces stuck together).
4. Repeat until you have enough merges (often tens of thousands).
5. Save the vocab and the ordered merge list.

To encode later: split the new text the same way you started, then walk the merge list from first merge to last. If a pair is present, glue it. If not, leave it. You are replaying history, not inventing.

To decode: concatenate the tile strings (careful with the special spaces some tokenizers store on the tile). You should get the original bytes back. If you do not, the tokenizer is broken or you stripped a special token you should not have.

Cousins exist. WordPiece scores pieces a bit differently. Unigram starts from a large set and prunes. Same job: a finite table that can still write new words. You do not need to implement them. You need to know they are not interchangeable with your BPE list.

A tiny example in words

Corpus: low lower newest newest as characters, including spaces.

Early merges often glue space-plus-letter or common letter pairs. Frequent pairs like e s or s t show up because newest appears twice. After a few glues, you will see longer chunks in the sequence and a smaller tile count.

That is the whole idea. Real BPE does this on billions of bytes. Your five-step toy is the same loop with a tiny corpus.

Glue the most common pair, five times

This toy counts pairs with a counter, glues the winner, and prints the sequence. It uses lists of characters, not a neural net.

Run to execute this in your browser. Nothing is sent to a server.

Frequent pairs glue first. Watch the length drop. The printed result list is still readable as glued chunks. Real BPE trains many more steps and uses bytes, so you would not recognize every tile as an English syllable. The loop you ran is the idea.

If you change the corpus to one rare name repeated, that name’s letters will glue into a long tile. If the name never repeats, it stays shattered. That is why internal product names tokenize “badly” until they appear often in the tokenizer’s training data — which they usually do not.

Bytes, not letters

Modern BPE is often byte-level. The starting alphabet is 256 byte values, not “English letters.” That means any Unicode string can be encoded. Emoji, Chinese, broken UTF-8 — all become bytes, then merges.

Coverage is complete. Efficiency is not equal. A language or a symbol set that was rare in the merge-training data will need more tiles for the same meaning. A UUID is almost all unique bytes. Merges barely help. Base64 is a pile of almost-random characters. Same tax.

Stop strings interact with this. If you stop generation when a certain English phrase appears, you must think in tiles, not in Python in on the decoded string only. A stop phrase can sit across a tile boundary. Decode, then search, or tokenize the stop string and match ids — but know which one your stack does.

How agents use this

If two stacks tokenize getUserById differently, copied few-shot tool traces will not match. Pin the model and its tokenizer together. Do not mix id sequences from two vocabs.

When a stop string must not appear inside JSON, tokenize that stop string and check. A quote or a brace that is a stop tile will cut the model off mid-argument.

Chinese, code, request ids, and stack traces are token-heavy. A “short” error payload can still blow the budget. Count it. Truncate with a tokenizer, not with text[:500] characters.

Company names, SKUs, and camel-case APIs will split. If the model must copy them exactly, put them in a short, loud place in the prompt (you will see why after positions and attention). If you can rename a tool to something that stays one or two tiles, do it.

• Pin: model, vocab, merge list, chat wrapping — one bundle.
• Ugly strings: UUIDs and hashes are many tiles; summarize or hash them yourself before prompting.
• Stop: define stops that cannot appear in legal tool JSON.
• Names: prefer token-cheap tool names if you control the schema.

Note:WordPiece and Unigram are cousins. Same job: a finite table that can still write new words.