The Embedding Table

After tokenization you have integers. Neural nets want vectors: lists of numbers. The embedding table is a matrix with one row per vocabulary item. Token id 17 means “return row 17.”

That is not a metaphor. The first layer of a GPT-style model is an array lookup. Learning embeddings means moving those rows so tokens used in similar contexts sit nearby. You already met that geometry on the math and ML tracks. Here it is tied to ids.

If the table is 50,000 tokens by 768 dimensions, that is already tens of millions of numbers before any attention layer. Tokenizer choice is an architecture choice. Change the vocab and you change which rows exist. You do not “just retokenize” a trained table.

A wrong picture

A wrong picture is: “the embedding is the meaning of the word, stored as text.” It is a list of floats. You cannot read it. You can only compare it, add it, or send it into the next layer.

Another wrong picture is: “two models with 768-d rows live in the same space.” Dimension match is necessary, not sufficient. The axes mean different things. Mixing rows from two tables is how retrieval “gets dumb.”

A third wrong picture is: “one-hot is totally different from an embedding.” A one-hot vector is an embedding: huge, sparse, all at right angles. A learned table is small, dense, and shares structure. That compression is the point. Lookup of a learned row is what production uses.

The lookup in words

Vocab size V. Width d. Table shape V by d. Token id i (an integer from 0 to V-1) returns row i, a list of d numbers.

No multiply is required for the lookup itself. Later layers will multiply. The embedding step is index, copy row.

Training nudges the numbers in those rows. Tokens that appear in similar neighbors move toward each other. refund and invoice might share a direction after training. please might sit elsewhere. Pad should stay a row you mask, not a row you treat as content.

Some models tie the embedding table to the output map (the unembedding you will meet later). Then the same numbers go id to vector and vector to id scores. A tokenizer change is a full retrain, not a config flag.

A production sentence embedder is usually a full transformer whose last hidden states (often pooled) become the vector you store. The table lookup is still the first move inside that embedder. Do not confuse “the embedding table row for one token” with “the vector for a whole chunk.” Agents that store memory usually store the second kind, which still depends on the first.

A tiny example in words

Toy table, width 3:

• pad: [0.0, 0.0, 0.0]
• refund: [0.1, 0.0, 0.9]
• invoice: [0.0, 0.1, 0.8]
• please: [0.5, 0.5, 0.1]
• now: [0.4, 0.6, 0.0]

The sentence please refund invoice becomes ids, then three lists of numbers. Mean-pool those three lists (add, divide by 3) and you get one 3-d summary. That summary is a toy document vector. Real systems may pool, or use the last token, or use a special pool token. The idea is the same: ids became lists of numbers, then we mixed the lists.

Refund and invoice both have a large last slot in this toy. Please does not. A later attention layer (next part of the track) can use that.

Lookup rows, then mean-pool

Lists of numbers only. No extra libraries. Print ids, tokens, rows, and the mean.

Run to execute this in your browser. Nothing is sent to a server.

You should see three ids, three tokens, and three rows. Mean pool prints a 3-number list that sits between please and the two billing-ish rows. Pad is all zeros in this toy. If you accidentally include pad in a mean without masking, you pull the mean toward zero. That is a real bug in clumsy pooling.

Change the sentence to please now. The pool should move toward the first two slots, away from the large third slot that refund and invoice owned. Same table, different ids, different list of numbers.

Size, tying, and mixing

The table is often one of the largest single parameter blocks in a small model. Wider models (bigger d) make every row longer. Bigger vocabs make more rows. Byte-level vocabs that include many specials still have a finite V.

Tied input/output embeddings save parameters and couple “what this token looks like” with “when we predict this token.” Untied maps can be more flexible and more expensive. You do not pick this per request. You pick it when you train.

When you cache vectors for retrieval, you cache a bundle: tokenizer version, model version, and any instruction prefix the embedder prepends. Change any of them and old vectors live in a different space. Mixing spaces looks like “retrieval got dumb.” It got incompatible.

The pad row should be ignored later by an attention mask. If you forget the mask, the model attends to zeros (or to whatever pad trained to be) and gets dumber. Masks are not optional decoration.

How agents use this

When you store memory as vectors, store (tokenizer, model, prefix) next to the floats. If any of those change, rebuild the index. Do not average vectors from two embedders. Same length is not the same space.

Token embeddings (one row per id) are not chunk embeddings (one vector per paragraph). RAG memory is usually chunk embeddings from a full model. Generation still begins with token rows. Both depend on the tokenizer. Both break if you mix versions.

If a tool name tokenizes into five ids, that is five rows mixed by attention before the model can “hold” the name. Prefer names that stay few tiles, as the tokens lesson said. The table cannot invent a row you did not train.

• Lookup: id in, list of numbers out. That is layer zero.
• Cache key: tokenizer + model + prefix, not only the text.
• Pool: do not mean-pool pad.
• Mix: never mix two tables.
• Tie: a vocab change may mean a full retrain.

Note:The pad row should be ignored later by an attention mask. If you forget the mask, the model attends to zeros and gets dumber.