Embeddings

An embedding is a list of numbers that stands in for a thing — a word, a sentence, a ticket, a tool description — so that geometry approximates meaning.

“Cat” and “kitten” should be closer than “cat” and “invoice.” Closer usually means cosine similarity: the cosine of the angle between two lists. It ignores length, which is useful because some models encode frequency in the length and you often do not want that.

You do not need to understand every dimension. You need:

1. The same encoder for queries and documents (or a trained pair)
2. A similarity function
3. An eval, because leftover error is real

This page stays on lists of floats, cosine, and retrieval. It does not teach how a language model builds those lists. You can ship RAG without that. You cannot ship RAG without measuring neighbors.

From id to vector

In RAG, each chunk of text is sent through a model once and stored. At query time you embed the query and take nearest neighbors. Until then, we can fake a tiny table and still practice the math agents run all day.

A vector here means a Python list of floats. Same length for every item in the table. If lengths differ, you cannot zip them into a score. Mixing two models in one index is how you get random retrieval: the slots do not mean the same thing.

If vectors are normalized to length 1, cosine and dot product are the same. Production bugs happen when you train with one score and search with another, or you normalize only one side.

Many embedding models expect prefixes: query: ... versus passage: .... If you embed both sides the same way, neighbors get worse in a way that looks like a “bad index.” Read the model card. If you change the model, re-embed the corpus. Partial re-embeds leave old geometry next to new geometry.

Run to execute this in your browser. Nothing is sent to a server.

What printed: two retrievals. Query cat pulls kitten then dog (animals sit in nearby lists). Query invoice pulls refund then sql (billing-ish lists). That is retrieval. Scale the table to millions of chunks and you have a vector index. The geometry did not change: closest lists first.

Cosine 0.82 is not 82% true. It is an angle. Calibrate with labels if you need a percent. For ranking, sort by cosine and measure P@k.

If a list is all zeros, norm is 0 and we return cosine 0. Empty embeddings are a real bug (blank chunk, failed encode). Do not let them silently win or silently vanish without a log.

What goes wrong in production

Agents fail at retrieval in geometric ways:

• Query is 20 tokens of chat fluff; the chunk is a table
• You embed the latest sentence but the question was two turns ago
• Top-k is too small
• You dump 50 chunks and the LLM attends to the wrong one (similarity is not usefulness)
• You embed with model A and search an index built with model B
• You skip prefixes the model card required
• You never re-embed after the corpus changed (drift of documents, not of user language)

Usefulness is not cosine. A chunk can be close in topic and still be the outdated policy. Ranking metrics with gold ids catch that. Cosine alone cannot.

A routing classifier can be “embed the utterance, embed each tool doc, pick max cosine.” Still eval it with a confusion matrix. Geometry is the feature. The decision still needs a split and a dummy.

Normalization and length

Length of the list (how many slots) is dimension. Length of the vector (the Euclidean norm) is a different word. Say norm for the size of the list as a geometric object. High-norm embeddings can dominate dot product search. Cosine divides it out. If your index uses inner product, normalize on write and on query, or you are ranking by a mix of meaning and magnitude.

Do not hunt for a story in slot 17. Individual slots of a trained embedding are usually not human features. The list as a whole is the feature.

Same encoder, same recipe, measured neighbors

Chunking is part of the embedding. A 50-token chunk and a 2000-token chunk of the same policy will not sit in the same place, and the query may match the wrong one. Overlap between chunks, headings stuffed as prefixes, and whether you embed the table as text are all features of the list you store. Change the chunker, re-embed, re-run P@k. Do not compare neighbors across recipes.

Query-side recipes matter as much. “Embed the last user sentence” vs “embed a rewritten standalone question” vs “embed the last three turns.” Each is a different list. Few-shot the rewriter if you must, but then the rewriter is part of the retriever and belongs in the freeze.

Dimension (how many slots) is not quality. A 256-slot list from a model trained for your domain can beat a 1536-slot list from a generic encoder. You will not see that without an eval of (query, must-include-chunk) pairs. Bigger lists also cost more RAM in the index. Pick from P@k and latency, not from a brochure.

When two items are near-duplicates, cosine will be high and ranking metrics can look fine while the generator still cites the stale copy. Dedup the corpus. Geometry does not know which PDF is in force. That is metadata you must store next to the list: version, date, product.

Common mistakes

• One index, two encoders.
• Treating cosine as a probability.
• Embedding the whole transcript when the question is one clause.
• Never measuring P@k, only “it found something.”
• Forgetting to re-embed after a model bump.

How agents use this

Memory is a table of lists. Retrieval is a score plus a sort plus a k. Routing can be the same table of tool docs. Logging neighbors and scores is how you debug. When the agent cites shipping, print the ranked names. If shipping won, you have a geometry or chunking bug. If refund won and the generator still cited shipping, you have a generator bug. Those two fixes are not the same prompt.

Watch out:Cosine 0.82 is not 82% true. It is an angle. Calibrate with labels.