Embeddings and Retrieval

An embedding is a list of numbers meant to place similar text nearby in a high-dimensional space. You do not read the numbers. You compare them. For RAG, cosine similarity is the default score: it cares about angle, not vector length, so a long chunk does not win just by being long.

You will not train an embedding model in this box. You will use the geometry. In production you call a vendor (or a local model) that maps a string to a list of floats. The math after that does not change. This classroom uses lists of floats and cosine by hand. No extra numeric libraries. If you can write a dot product, you can debug a retriever.

Retrieval recipe

1. Embed the query → vector q
2. For each chunk vector d, score cosine(q, d)
3. Take top-k (the k highest scores)
4. Drop scores below a threshold, or you retrieve the “least irrelevant” junk (next lesson)

Cosine of two lists a and b of the same length is dot(a, b) / (norm(a) * norm(b)). Dot is the sum of pairwise products. Norm is the square root of the sum of squares (the length of the list as a vector). If either length is 0, cosine is 0 in this classroom — a zero vector is not a match.

If you normalize each vector to length 1 once at ingest (and the query at search time), cosine equals a dot product. Indexes like that. You still print cosine in traces so humans see a familiar scale (about -1 to 1, often 0.1 to 0.9 in practice for text).

What embeddings are good and bad at

Good: paraphrase (“OOM” vs “out of memory”), synonyms, “how do I raise worker RAM” vs a runbook that says “memory limit.” That is why people switched from keyword-only search.

Bad: exact ids (INV-17), rare names, SKUs, stack hashes, and negation (“never refund in cash” vs “refund in cash”). The words sit near each other. The vectors sit near each other. Hybrid search (next part) exists because of this.

Embeddings drift. If you change the model, you must re-embed the corpus. Mixing two models in one index is random retrieval: the same English maps to different axes. Store the model name (and version) next to every vector. Ingest records it. Search refuses to mix.

Similarity is not probability. Cosine 0.81 does not mean “81% true.” It means “pointed a similar direction.” Do not show users a percent unless you calibrated one. You almost certainly did not.

Run to execute this in your browser. Nothing is sent to a server.

If refunds ranked first on the OOM query, your embeddings (or your chunking) are lying. Debugging RAG starts with printing the scores. The toy dimensions here are labeled so you can see why OOM aligns with runner-oom. Real models have hundreds or thousands of unlabeled dimensions. You still print the score. You still check that the top name is the gold chunk on a labeled question.

Unequal list lengths would make zip silently drop extras. Production code should refuse to score two vectors from different models or different sizes. That refusal is an ingest/search contract, not a cosine trick.

Caching and cost

Embedding the query costs a model call (or a local forward pass). Cache query embeddings for identical questions in a session if you want. Do not cache document vectors across embedding-model versions. Do not cache a rewritten query under the raw user string without recording the rewrite (query-rewrite lesson).

Batch document embeddings at ingest. Search is one query vector against many document vectors. The expensive part at ingest is the embedder. The expensive part at search, for small corpora, is still often the embedder plus the prompt, not the cosine loop.

Geometry you can debug without a vendor story

Cosine near 1 means two lists point the same way. Cosine near 0 means they are orthogonal in this space, not that the texts are “unrelated in English.” A runbook and a billing page can still share a small cosine because both mention “customer.” Thresholds (next lesson) exist because the tail is not empty.

Long chunks often have larger raw dots before you divide by length. Cosine’s division is why we use it. If you accidentally rank by dot product on unnormalized vectors, the longest FAQ wins. If you rank by Euclidean distance, you are answering a different question; stick to cosine unless you know why you switched.

Dimension count is the model’s choice. You do not pick 4 labeled axes in production. You still write zip and refuse mismatched lengths. A 768-d vector next to a 1024-d vector is not “close enough.” It is a bug.

Prefixes: some embedding APIs want a query prefix versus a document prefix. If you embed chunks as documents and questions as queries, store that in the model-version string. Mixing prefixes looks like a “bad model” in dashboards. It is a contract bug.

Query embedding caches: cache the exact string you embedded, including rewrite. Two users saying “OOM” after rewrite may share a vector. Do not cache across model versions. Do not cache a tenant-specific query under a global key.

If refunds rank first on an OOM query in production, print the top 10 ids and scores before you fine-tune anything. Nine times out of ten you will see a chunking chimera or a mixed model, not a need for a new vendor.

Common mistakes

• Treating cosine as “confidence the answer is true.”
• Mixing models in one table because “they’re both 768 dimensions.”
• Embedding the user question with a different prefix than the chunks (some models want query: vs passage:). If you use a prefix, store it in the model-version string.
• Hoping embeddings will pin INV-17. They will not reliably. Hybrid search will.

How agents use this

Normalize query and document vectors, then cosine is a dot product. Always log top scores next to chunk ids. When the agent “hallucinates a runbook,” the first check is: was the runbook vector even in the top-k? If cosine to gold is 0.2 and pizza is 0.19, you do not have a generation problem yet.

Retrieve is a library call: text in, ranked chunks out. The later agent loop will call that library. If the geometry is wrong, the loop cannot save you.

Print scores. Name the model. Re-embed when the model changes. That is the whole operational surface of embeddings.