Embedding Geometry
Nearest neighbors in a list of 2-d and 3-d points. RAG is this picture in a few hundred dimensions — plus a cutoff.
An embedding space is just a vector space where you decided that nearby means related. The embedder is a function from text (or images, or tool traces) to a point. Retrieval is nearest neighbor search: given a query point, return the stored points with best cosine (or smallest distance).
You cannot see 1536 dimensions. You can see 2 and 3. The algorithms do not change: same list, same cosine, same sort. Clusters that look obvious in 2-d are the same phenomenon as “all refund FAQs sit in a blob” in high-d. Failures also match: a query on the boundary between two blobs retrieves a mix; a query in empty space retrieves something anyway, because top-k always returns k.
That last point is why RAG needs a score cutoff, not only k. The nearest neighbor of a garbage query is still a neighbor, not a refusal.
A wrong picture
A wrong picture is: “more dimensions always separate topics better, so ranking gets easier.” Extra axes can separate (an infra axis in 3-d). They also make pairwise distances look more similar. That is one reason cosine and normalization matter more, not less, as models get wider.
Another wrong picture: “top-k means the chunks are relevant.” k neighbors are always returned, even if every score is poor. Without a similarity cutoff (or a refusal path), RAG will stuff weak chunks into the prompt. Look at the score.
A third: “improving the LLM fixes a broken neighborhood.” The generator cannot recover a chunk that never made top-k. Evaluate the retriever with recall@k on (question, gold chunk) pairs before you judge the generator.
Mixing embedders, stretching only documents, and averaging across models: all two-space bugs from earlier lessons. Geometry here assumes one space.
The formula in words
For each stored vector, cosine(query, vector). Sort descending. Take k. Optionally drop any with cosine below a threshold.
Tiny numeric. Billing cloud near [0.9, 0.1], auth near [0.1, 0.9]. Query [0.85, 0.15] should pick billing. A vague query in 3-d [0.4, 0.4, 0.4] still gets two “hits” — read the scores; if they are mediocre, refuse.
Query sits in the billing blob. Auth is the other cloud. Nearest neighbor is geometry plus a sort.
Two clusters: billing vs authMoving parts
| Piece | Role |
|---|---|
| Query vector | The question, in the same space as the store. |
| Store | Labeled points (text + vector). |
| Score | Cosine (or dot, if both are length 1). |
k | How many neighbors to keep. Always returns k. |
| Cutoff | Drop neighbors below a similarity. This is the refusal. |
Top-k without a cutoff always returns k points, even if every score is poor. That is geometry, not a product opinion.
A second walkthrough (by hand)
Query q = [0.85, 0.15]. Magnitude sqrt(0.85^2 + 0.15^2) = sqrt(0.745) ≈ 0.863.
Billing b = [0.9, 0.1]. Magnitude sqrt(0.81 + 0.01) = sqrt(0.82) ≈ 0.906.
Dot 0.850.9 + 0.150.1 = 0.765 + 0.015 = 0.780.
Cosine 0.780 / (0.863 * 0.906) ≈ 0.780 / 0.782 ≈ 0.997.
Auth a = [0.1, 0.9]. Same magnitude ≈ 0.906. Dot 0.850.1 + 0.150.9 = 0.085 + 0.135 = 0.220.
Cosine 0.220 / 0.782 ≈ 0.281.
A cutoff of 0.35 keeps billing and drops auth. Top-2 with no cutoff would still return auth as a “hit.” The 0.281 is a neighbor, not a relevant paragraph.
Zero vector: magnitude 0, cosine undefined (divide by zero). Treat as a failed embed. Do not rank it.
Tied scores: two billing FAQs at 0.99. Break the tie with recency or a trusted source. Geometry should not be the only policy.
A Friday ticket
Friday 18:05. A garbage query (“asdf”) stuffed a cafeteria-menu chunk into a refund prompt. Top-2 had done its job: it returned two neighbors. The winning cosine was 0.22. Nobody had logged the score. The generator quoted pizza hours as a refund policy.
They added if score < 0.35: refuse and printed the top-k scores on every retrieve line. The neighborhood did not get smarter. The policy stopped treating a poor neighbor as evidence.
A tiny vector store
Three labeled clouds in 2-d: billing, auth, infra. We embed a query by hand (in production the model does this) and pick top-2. Then we do the same in 3-d to show an extra coordinate can separate what 2-d mixed.
Run to execute this in your browser. Nothing is sent to a server.
The 2-d billing query should print two billing rows with high cosines (near 0.99 and 0.99). The 3-d infra query should print the two infra rows with high cosines (near 0.99). The vague query still prints two “hits.” Look at the score. If both cosines are mediocre (often ~0.7 here, not 0.99), the agent should say “I don’t have this in memory” instead of quoting a random snippet. That check is one if score < 0.35. Geometry plus a cutoff is an API.
(Your vague scores depend on the points; what matters is they are weaker than the on-cloud queries and still returned.)
What goes wrong
- No cutoff: k neighbors always come back. A 0.22 winner is still a winner. Refuse on the score, not on the English.
- Zero / failed embeds: cosine divides by length. Length 0 is undefined. Drop the row. Do not insert it.
- Mixed spaces: query from model A, store from model B. Same dimension is not the same geometry. Scores become noise.
- Ties: two chunks at 0.81. Sort is stable or not depending on the language. Add recency or source trust.
- Empty space: a query in a hole still has a nearest neighbor. High-d makes many distances look similar. Cutoff matters more, not less.
Production logs: top-k scores next to texts, the cutoff, whether you refused, and len(vec) / magnitude on insert. Assert dimension match, no zero vectors, and that a fixture query (billing) out-cosines a fixture distractor (auth). Text without scores is a story.
What RAG actually adds
Real RAG is this neighbor loop plus: chunking (how a document becomes several points), metadata filters (only search label == billing), and a generator that reads the neighbors. Improving the LLM does not fix a broken neighborhood.
Long-running agents store memories as points. If you never decay or cluster them, the store becomes a fog: everything is a weak neighbor of everything. Periodically merge near-duplicates (high cosine to each other) and drop low-magnitude failed embeds.
When you evaluate retrieval, gold chunk should out-cosine the distractors. If it does not, fix chunking or the embedder, not the prompt poem.
How agents use this
On each turn, retrieve a few memories. Print the top-k scores in every retrieve log line. Text without scores is a story; scores are evidence.
- Tokens: retrieved text becomes tokens in the prompt. Cost is extra prompt tokens. Bad neighbors cost tokens and confuse the generator. A refusal (no chunks) is cheaper than four 0.22 chunks.
- Ranking: this lesson is ranking. Filters change the candidate set. Cutoff changes precision/recall (next lesson).
- Loss: train or choose embedders so gold chunks rank above distractors. Do not use generator CE as a substitute for recall@k.
- Sampling: retrieval is greedy given the store. The generator then samples. Failures split cleanly: wrong neighbors vs unlucky decode.
You have the Mathematics track’s toolkit: functions, sums, logs, vectors, dot products, matrices, maps, derivatives, gradients, the chain rule, probability, Bayes, distributions, expectation, entropy, softmax, descent, sampling, cross-entropy, attention, and this geometry. The ML and transformer tracks will reuse every object.
Tip:Print the top-k scores in every retrieve log line. Text without scores is a story; scores are evidence.
Check your understanding