Vectors

A vector is an ordered list of numbers. In this track it is a Python list of floats, like [0.2, -1.1, 3.0]. Dimension is len(v). Two vectors can be added only if they have the same length.

In 2-d, a vector is an arrow on the plane. In 3-d, an arrow in space. A 1536-d embedding is an arrow you cannot draw — but you still add, scale, and measure length with the same formulas.

The numbers inside are components. Changing one component moves you along one axis. Later, a partial derivative will be “what happens if I move only this component.”

Vendors return a list of 384, 768, or 1536 floats for a string. That list is the vector. “Embedding” is a name for “vector that stands for text (or an image, or a tool).” There is no extra magic object behind the JSON.

A wrong picture

A wrong picture is: “an embedding is a paragraph stored in a clever way, still basically text.” It is not text. After the embedder runs, you have floats. Similarity, clustering, and retrieval are arithmetic on those floats. If you mix two embedders, you mix two spaces. Same length is not the same space: a 768-d list from model A and a 768-d list from model B are not comparable.

Another wrong picture is: “length of the vector is how important the text is.” Length (magnitude) is the size of the arrow, not the quality of the document. A long chunk can produce a longer vector for boring reasons. Cosine will ignore length on purpose. Distance will not. Know which score you use.

A third wrong picture is treating a zero vector as “similarity 0.” Zero has no direction. You cannot normalize it. Cosine is undefined (you would divide by zero). Empty chunks and failed embedding calls produce zeros. Treat length 0 as an error, not as a neighbor.

The formula in words

• Add: slot by slot. [a, b] + [c, d] = [a+c, b+d]. Picture: walk the first arrow, then the second.
• Scale: multiply every slot by the same number. 3 * [1, -2] = [3, -6]. Negative scale reverses direction. This is how you take a step in training: new point = old point + (step size) times a direction.
• Magnitude (length, L2 norm): square each slot, add, take the square root. In 2-d that is Pythagoras.
• Normalize: divide a nonzero vector by its length. Same direction, length 1. A unit vector has length 1. Cosine will need that.
• Distance between two points: length of their difference. Difference is add, with a scale of -1 on the second vector.

If lengths differ, stop. Raise an error. Silent zip that truncates the longer list is a production bug.

A tiny example

a = [3, 4]. Magnitude is sqrt(9+16) = 5. That is the 3-4-5 triangle. 2 * a = [6, 8]. Unit a is [3/5, 4/5] = [0.6, 0.8], length 1.

b = [1, -2]. a + b = [4, 2]. Difference a - b = [2, 6]. Distance is sqrt(4+36) = sqrt(40) ≈ 6.325.

In embedding land, “the user’s question” and “the refund FAQ” are two lists. Distance small or cosine high means “treat as related.” You will score that in the next lesson. Here you only need: they are lists you can add, scale, and measure.

Add and scale

Vector addition is one component at a time. Scalar multiplication stretches or flips. Magnitude is the square root of the sum of squares.

Run to execute this in your browser. Nothing is sent to a server.

You should see a + b as [4, 2], 2 * a as [6, 8], |a| as 5.0, unit a as [0.6, 0.8], and the length of unit a as 1.0 (tiny float noise is fine). Distance prints about 6.325. [3, 4] has length 5. Distance between two points is the length of their difference.

Try normalize([0, 0]) in your head: magnitude 0, the function raises. That is the correct behavior for a failed embed. Catch it at the boundary: do not insert a zero into the memory bank.

When people average embeddings (one vector for a whole document from its chunks), they add then scale by 1/n. That is valid only if the vectors live in the same space. They do not, if you mix models.

What embeddings actually are

An embedder is a function from text to a list of floats. Same text, same model, same list (or close, if the vendor adds tiny noise). Different models, different lists. Changing a word moves the point. Retrieval is nearest neighbor search in that cloud of points.

Dimension must match the index you search. Putting a 1536-d query into a 768-d index is a shape error. Print len(vec) in retriever tests. Shape is the first test. Magnitude is the second: a typical unit embedding has length 1 (if the API normalizes) or some stable range (if it does not). A magnitude of 0 is a failure. A magnitude of 1e6 is also a failure.

You cannot read a 1536-d list as a sentence. You can still compare two lists. That is the whole point.

How agents use this

Agent memory is often “keep the last k embedding vectors and the text that made them.” Logging only the text hides geometry bugs (wrong model, mixed 768-d with 1536-d, forgot to normalize). Print len(vec) and magnitude(vec) in retriever tests. Shape and length are the first tests of RAG.

• Tokens vs vectors: tokens are discrete ids. Embeddings are continuous lists. A sentence is both: a list of token ids for the generator, and one (or several chunk) vectors for retrieval.
• Ranking: next lesson turns two vectors into one score. This lesson’s job is to keep those vectors well-formed.
• Training: weights are a giant vector. A downhill step is add, with a negative scale of the gradient. Same two operations.
• Averaging memories: add, then scale by 1/n. Only inside one model’s space.

Never add vectors from two different embedding models. Same length is not the same space. If you concatenate hand-made features (latency, token count) onto an embedding, you have a new space: scale those extra slots or milliseconds will dominate cosine.

Zero vectors: drop them. Do not search them. Do not average them into a document vector (they pull the mean toward the origin for no semantic reason).

Watch out:Never add vectors from two different embedding models. Same length is not the same space.