Linear Maps

A linear map is a function T(x) = W x that sends vectors to vectors and respects addition and scaling: T(u+v) = T(u)+T(v) and T(k v) = k T(v). Every matrix defines one.

You already multiply. This lesson is what it does to geometry, which is how you debug feature transforms.

Why agents care: you constantly build lists like [latency_ms, cosine] or you stretch embedding axes without meaning to. A linear map can scale, stretch, or rotate that space. Cosine after a bad stretch is a different geometry. Queries and documents must live in the same geometry.

A wrong picture

A wrong picture is: “stretching a space does not change angles, so cosine is safe.” Uniform scale (the same factor on every axis) keeps directions. Stretch along one axis only does change angles. Neighbors that were close can separate. Cosine after a bad stretch is not cosine before.

A worse picture: apply a stretch W to documents and leave the query raw. Then you compare two different coordinate systems and call it relevance. Similarity assumes one space. A linear map must be applied to every vector you compare, or to none.

Another wrong picture is: “one linear map can fold space into any shape.” It cannot. Linear maps send grids to parallelograms. They cannot fold. Bends like ReLU come after an affine step. A stack with bends can. One W cannot. That is why a single linear classifier is a plane in feature space, not a squiggle.

The formula in words

T(x) = W x. Linear: additivity and scaling as above. Origin stays at origin: T(0) = 0.

Uniform scale by s is the diagonal matrix [[s, 0], [0, s]]. Every arrow grows by s. Angles stay. Lengths change. Cosine of two scaled vectors matches cosine of the originals (the s cancels).

Stretch along x only is [[s, 0], [0, 1]]. The unit square becomes a rectangle. Angles change. Cosine after stretch is a new score.

Rotation turns arrows without changing length. Rotations keep dot products. A 90 degree rotation of [1, 0] is [0, 1] (with the usual convention).

Affine: T(x) = W x + b. Bias moves the origin. Feature transforms in classical ML (standardize columns) are linear or affine maps you choose. In deep models, W is learned. The picture does not change: data is moved so that a later dot product or chance becomes more useful.

A tiny example

Point p = [1, 0]. Scale by 2 on both axes: [2, 0]. Stretch x by 3: [3, 0]. Rotate 90 degrees: about [0, 1].

Nasty stretch sx=0.1, sy=10 on a document [0.6, 0.8] yields [0.06, 8.0]. The y-axis eats the space. If the query stays [1, 0] unstretched, you are matching a nearly vertical arrow against a horizontal one. Ranking becomes meaningless.

That is also feature scaling: one coordinate was in dollars, one was a 0–1 score; you rescale so cosine is not eaten by dollars. Milliseconds vs cosine is the same bug.

Scale and rotate in 2-d

Run to execute this in your browser. Nothing is sent to a server.

Read the prints. Scale 2,2 gives [2.0, 0.0]. Stretch x*3 gives [3.0, 0.0]. Rotate 90 gives about [0.0, 1.0] (the tiny leftover is float cosine of pi/2). Raw doc is [0.6, 0.8] next to query [1, 0]. After the nasty stretch, the doc is [0.06, 8.0] and the stretched query is [0.1, 0.0]. The y-axis ate the space. Neighbors that were close can separate.

Applying W to the document but not the query is worse: you compare two different coordinate systems. The last print line applies W to both — that is the legal way to stretch, and it still changes angles relative to the unstretched world. If your index was built unstretched, you must not stretch only at query time.

Affine maps and features

T(x) = W x + b is affine: linear plus a shift. Bias moves the origin. ReLU and other bends come after this, which is why a network can be more than one stretch.

Standardize a column: subtract the mean, divide by the spread. That is affine (a scale and a shift per axis). Do it with training-set means, then apply the same numbers to queries. Fitting means on the query batch leaks; more important here: different means for docs vs queries is another two-space bug.

When a vendor says they fine-tune a projection on frozen embeddings, they are learning a small W so that cosine in the projected space matches your labels better. You now know what object they are learning. Apply it to the whole corpus and to every query.

How agents use this

Agents featurize constantly: bag of tool-name flags, token counts, cosine scores, latency. If you concatenate [latency_ms, cosine] and then nearest-neighbor in that plane, milliseconds will dominate unless you scale. A diagonal scale (or divide each column by its spread) is the fix. The same bug appears when mixing embedding dimensions with hand-made features in one list.

• Tokens: token counts as a raw feature will dwarf 0–1 flags. Scale or keep them in a separate head.
• Ranking: same W on queries and documents. Rebuild the index after you change W. An old index plus a new projection is two spaces.
• Loss: the projection is trained with a ranking or classification loss on your labels. If labels are “which chunk was useful,” cosine in the new space should lift those chunks. If labels are noisy, W will still fit the noise.
• Sampling: linear maps are deterministic. They do not replace temperature. They change the space you retrieve in, before the generator samples tokens.

If retrieval quality collapses after a “simple preprocess” step, print two or three points before and after W and check whether angles survived. Compare cosine(query, doc) before and after. If the ranking flips on examples you trust, the map is not a no-op.

Tip:If retrieval quality collapses after a “simple preprocess” step, print two or three points before and after W and check whether angles survived.