Positions

Attention (next lessons) looks at which tokens are present and how their vectors match. By itself, classic attention does not know where they sat. Without a position signal, “pay the invoice tomorrow” and “tomorrow pay the invoice” are the same bag of rows.

Agents would not know whether the latest tool result is at the end or buried in the middle. Order is not a stylistic extra. Order is how “never delete” at the start can still lose to “delete them” at the end — and how a model can tell those two clauses apart at all.

Positional encodings mark the index. Old models added a learned table (one vector per position) or a sine/cosine pattern. Many modern models use RoPE (rotary embeddings): they rotate query and key vectors by an angle that depends on position, so the dot product cares about relative distance.

You do not need the trigonometry. You need the contract: same tokens, different order, different meaning.

A wrong picture

A wrong picture is: “the model reads left to right like a person, so it must know order.” The mix inside a layer is a set of weighted sums. Without a position mark, the mix is a bag. Causal masking (later) hides the future, which is also an order fact, but it does not by itself encode “this is position 7.”

Another wrong picture is: “a 128k window means 128k equal slots.” Long-context models stretch or interpolate positions. Quality often dips in the middle of a packed window. That is a position-and-attention problem, not a vibe. A later lesson named Lost in the Middle is this fact as packing advice.

A third wrong picture is: “absolute learned tables grow forever.” They cannot grow past the train length without a trick. Relative and rotary schemes generalize further — still not infinitely, still not evenly. “We trained to 4k and you stuffed 100k” is a position story even when the brochure says you may.

How position is added, in words

Absolute learned: a second table, one row per index 0, 1, 2, … Add (or concat) that row to the token embedding. Token pay at index 0 is not the same vector as pay at index 9.

Sinusoidal: a fixed pattern of sines and cosines so nearby indices look related and you can in theory extrapolate. Still an absolute mark, just not a learned table.

RoPE: do not add a position row. Rotate the query and key so that the match score depends on how far apart two positions are. Relative distance becomes geometry. This is the common modern default. You will not code the rotations here. You will remember that distance between tokens starts to matter in the dots.

All three exist to break the bag-of-tokens symmetry. After they are applied, two sequences with the same ids in different orders produce different vectors going into attention.

A tiny example in words

Toy token table:

• pay: [1.0, 0.0]
• now: [0.0, 1.0]

Toy position: index i adds [0.1 i, -0.1 i].

pay now is index 0 then 1. now pay is index 0 then 1 with the words swapped. The vectors will not match. The sums will not match. That is enough to prove order entered the lists of numbers.

Add a position list so order changes the vectors

Lists of numbers only. Print both orders.

Run to execute this in your browser. Nothing is sent to a server.

pay now and now pay print different pairs of lists. same words is False as Python lists, but even if you compared bags of words, same vectors is False because the index changed. The sums differ too. Order is in the numbers.

If pos_vec always returned [0.0, 0.0], the two orders would still have different sequences of rows (pay then now vs now then pay), but a later mix that ignored order could still smash them. The added position is what makes even a careless mix feel the index.

Long context is not even

Absolute learned tables stop at the max index seen in training unless you interpolate. Rotary schemes can be stretched. Stretching is a compromise: you reuse a pattern outside the range it was tuned on. Expect quality to be best near lengths the model actually trained on, worse far past that, and often worse in the middle of a packed window even inside the advertised max.

Positions also interact with recency. Later indices are a different region of the rotation (or a different row of the absolute table). Putting the latest observation at the end is not only human habit. It is a different place in position space.

Mixing two sequences without a separator and without positions is how “the error from step 1” and “the error from step 9” become one blob. Special tokens (last lesson) are separators. Positions are indices. You want both.

How agents use this

Put the goal and the latest observation near the ends (start and last tokens). Do not hide the only relevant chunk at token 40,000 and hope rotation will find it. Recency is partly architecture: later positions are a different region.

When you concatenate a spec, old tools, retrieved chunks, and a new user line, you are choosing positions for every tile. That choice is a product decision. A later lesson on packing the window will turn this into a suitcase policy. Here the rule is already: order is meaning.

Do not shuffle few-shot examples “for variety” unless you measured it. You are moving them through position space. Do not insert a huge blob in the middle “just in case.” You are pushing the gold chunk into the dip.

• Ends: pin policy first; put fresh evidence last.
• Separators: special tokens between sources so two errors do not merge.
• Length: advertised max is not even quality.
• Copy: ids that must be copied exactly should not sit only in the faded middle.
• Debug: “it ignored the spec” is sometimes “the spec’s positions fell off or sat in the dip.”

Watch out:Mixing two sequences without a separator and without positions is how “the error from step 1” and “the error from step 9” become one blob.