Lost in the Middle

Empirically, transformers use the beginning and the end of a long prompt more reliably than the middle. The brochure’s 128k tokens are not 128k equal slots.

So: pin the spec at the front, put the latest observation at the end, and do not hide the only relevant chunk at token 40,000. Recency bias is not a myth you debate. It is a packing rule.

Attention mass dilutes over a long row. Positions in the middle are also a region the model saw less often in training (or interpolates worse). Both stories point at the same engineering move: shorten and order.

You met positions, attention, and the context window. This lesson is those three as a U-shape: first and last matter more. Packing is how agents live with that shape.

A wrong picture

A wrong picture is: “please read everything carefully” flattens the U. It does not. Editing the sequence does. Extra English is more tokens in the same dip.

Another wrong picture is: “if I dump 20 chunks, the gold one will be found because attention can look anywhere.” In principle it can. In practice the gold chunk in slot 17 of 20 is a different system from the same chunk in slot 2 or next to the latest user line. Eval both the text and the position.

A third wrong picture is: “the middle is useless, so delete it always.” The middle can still help. It is just less reliable. Prefer shorter windows. When you must keep a long one, repeat the goal at the end and put gold evidence near an end, not only in the fog.

Why a U

Softmax dilution: one query’s weights sum to 1. More keys, smaller typical weights, more competition. Needles drown in hay.

Position geometry: rotary and absolute schemes are not even across a 100k window. Training lengths cluster. The middle of a packed window is often a worse neighborhood.

Recency and primacy: humans have this too; the architecture adds its own version. You do not need to pick one theory to pack well. You need: ends are louder.

Lost-in-the-middle is documented on retrieval-style tasks: a fact placed in the middle is used less than the same fact placed at the start or end. Your agent traces are retrieval-style tasks. Tool dumps are documents. Pack them like documents.

A tiny example in words

Chunks: spec, old-1, old-2, GOLD, old-3, latest. A toy score that likes the ends (distance to the nearest end) will rank spec and latest above GOLD. Move GOLD next to latest. The toy score rises. Real models are not this cartoon, but the packing move is the same.

Rerank, then place the top chunks at the end (or right after the spec), not in a random dump.

A score that likes ends

Lists of names and numbers. Print ranks before and after the move.

Run to execute this in your browser. Nothing is sent to a server.

Moving the gold chunk toward the end raises its toy score. Repeating spec at the end gives the policy two loud seats. Real models are not this cartoon, but retrieval papers keep finding the same U-shape: first and last matter more.

If you must keep a long window, repeat the goal at the end (“remember: never delete rows”) instead of assuming the opening spec is still loud. Pinning the front and repeating at the end is allowed. Hiding GOLD at index 40,000 is not.

Eval is position-sensitive

Eval retrieval with (query, must-include-chunk) and the position you stuffed it. A gold chunk in slot 17 of 20 is a different system from the same chunk in slot 2. Log the index. When quality drops, check whether a packer started burying gold.

Rerankers exist to pick the few chunks that deserve an end seat. Dumping all 20 “just in case” is how gold ends up in the dip. The n² lesson said selection matters for cost. This lesson says selection matters for being used.

Common packing failures

A “fair” round-robin of twenty equally long chunks. Fair to the packer. Unfair to GOLD. Fairness is not a softmax property.

Putting the spec in the middle because “the user should see tools first.” Tools first means the policy sits in the dip. Pin the policy. Tools can follow. Latest observation last.

A sliding window that keeps the last k messages without counting tokens. Five huge tool dumps can be most of the window. The spec pin is still there, but the gold chunk from retrieval is gone. Count tiles. Cap tool dumps.

Repeating the entire spec five times “for emphasis.” You just spent five copies of the pin and pushed evidence toward the middle. Repeat a short rule at the end, not a novel.

Evaluating retrieval only on “was the chunk in the prompt at all.” In-the-prompt-in-the-middle is not the same system as in-the-prompt-at-the-end. Log the index. Fail a case if GOLD sat in the dip when you claimed the retriever worked.

How agents use this

Eval retrieval with (query, must-include-chunk) and the position you stuffed it. Rerank, then place the top chunks at the end (or right after the spec), not in a random dump.

Put the latest tool observation near the end. Pin the spec at the front. Repeat the non-negotiable rule at the end when the window is long. Do not rely on a middle needle.

• U-shape: start and end are used more reliably.
• Move gold: toward an end after rerank.
• Repeat goal: last tokens can restate the spec.
• Eval position: same chunk, different index, different system.
• Do not prompt the U away: edit the sequence.

Watch out:Adding “please read everything carefully” does not flatten the U-shape. Editing the sequence does.