Curriculum/Neural Nets & Transformers
Context Windows
A hard token budget for one forward pass. Pack it like a suitcase. Silent truncation forgets the spec.
A context window is the maximum number of tokens the model can attend over in one forward pass: system + tools + history + this user message + room for the answer.
It is not RAM for your app. It is a hard square of attention. Go past it and the call errors, or a client silently truncates the front of the transcript. Silent truncation is how agents forget the system prompt and keep the latest rant.
Long-context models still want packing discipline. They just fail later and cost more. Attention is still n² in the naive picture. Softmax still dilutes. Positions still treat the middle as a different, often weaker, region. A 128k window is not permission to be lazy.
A wrong picture
A wrong picture is: “the window is pages.” Pages are not the unit. Tokens are. JSON, code, and ids pack worse than English. Count tiles.
Another wrong picture is: “drop the oldest messages; that is FIFO, so it is fair.” Oldest-first often drops message 0, the spec. The model then only sees the latest user line, which might be “delete them actually.” Pin must-have tokens so they cannot fall off.
A third wrong picture is: “if 1+2 do not fit, start anyway and hope.” Do not start the call. Retrieve less. Summarize. Split the task. A truncated policy is a different product.
Packing
- Must-have — policy, schema, goal, current observation
- Useful — retrieved chunks, recent tool results
- Nice — old thoughts, full file dumps
- Dead weight — duplicated stack traces, base64, entire databases
If 1+2 do not fit, do not start the call.
| Policy | Effect |
|---|---|
| Drop oldest messages | Forgets the spec; keeps the latest errors |
| Drop middle | Can keep system + last turn; loses the clue in between |
| Summarize old turns | Loses details; keeps plot |
| Sliding window on tools | Keeps last k observations |
Most amateur loops drop the oldest and therefore drop the system prompt that was message 0. Pin the spec. Put the latest observation at the end (positions lesson, lost-in-the-middle lesson). Reserve output room so generation does not immediately overflow.
Agents die at the limit by looping retries, dumping 40 retrieved chunks, copying whole transcripts between crew members, or stuffing images. Death looks like: looping, ignoring tools, emptying JSON, or cheerfully violating the policy that is no longer in context.
A tiny example in words
Limit 24 toy tiles (words). Spec is pinned: never delete rows; prefer sql. History includes a timeout dump and a last user line “delete them actually.” A packer that keeps the spec and fills remaining space from the end of history should still show the spec at the front. A packer that only keeps the newest messages with no pin would keep the crime and lose the law.
That is the safety feature. Not a sermon. A list of ids that still includes the policy tiles.
Pin the spec, then fill from the end
Word tiles as a stand-in for tokens. Print packed text and whether the spec survived.
Pin the spec. Fill the rest from the end. Oldest-first drop often throws away the policy.
One suitcase: 24 tilesRun to execute this in your browser. Nothing is sent to a server.
The latest user said “delete them.” If the spec fell off the front, the model only sees the crime. Pinning is a safety feature. drop oldest should show the spec missing when the history is fat. pack should keep SPEC: at the front.
Tune LIMIT down until even the pin barely fits. Then 1+2 do not fit. The correct product behavior is refuse or split, not a silent chop.
Measure, reserve, refuse
Budget tokens like money. Reserve output room. Measure prompt tokens every step with the same tokenizer the model uses. When usage is past about 70% of the window, summarize or retrieve, do not hope.
Put encyclopedias in retrieval, not in the system prompt. The spec should be short and loud. The knowledge should be fetched. That is packing plus the embedding-table lesson (do not stuff a library into layer zero every time).
Repeating the goal at the end (“remember: never delete rows”) is allowed when the window is long and the middle is fog. It is not a substitute for pinning the front. Do both when the task is dangerous.
Common mistakes
Counting characters or words, then being surprised the call overflowed. Count tiles with the model’s tokenizer.
Dropping the oldest and dropping the latest tool result to “keep the spec.” You kept the law and lost the evidence. Pin the spec and keep the current observation. Cut the middle dumps.
Starting the call at 99% of the window with max output 1024. Decode has nowhere to go. Reserve output room or the stack truncates the answer, which looks like a stop bug.
Stuffing forty retrieval chunks because the window “has space.” Space is not even quality. You met lost-in-the-middle. Select.
Silent truncation in a client library with no log. The pin is gone. The trace still looks long. Log whether message 0 is still present, every turn.
How agents use this
Budget tokens like money. Reserve output room. Measure prompt tokens every step. When usage is past 70% of the window, summarize or retrieve, do not hope. Long-context models still want the same packing discipline; they just fail later and cost more.
Log window use on every turn: prompt tiles, reserved output, whether the pin is still present. “Forgot the policy” is often truncation, not rebellion.
- Pin: must-have tokens cannot drop.
- Tail: latest observation last.
- Refuse: if must-have plus current obs do not fit, do not call.
- Dead weight: traces, base64, whole tables — cut them.
- 70%: summarize or retrieve before you hit the wall.
Tip:Put the goal and the non-negotiable rules in a pinned block you never truncate. Put encyclopedias in retrieval, not in the system prompt.
Check your understanding