Packing the Context Window

Retrieved text competes with instructions, the user ask, and (later) tool results and chat history. A 2 MB dump wrecks the bill and the answer. The model’s context window is finite. Attention also dilutes over a long row. Packing is a product choice, not an accident of k=20.

Pack with numbers:

• Max characters (classroom) or tokens (production) for all chunks together
• Keep higher scores first (after hybrid, RRF, rerank, MMR)
• If you cut a chunk, mark truncated: true
• Put sources in a labeled data block, not in the system prompt

The Prompting track already said instructions vs data. Here you build that data block. The next part will treat the block as untrusted. Packing is how it stays small enough to wrap.

Drop low scores before you truncate high ones

Order by score descending. Fill the budget. If the next chunk does not fit, you may truncate it or skip it. Truncating a high-score policy mid-sentence is sometimes worse than skipping a low-score extra. A simple policy: never truncate below a minimum span (for example 80 characters) — skip instead. Always set truncated when you cut.

Low-score pizza should be last and should fall off. If pizza is first, your scores are the bug, not packing.

History vs retrieval budget

Budget retrieval separately from chat history. If working memory is already huge, retrieve fewer chunks. Do not keep k=20 because a paper did. The window is shared.

Truncation without a get_doc(id) follow-up is how models invent the rest of the file. They complete the sentence in the style of a runbook. Mark truncated, and either fetch the parent section or refuse to use that chunk as the only evidence for a precise procedure.

Run to execute this in your browser. Nothing is sent to a server.

Low-score pizza is last. With a tight LIMIT, a and b fill the window; c is cut or dropped. used never exceeds 80. trunc=True is a signal to the rest of the system, not a comment for humans only. Raise LIMIT and watch pizza sneak in. That is why k and LIMIT are two knobs: retrieve broadly, pack narrowly.

In production, count tokens with the same tokenizer the model uses. Character caps are a classroom stand-in. A JSON wrapper around each chunk also costs tokens — budget the fences, ids, and headings, not only the body text.

Where the block lives

Assemble: instructions (your policy) + user question + DATA ... chunks ... END DATA. Chunks do not go in the system prompt as new policy. The wrapping lesson is next part. Packing decides which bytes sit between the fences.

Keep source ids in the block so citations can point. If you pack text and drop ids to save 20 tokens, you cannot cite.

Budgets, fences, and truncation policy

Count the fences, headings, and ids in the budget, not only body text. A JSON wrapper per chunk can double the cost of short snippets. Production tokenizers disagree with “four characters per token.” Use the model’s tokenizer when you have it. Characters are the classroom stand-in.

Policy for a chunk that does not fit: (1) skip if remaining room is below a minimum span; (2) otherwise truncate and set truncated: true; (3) never truncate away the only identifier in a high-score hit if you can skip a later pizza chunk instead. Skipping low-score extras is better than cutting the gold policy mid-sentence.

History vs retrieval: if the assembler already spent 60% of the window on chat, retrieve n=2, not n=20. Working memory should hold a budget field for “evidence tokens left.” That field is packing, not the agent loop.

The data block is labeled. Packing decides bytes inside the label. Putting packed chunks in the system prompt “so they are followed” undoes the wrap lesson. Keep policy in instructions and evidence in DATA.

get_doc is how truncated=true becomes honest: fetch the parent section if the user (or a later hop) needs the rest. Without it, the model completes the runbook in folklore style.

Common mistakes

• k=20, no character cap, history of 30 turns, then surprise latency.
• Truncating without truncated.
• Packing by original retrieve order instead of score (pizza first if the wiki listed menu first).
• Spending half the budget on near-duplicates because MMR never ran.

How agents use this

Budget retrieval separately from chat history. If working memory is already huge, retrieve fewer chunks. Truncation without get_doc(id) is how models invent the rest of the file.

The retrieve library should return packed evidence, not a raw pile, when the caller is a prompt assembler. Or the assembler packs. Pick one place. If both pack, you double-drop gold.

This is still the library: chunks in, a bounded data block out. The agent loop will pass that block as an observation. If you pour 40 chunks, the loop’s next thought is expensive and fuzzy. Pack here.

Log used_chars, limit, each id, and truncated. On-call should see that gold was retrieved at score 0.88 and then cut to twenty characters. That is a packing incident, not a “dumb model.” Keep ids even when you shorten text. Put the block in DATA, not in the system prompt. If history already ate the window, lower n before you lower tau — empty evidence with refuse is better than a truncated rumor next to a half policy.

Skip a low-score extra before you cut a high-score policy below a minimum span. Count fence tokens in the budget. Character caps in this classroom stand in for the model tokenizer. The window is shared; retrieval does not get a blank check because k was 20 in a paper. Mark truncated. Offer get_doc for the parent. Do not pour forty chunks into the next prompt.