Chunking

A chunk is the unit you embed, store, retrieve, and cite. The model never sees “the document” unless you fetch the whole file later. It sees chunks. Too big, and you retrieve noise: the refund policy glued to the cafeteria menu. Too small, and you retrieve a sentence that lost its subject: “Never cash.” Never cash what?

Chunking is the unglamorous reason RAG “doesn’t work.” People tune prompts for a week. The gold fact is split across two windows, or buried in a 4,000-token blob the retriever scores as “generally about billing.”

Size

Count tokens in production. A token is a piece of text the model bills and attends to; English prose is often roughly four characters per token, but that is a rumor, not a law. In this classroom, characters stand in so you can see splits without a tokenizer.

Typical prose chunks: about 200–500 tokens. Smaller than a tweet and you lose context. Bigger than a short section and you retrieve a chapter. There is no universal number. Measure recall on your questions (a later lesson). A policy PDF is not a chat log. A runbook is not a table of SKUs.

Code and tables want logical chunks: a function, a table, a list of steps under one heading. A blunt character window will cut a markdown table through the header row. The retrieved fragment then has numbers and no column names. The model will invent columns.

Headings first

Markdown and HTML with outlines should split on headings, then sub-split long sections by size. A chunk that starts at ## Refunds retrieves for refund questions even if the embedding is mediocre, because hybrid search (later) can still match the heading words. A chunk that mixes the end of Refunds with the start of Security is a chimera: one vector that means two policies.

Preserve the heading path in metadata (Runbook > Refunds > Timing). The next lesson stores ids and offsets. This lesson is the split itself: structure before a sliding window.

Window-split only inside a fat section. If ## Architecture is 3,000 tokens, split it with overlap (next lesson). If ## Refunds is 120 tokens, leave it whole.

What you are optimizing

You are optimizing three things at once, and they fight:

• Recall: the gold fact is in some retrieved chunk
• Precision: that chunk is not mostly unrelated sentences
• Citeability: a human can open the source and see the same words

A 2,000-token chunk is easy to recall (the fact is “in there somewhere”) and hard to cite (which paragraph?). A 40-token chunk is easy to cite and easy to orphan. Headings are the cheap way to get all three on real docs.

Inspect 20 random chunks before you trust an index. Read them like a prompt. If you would not want that text next to a customer question, do not embed it. You will see nav crumbs, “click here,” and half tables. That inspection is cheaper than an eval dashboard you never open.

Re-chunk when the document changes

Stale chunks with new headings are a silent retrieval bug. The file name is the same. The checksum is not. Ingest should drop old chunk ids for that source and write new ones. If you only insert, you retrieve both “refunds take 5-7 days” and last year’s “refunds are instant” from the same path.

Parent/child patterns (tiny child chunks that point at a section parent) are an overlap-and-ids topic. The rule starts here: every chunk knows which document and heading it came from, or you cannot fetch the full section when the snippet is too small.

Run to execute this in your browser. Nothing is sent to a server.

The size window cuts Refunds mid-thought. You will retrieve a fragment that mentions INV-* without “never cash,” or “never cash” without refunds. Heading chunks keep each policy intact. Start with headings; window-split only inside a fat section.

Change the window size in the box and watch facts tear. That experiment is the whole lesson: the split is the product.

Lists, code, tables, and languages

Numbered procedures should stay together when the number is the policy (“1. Raise memory. 2. Replay DLQ. 3. Page if still red”). A window that keeps step 1 and 2 but drops 3 retrieves an incomplete runbook. Prefer splitting before the list or after the whole list, not through it.

Fenced code in a markdown doc wants the fence intact. A chunk that starts mid-function with no signature will retrieve for random identifiers inside the body. Keep the heading that names the function in the same chunk, or in metadata.

Tables: keep the header row with every body slice if you must split a long table. Otherwise the model invents column names. If a table is the answer (SKU lists), consider a tool or a structured store, not RAG. That is the previous lesson showing up inside chunking.

Non-English and mixed-language docs: split on the same headings. Do not assume a character budget that works for English works for other scripts; tokens differ. In production, count tokens. In this classroom, still inspect chunks: a “200 character” rule can cut a single sentence in some languages.

Overlap (next lesson) exists because even heading-first splits must cut fat sections. Do not skip headings because you plan to overlap. Overlap is a bandage for length, not a replacement for structure.

Your eval questions should include at least one fact that sits at a heading boundary and one fact in a table. If both fail, you know which splitter to fix. Chunking that only works on the demo FAQ is not done.

Common mistakes

• One chunk per file “to keep context.” You retrieve a novel.
• Fixed 512 characters on a repo of Python. Functions die mid-signature.
• Splitting on . in “5.7 days” and version numbers.
• Never looking at chunks, only at demo questions that happen to fit.

How agents use this

Retrieve returns chunks. Citations point at chunks. Memory stores, if you use them for semantic facts, should store curated chunks or structured fields, not a second copy of a bad split.

When an agent “didn’t read the runbook,” open the chunk that was retrieved. If the fact is not in that chunk, do not rewrite the prompt. Re-chunk. If the fact is in the chunk and the answer ignored it, you have a generation or packing problem (later lessons).

Re-chunk when the doc changes. Inspect 20 random chunks before you trust an index. If you would not want that text in a prompt, do not embed it. The agent loop is later. The library is already lying if the atoms are wrong.