Overlap, Offsets, and Chunk Ids

A sliding window overlaps so a sentence cut in half still appears whole in at least one chunk. If chunk 1 ends on “Refunds take 5-” and chunk 2 starts on “7 days,” neither chunk holds the fact. With overlap, one of them contains “5-7 days.” 10–20% overlap is a sane default. You store more text. That is usually cheaper than missing the join between two facts.

Overlap is not a substitute for heading splits. You overlap inside a section that is still too big. You do not overlap two different policies on purpose. Mixing Refunds and Security in the overlap band is how a chimera is born.

Metadata you must keep

A chunk without metadata is a quote you cannot open. Keep at least:

• id — stable, like runbook.md#refunds or runbook.md:bytes:120-340. Stable means the same bytes get the same id until the source checksum changes.
• source — path or URL the UI will open
• heading path — Refunds > Timing so the user sees where they are
• offsets into the original (byte or character start/end) so the UI can highlight
• checksum / version — so you can reindex when bytes change
• parent id — the section or document this snippet came from

If you cannot point a citation back at a byte range, you do not have citations. You have vibes with footnotes. Models love footnotes. UIs that cannot highlight them train users to ignore them.

Ids that include only “chunk 17” die when you re-chunk. After a heading edit, chunk 17 is a different sentence. Cite runbook.md#refunds (and a checksum) so a stale citation can fail loudly: “this highlight no longer matches.”

Do not cut structure

Do not cut a markdown table through the header row. The body rows without column names are numbers looking for a story. Do not cut a function in half if you can keep the signature with the body. Do not cut a numbered procedure between step 2 and step 3 if both steps are the policy.

When a logical unit is larger than your max chunk, split on inner headings or on blank lines, still with overlap, and keep the same parent id on every child. A later get_doc(parent_id) can fetch the full section if the snippet was truncated.

Run to execute this in your browser. Nothing is sent to a server.

Offsets round-trip: the stored slice is the original substring. That is what a highlight in the UI needs. If matches were False, your offsets are a lie and every citation will highlight the wrong sentence.

Classroom ids here are c1, c2 so you can see the window. Production ids should include the source path and a range or a heading slug. Re-run with overlap=0 and watch “5-7” tear. That is the overlap argument in one print.

When the document changes

When a doc changes, re-embed by checksum, not by “the file name looks the same.” Name-stable plus bytes-changed is the usual wiki edit. Drop old chunk ids for that checksum, write new ones, keep the same source URL so links still work.

Keep parent ids so a tiny chunk can fetch the full section. Truncation without get_doc is how models invent the rest of the file. Packing (later) will mark truncated: true. Offsets and parents are how you recover.

Ids that survive re-chunking

A good id is a function of source plus a stable locator, not of “nth window in this run.” Heading slugs work until someone renames the heading. Byte ranges work until someone inserts a paragraph above. The honest pattern is: the id contains source and checksum (or version), and the UI uses offsets for this checksum. If the file moved, the citation fails closed: “this quote was for version abc, the live file is def.” That is better than highlighting the wrong sentence.

Parent ids let you store small children for retrieve and large parents for get_doc. Child text is what you embed. Parent text is what you show when the user clicks “open section.” Do not embed the parent and the child as unrelated vectors without a link; you will retrieve both as near-duplicates and waste the window (MMR later).

Overlap percentage is a storage and duplicate knob. Ten percent on 400-token chunks is 40 tokens of repeated text. On a million chunks that is real disk. Fifty percent overlap is how you pay for embedding twice and then fight duplicates at pack time. Start at 10–20%, measure recall on questions whose gold fact sits on a split, then stop.

This classroom counts characters. Production PDFs may use page boxes, not bytes of a binary file. Pick a locator the UI can actually open. A citation to “page 7” is valid if the viewer supports it. A citation to a byte offset in a binary PDF is not.

When you re-embed because the model changed, locators can stay if bytes stayed. The vector is new; the highlight range is not. Store model name on the vector row, not inside the chunk id, or every model upgrade 404s every old citation.

Common mistakes

• Overlap 50% “to be safe.” You triple storage and retrieve near-duplicates. MMR (later) then has to clean your mess.
• Ids that reset every ingest (uuid() each run) so citations from yesterday 404.
• Offsets in tokens while the UI highlights bytes, or UTF-8 vs UTF-16 confusion. Pick one unit, test a non-ASCII heading.
• Storing text but not offsets, then trying to search the quote in a file that has been edited.

How agents use this

Citations the user can click are a retrieve feature, not a generation feature. The model can emit [1]. Your code must map [1] to an id, then to a byte range, then to a highlight. If any hop is missing, hide the superscript.

When retrieve returns a child chunk, the prompt can include the heading path so the model knows the subject. If the child is too small to answer, a later hop can get_doc the parent. That is still this track: ids and offsets. The choose-when-to-fetch loop is later.

Do not skip metadata because the demo looked fine in a notebook. The notebook did not have to highlight a span in a 40-page PDF.