Projects/RAG Customer Support Agent/Part 2

Chunk the Handbook

Split a small handbook into overlapping chunks with stable ids, headings carried into the chunk text, and no empty slices.

Chunking is the unglamorous half of RAG quality. Embeddings cannot retrieve a fact that you split in half or buried in a 4,000-token blob. This part writes a deterministic chunker for a tiny handbook: split on headings, then pack sentences into windows with overlap.

What a chunk must contain

Stable id chunk-01, chunk-02, … zero-padded so lexicographic sort matches order.
text used for embedding and for citation display.
heading copied into the text prefix so "Refunds" context is not lost when a sentence is generic ("within 14 days").
start/end character offsets into the original handbook (debugging).

Do not use a random uuid. Ids must be stable across runs for evals.

Strategy for a tiny doc

Split on lines matching # Heading.
Inside each section, split into sentences on . .
Pack sentences until len(text) >= 180 or the section ends.
Overlap: last sentence of chunk N starts chunk N+1 when the section is long enough.

For Joeven's handbook, each heading may already be one chunk. That is OK. Still write the packer so a longer FAQ later does not become one vector.

What not to do

Chunk by raw text[i:i+200] through the middle of a word — unless you also overlap a lot. Sentence boundaries are cheaper.
One chunk for the whole handbook — cosine still "works" but citations are useless and distractors pollute.
Drop headings. Isolated sentences like "Burst above that returns HTTP 429" need the rate-limit heading.

Live Pythonpython

import json
import re

HANDBOOK = """
# Refunds
Refunds are issued in 5-7 business days to the original payment method. Digital goods are refundable within 14 days of purchase if unused.

# Rate limits
Each API key may make 60 requests per minute. Burst above that returns HTTP 429. Enterprise plans may raise the cap by opening a ticket.

# Privacy
Support will never ask for your password or full API key. Rotate a key if it leaked.

# Incidents
Sev-1 incidents are acknowledged within 15 minutes. Status is posted at status.example.com.

# Mug shop
Acme mugs ship in 30 days. Mug refunds take 90 days. Unrelated to the API product.
"""

def split_sections(doc):
    parts = re.split(r"(?m)^# ", doc.strip())
    sections = []
    for part in parts:
        part = part.strip()
        if not part:
            continue
        title, _, body = part.partition("\n")
        sections.append((title.strip(), body.strip()))
    return sections

def sentences(text):
    bits = re.split(r"(?<=\.)\s+", text.strip())
    return [b.strip() for b in bits if b.strip()]

def chunk_handbook(doc, target=180):
    chunks = []
    n = 0
    for title, body in split_sections(doc):
        sents = sentences(body) or [body]
        buf = []
        size = 0
        for s in sents:
            buf.append(s)
            size += len(s)
            if size >= target:
                n += 1
                text = f"{title}. " + " ".join(buf)
                chunks.append({"id": f"chunk-{n:02d}", "heading": title, "text": text})
                buf = buf[-1:]  # overlap last sentence
                size = len(buf[0]) if buf else 0
        if buf and (not chunks or " ".join(buf) not in chunks[-1]["text"] or chunks[-1]["heading"] != title):
            # flush remainder if it adds a new section or leftover sentences
            leftover = f"{title}. " + " ".join(buf)
            if not chunks or leftover != chunks[-1]["text"]:
                n += 1
                chunks.append({"id": f"chunk-{n:02d}", "heading": title, "text": leftover})
    # dedupe identical texts
    seen = set()
    out = []
    for c in chunks:
        if c["text"] in seen:
            continue
        seen.add(c["text"])
        out.append(c)
    # re-id after dedupe
    for i, c in enumerate(out, start=1):
        c["id"] = f"chunk-{i:02d}"
    return out

CHUNKS = chunk_handbook(HANDBOOK)
for c in CHUNKS:
    print(f"{c['id']} [{c['heading']}] {c['text'][:70]}...")
print("n_chunks", len(CHUNKS))
print(json.dumps(CHUNKS[0], indent=2))

Output

Run to execute this in your browser. Nothing is sent to a server.

Step-by-step quality checks

Print every chunk. Read them. If a refund sentence sits in the mug chunk, fix the splitter.
Assert no empty text.
Assert ids unique and sequential.
Assert each heading appears in at least one chunk.
Keep chunk count small (5–12). If you have 80 chunks on a 1-page doc, your target length is too small.

Overlap tradeoff

Overlap helps questions whose answer straddles a boundary. Too much overlap duplicates vectors and can crowd top-k with the same section. For a handbook, heading-based sections + light overlap is the sweet spot.

Versioning

When the handbook changes, rebuild the index and bump a doc_version integer stored next to chunks. Answers should cite version in traces. You will not build a diff indexer here; just store DOC_VERSION = 1.

Note:Character offsets are optional in the Try it box but useful when you later highlight the source in a UI.

Exercise

Add a sentence to Refunds that is long enough to force two chunks in that section (target=80). Confirm overlap kept the connecting sentence. Confirm mug refunds did not merge into API refunds.

Check your understanding

Why copy the section heading into each chunk's text?

Headings are illegal in RAG
So generic sentences keep their policy context when embedded and retrieved
Cosine requires markdown
To make chunks longer than the handbook

Overview and Architecture

Up next

Bag-of-Words Vectors and Cosine Retrieve