Projects/RAG Customer Support Agent/Part 3

Bag-of-Words Vectors and Cosine Retrieve

Build a vocabulary, vectorize chunks and queries as lists of floats, and retrieve top-k by cosine similarity without NumPy.

Retrieval is geometry. Each chunk becomes a vector. The query becomes a vector in the same space. Cosine similarity is the cosine of the angle: 1 means parallel, 0 orthogonal, negative if you use raw counts with centered vectors (you will use non-negative counts, so scores stay in [0, 1]).

You will not call OpenAI embeddings. You will use bag-of-words: a fixed vocabulary from the handbook (plus a few query words at index time? No — freeze vocab from chunks only; unknown query tokens are ignored). That is realistic: out-of-vocab words should not crash you.

Vocabulary

Tokenize: lowercase, [a-z0-9]+, drop tokens of length 1. Optionally drop a tiny stoplist: the a an of to in is. Build token → index from chunk texts only, sorted for stability.

Vector

vec[i] = count of vocab[i] in the document (term frequency). Binary (0/1) also works for tiny docs. TF is fine. Do not implement full BM25 unless you finish early; cosine TF is the lesson.

Cosine

` dot(a,b) / (||a|| ||b||) `

If either norm is 0, similarity is 0. Never divide by zero. Empty query (all stopwords) retrieves nothing useful — refuse later.

Top-k

Score all chunks (N is tiny). Sort by score descending, then by id for ties. Return k=2 or 3. Always return the score to the generator. Hidden scores cause overconfident answers.

Live Pythonpython

import json
import math
import re

CHUNKS = [
    {"id": "chunk-01", "text": "Refunds. Refunds are issued in 5-7 business days to the original payment method."},
    {"id": "chunk-02", "text": "Rate limits. Each API key may make 60 requests per minute. Burst returns HTTP 429."},
    {"id": "chunk-03", "text": "Privacy. Support will never ask for your password or full API key."},
    {"id": "chunk-04", "text": "Incidents. Sev-1 incidents are acknowledged within 15 minutes."},
    {"id": "chunk-05", "text": "Mug shop. Acme mugs ship in 30 days. Mug refunds take 90 days."},
]
STOP = set("the a an of to in is for your or".split())

def tokenize(text):
    return [t for t in re.findall(r"[a-z0-9]+", text.lower()) if len(t) > 1 and t not in STOP]

def build_vocab(chunks):
    vocab = sorted({tok for c in chunks for tok in tokenize(c["text"])})
    return {tok: i for i, tok in enumerate(vocab)}

def vectorize(text, vocab):
    v = [0.0] * len(vocab)
    for tok in tokenize(text):
        i = vocab.get(tok)
        if i is not None:
            v[i] += 1.0
    return v

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def norm(a):
    return math.sqrt(sum(x * x for x in a))

def cosine(a, b):
    na, nb = norm(a), norm(b)
    if na == 0 or nb == 0:
        return 0.0
    return dot(a, b) / (na * nb)

VOCAB = build_vocab(CHUNKS)
MATRIX = [(c, vectorize(c["text"], VOCAB)) for c in CHUNKS]

def retrieve(query, k=2):
    qv = vectorize(query, VOCAB)
    scored = []
    for chunk, vec in MATRIX:
        scored.append((cosine(qv, vec), chunk["id"], chunk["text"]))
    scored.sort(key=lambda row: (-row[0], row[1]))
    return [
        {"id": cid, "score": round(score, 4), "text": text}
        for score, cid, text in scored[:k]
    ]

for q in [
    "How long do refunds take?",
    "What is the API rate limit?",
    "Do you sell equine dental insurance?",
    "mug refunds",
]:
    print("Q:", q)
    print(json.dumps(retrieve(q), indent=2))
    print("---")
print("vocab size", len(VOCAB))

Output

Run to execute this in your browser. Nothing is sent to a server.

Step-by-step: read the scores

Run the four queries. You should see:

Refund question → chunk-01 high, maybe chunk-05 (mug refunds) as rival. That rivalry is the point. Top-1 must be API refunds for the generic "refunds" query if "5-7" and "business" match; "mug refunds" should flip to chunk-05.
Rate limit → chunk-02.
Equine insurance → low scores (near 0). Remember the max score; part 4 uses tau.
If equine accidentally matches "incidents" because of a shared rare token, your stoplist or handbook needs tightening — or tau saves you.

Why not Euclidean distance

Cosine ignores vector length. A long chunk with many tokens would look far in L2 just for being long. Cosine is the default for text. (Production embeddings are still compared with cosine or dot product after normalization.)

k and tau are different knobs

k is how many chunks the generator may see. tau is whether you generate at all (on the best score, usually). You can retrieve k=3 but still refuse if scores[0] < tau. Do not pass low-score chunks in as if they were facts.

Watch out:Printing only top-1 hides near-ties. Always log top-3 scores in traces.

Exercise

Implement retrieve_with_threshold(query, k, tau) that returns [] if the best score is < tau. Print equine vs refunds at tau=0.2 and tau=0.5. Pick a tau that lets refunds through and equine not. Write the number down for part 4.

Check your understanding

If a query vector is all zeros, what is cosine with any chunk?

1
0 (by the zero-norm rule)
Infinity
The chunk length

Chunk the Handbook

Up next

Answer with Citations