RAG & Memory

Why RAG

Weights are frozen and uncited. RAG fetches snippets you control, then answers from those snippets.

RAG vs a Lookup Tool

Ids, invoices, and tickets are tools. Prose runbooks are RAG. Mixing them makes a slow, fuzzy database.

Ingest and Freshness

The index is only as true as the last successful ingest. Stale chunks quote yesterday with a straight face.

Chunking

Documents are too big for a prompt. Split by headings first, then by size, and remember what you broke.

Overlap, Offsets, and Chunk Ids

Overlap saves split sentences. Stable ids, heading paths, and byte offsets make citations real.

Embeddings and Retrieval

Embed text into vectors, score with cosine similarity, return the nearest chunks. Geometry, not magic.

Scores and Thresholds

Top-1 with cosine 0.12 is a miss, not “the best match.” Print scores. Return no evidence when nothing is close.

Vector Indexes

Brute force is exact and slow. ANN indexes are fast approximations. Know what you are trading.

Metadata Filters and Tenants

Vectors cannot keep tenants apart. Filter by tenant_id (or use per-tenant indexes) before you rank.

Hybrid Search

Keyword search catches IDs and rare tokens. Vectors catch paraphrase. Fuse both.

Reciprocal Rank Fusion

RRF adds 1 / (60 + rank) from each list. It ignores raw score scales, which is why people like it.

Reranking

Retrieve broadly, then score a shortlist with a slower, sharper model. Two stages beat one.

MMR and Coverage

Three paraphrases of the same sentence waste the window. Penalize near-duplicates so the prompt covers sub-questions.

Packing the Context Window

Cap tokens. Drop the lowest scores first. Mark truncated. Do not pour 40 chunks into the next prompt.

Citations and Faithfulness

A quote the user can open is a citation. Fluent sentences that are not in the sources are hallucinations.

Refuse When Nothing Matches

Empty retrieval is a success state. Inventing a procedure because the model wants to help is how cash refunds happen.

Retrieved Text Is Data

A wiki page is an observation, not a new boss. Wrap chunks. Never let a document mint tools or skip policy.

Agentic RAG

The model can retrieve, reformulate, retrieve again, or stop. Retrieval becomes a tool with a budget.

Query Rewrite

The user said “that OOM thing.” The index wants “runner out of memory worker limit DLQ.” Rewrite, then search.

Memory Types

Working, episodic, semantic, and procedural memory are different stores — not one magic vector soup.

Working Memory

Keep goal, budget, and last observation in a small JSON state you rewrite. Do not append the novel forever.

Memory Write-Back

Do not let the model upsert “facts” from a hostile page. Confirm, source, and expire what you store.

Recall@k and RAG Evals

You do not need an LLM to test retrieval. You need gold chunk ids and a number: did they appear in the top k?

When RAG Fails

Fix chunking, filters, and citations before you fine-tune. The next track is agents: loops that call retrieve as one action among many.