Why RAG

RAG (Retrieval-Augmented Generation) is a workflow, not a magic database and not a smarter model. You keep a corpus of documents you own. At question time you fetch a few snippets from that corpus, put those snippets in the prompt as data, and ask the model to answer from that data. Then you cite which snippet supported which claim.

The five steps are always the same, even when vendors wrap them in a “knowledge base” product:

1. Turn the user question into a query (sometimes rewritten; later lessons)
2. Retrieve a few snippets from a corpus you own
3. Put those snippets in the prompt as data, not as new commands
4. Generate an answer tied to that data
5. Cite which snippet supported which claim

You do this because model weights are a blurry encyclopedia with no update button. A weight is a number the model learned during training. After training, those numbers are frozen until you train again. The model can sound current. It is not. It can sound like it read your runbook. It did not, unless you put the runbook in the prompt.

What a closed-book model cannot do

A closed-book answer is an answer from weights alone: no retrieve, no tools, no files. Closed-book is fine for “who wrote Hamlet” if you can live with a cutoff and no citation. It is not fine for “what do we do when the runner OOMs” if that procedure lives in runbook.md on your wiki.

Four problems show up the first week you ship a support bot on weights alone:

A knowledge cutoff is the last date the training set covers. Shipping a larger model does not move that date. Fine-tuning on your wiki can memorize procedures, but it is slow, expensive, and still uncited. It also tends to leak: training on private tickets is a legal and security choice, not a retrieval choice. RAG keeps the private text in a store you authorize per request.

Hallucination here means a fluent claim that is not supported by the evidence you promised. Bigger models still hallucinate. They hallucinate more politely. RAG does not make hallucination impossible. It makes hallucination checkable: you can ask whether the claim is a substring or a fair paraphrase of a retrieved chunk.

RAG is not cheaper intelligence. Retrieval costs tokens, index storage, and an ingest job. What you buy is engineering. You can test whether the right chunk came back. You can test whether the answer stayed faithful to it. You can delete a page and know it will stop being quoted after the next successful ingest. You cannot do those things to a weight.

RAG is still a prompt

Retrieved text is untrusted. A wiki page can contain “ignore previous instructions.” A PDF can contain a fake refund policy. A web page you fetched can contain a tool-shaped order. Wrap chunks, cap how many tokens they may use, and refuse to follow orders that live inside documents. The Prompting track’s injection lesson applies here. This track will spend a whole lesson on wrapping data. For now: retrieved text is an observation, not a new boss.

Do not paste chunks into the system prompt as if they were policy. Policy is written by you. Chunks are written by whoever edited the wiki, including yesterday’s intern and last year’s contractor.

Run to execute this in your browser. Nothing is sent to a server.

Closed-book cannot know your runbook. The first print is a shrug or an invention. RAG can know it — if the runbook is in the corpus and the retriever finds it. The second print quotes runbook.md and names the file. The third print has no overlap with the corpus, so this classroom falls back to closed-book for a public fact. That fallback is a product choice. Many support agents should refuse instead of answering Hamlet from weights while claiming they only use the handbook. Later lessons make refuse a first-class result.

The rest of this track is that “if.” Chunking, embeddings, hybrid search, citations, and memory stores exist because retrieve is where RAG actually fails. Generation is the last ten percent.

What RAG is not

RAG is not a database. If you can look up INV-17 by id, use a tool, not a search. The next lesson is that fork.

RAG is not “dump the whole wiki into the context window.” A long context still has a middle you lose, a bill you pay, and no citation discipline unless you add it. Retrieve-then-pack is how you stay inside a budget.

RAG is not automatically grounded. If you retrieve the wrong chunk, a faithful answer to the wrong chunk is still wrong for the user. If you retrieve the right chunk and the model ignores it, you have a generation bug. You will learn to tell those apart by printing scores and checking quotes.

Common mistakes

• Training a larger model because the bot did not know yesterday’s runbook. Update the corpus.
• Calling a vendor “RAG” and skipping citations. A search box plus a chat is not a product until quotes are checkable.
• Treating retrieve as optional flavor. If the UI says “answers from the handbook,” empty retrieve must not become a closed-book procedure.
• Stuffing fifty chunks “to be safe.” You drown the fact and raise the bill.

How agents use this

If a field is keyed by id, use a tool. RAG is for prose you cannot hash-lookup: runbooks, policies, FAQs, design docs. Agents fail when they search a wiki for invoice INV-17 instead of calling get_invoice. They also fail when they skip retrieve and invent the OOM procedure from pretraining.

In this classroom, “how agents use this” means: the retrieve step is one action you will later put next to tools. The agent loop — choose retrieve vs get_invoice vs stop — is a later track. Here you learn the library: chunks, search, citations, memory stores. If you cannot retrieve the right paragraph, a loop will only retrieve the wrong paragraph faster.

Start every RAG debug by asking three questions: was the doc ingested, did the retriever return it, did the answer quote it? Those are three tests. A green chat UI with a failed ingest is a liar. A perfect prompt with cosine 0.12 is still a miss.