When RAG Fails

When RAG looks “dumb,” walk the pipeline in order. Do not skip to a larger generator at step 0. The model cannot cite a chunk that was never retrieved, never packed, or never ingested.

1. Is it even a RAG job? Ids belong to tools. Fuzzy search is a slow, wrong database.
2. Ingest — is the doc published, fresh, and safe? Check indexed_at and skip reasons.
3. Chunking — does a human see the fact in one chunk? Inspect the gold section.
4. Retrieve — print scores; check tenant filters; check hybrid for ids; check ANN vs brute on a subsample.
5. Pack — did you drop the gold chunk on a size cap or drown it in duplicates?
6. Generate — is the quote a real substring of packed evidence?
7. Write-back — did you poison semantic memory last week?

That list is the on-call guide. Each stage already had a lesson. Failures map to stages. Adjectives in the prompt are not a stage.

Maps from symptoms

• “Didn’t read the runbook” → often ingest, chunk, retrieve, or pack. Open the trace’s chunk ids.
• “Invented a procedure” → empty retrieve without refuse, or unfaithful generate. Check tau and the citation checker.
• “Cited the neighbor tenant” → filters. Incident, not a prompt tweak.
• “INV-17 retrieved a blog” → needed a tool, or hybrid/pin failed.
• “Sure, three times” → near-duplicate pack, no MMR.
• “Yesterday’s policy” → ingest lag or missing deletes.
• “Followed a wiki order to skip policy” → chunks in instructions, no wrap.

When this track is not enough

If the product must choose retrieve vs get_invoice vs stop, you need the Agents track: loops, state, when not to agent. This track gave you the library: chunks, embed, search, citations, memory stores. The loop is later on purpose. If you cannot retrieve the right paragraph, a loop will only retrieve the wrong paragraph faster.

If you need graders, traces at job scale, and judges, that is Evals. You already have recall@k and quote-in-source. Use them.

RAG stays the library. Agents stay the loop. Tools stay the hands. Mixing all three into one vector soup is how this track started, in reverse.

Run to execute this in your browser. Nothing is sent to a server.

Each failure maps to a stage you already tested. Invoice id without a tool → use_tool_not_rag. Missing from index → ingest. Gold never in top-k → retrieve (which includes filters, hybrid, ANN). Packed out → raise budget or drop duplicates. Quote not a substring → citation checker. The last case is ok if write-back was not poisoned.

Walk a real ticket through this function as a checklist even when you do not have all keys — fill them from the trace. The first True (or missing ingest) wins. That order is the point.

Symptoms to stages, and what comes after this track

“Didn’t read the runbook” is not one bug. Open chunk ids in the trace. Missing from index is ingest. Fact split across windows is chunking. Gold not in top-k is retrieve (filters, hybrid, ANN, tau). Gold retrieved but absent from DATA is pack. Quote not a substring is citation. Neighbor tenant is a leak. Yesterday’s policy is freshness. A wiki order that skipped runtime policy is wrap.

Do not fine-tune because recall@k is 0. Do not buy a new vector database for 400 chunks. Do not rewrite personality because tau is 0.12.

The Agents track is for choosing retrieve versus get_invoice versus stop. This track is the library those actions call. If the library is wrong, the loop is a faster wrong. The Evals track adds judges and job-scale traces. You already have recall@k and quote-in-source.

Keep the checklist next to the golden set. On-call starts at ingest, not at adjectives. RAG stays the library. Memory stays four stores, not one soup. Citations stay substrings.

Common mistakes

• Fine-tune because recall@k is 0.
• New prompt personality because tau is 0.12.
• New vector database because 400 chunks were brute-force fine.
• Skipping the golden set because the demo was pretty.

How agents use this

Keep this checklist next to the golden set. When a ticket says “the bot is stupid,” start at ingest, not at adjectives in the prompt.

The later agent loop will add failures (wrong tool, infinite steps). Those are not RAG failures. Do not debug them with cosine. Do not debug cosine misses with a bigger loop.

You now have the library. Use it. Measure it. Do not skip a stage.

Walk the seven stages with the trace open: tool-vs-RAG, ingest, chunk, retrieve (filters, hybrid, ANN, tau), pack, citation substring, write-back poison. The first failing stage is the ticket. Fine-tunes and new databases are last. The Agents track starts when this checklist is green and the product still must choose retrieve versus lookup versus stop. Until then, you are polishing a library that cannot find the paragraph.

Print scores. Open the chunk. Check indexed_at. Check tenant. Check whether the quote is a substring. That walk is faster than a new model. RAG remains the library. Agents remain the loop. Tools remain the hands. Memory remains four stores. If you remember one order, remember ingest before prompt adjectives.

When recall@k is zero, stop at retrieve. When the quote is fake, stop at citations. When the neighbor tenant appears, stop at filters and treat it as a leak. Do not skip a stage because a larger generator is on sale. The checklist is the on-call guide: tool versus RAG, ingest, chunk, retrieve, pack, cite, write-back. The first miss is the ticket. This track ends at a working library. The loop is later. If the library cannot find the paragraph, a loop will not teach it. Start at ingest. Print scores. Open the chunk. Cite a real substring or refuse. Do not fine-tune step 0.