RAG vs a Lookup Tool

Teams “add RAG” to a workflow that needed SQL. Then they wonder why “invoice INV-17” retrieves a blog post about invoices. The blog post is about invoices. The user asked for this invoice. Cosine does not know the difference between a type and a row.

A tool here is a function with a name and arguments that hits a system of record: get_invoice(id), get_ticket(id), get_user(id). The answer is a row (or a small JSON object), not a paragraph. Auth already exists on that API. The id is stable.

RAG is for prose you cannot name as a row in advance. The user asks “how long do refunds take” and the answer lives in billing.md under a heading. You retrieve a quote. You cite the file.

Use a tool when

• You have a stable id (INV-17, usr_9, ticket 9182)
• The answer is a row, not a paragraph: status, cents, assignee, due date
• Auth already exists on that API, including “this user may not see this row”
• Wrong answer is an incident, not a slightly off FAQ

Money, entitlements, and “did we already refund this” are tool jobs. A wiki sentence that happens to contain INV-17 is not a ledger.

Use RAG when

• The answer lives in prose: runbooks, policies, FAQs, architecture notes
• You cannot name the row in advance (“what do we do on runner OOM”)
• You need a quote the user can open
• Near-enough is acceptable if you cite, and a miss should refuse rather than guess a row

A how-to can be a little fuzzy if the citation is honest. An invoice cannot.

Fuzzy search is not a database

Retrieving “something about refunds” is fine for a how-to. It is not fine for “refund this invoice.” Wrong row plus a confident sentence is an incident. The model will not say “I searched the wiki.” It will say “INV-17 is open for forty dollars” because a nearby paragraph mentioned forty dollars and another mentioned INV-17.

Keyword overlap makes this worse: the query “status of invoice INV-17” shares the word “invoice” with every billing essay. The id token may appear in a tutorial. Tutorials are not the ledger.

SQL (or an API) returns not_found when the id is missing. RAG returns the least-bad paragraph unless you add a threshold. Least-bad is how cash-refund rumors start: the wiki said “never cash” in one chunk and “customers ask about cash” in another, and the model mixed them.

Run to execute this in your browser. Nothing is sent to a server.

The tool returns the row: 4000 cents, open. RAG returns a policy paragraph about refund timing. Both can be useful in one product. They are not substitutes. The tool miss for INV-99 is not_found. RAG would still return a paragraph if you asked it, because “invoice” overlaps the wiki. That is the bug.

Both, with a rule for when not to search

Give the model both later: get_invoice(id) and retrieve(query). This track does not build the choose-a-tool loop. It builds the rule you will put in the descriptions: if the user pasted a stable id, look it up first. Do not search the wiki for the id unless the lookup returned not_found and you still need a how-to.

Descriptions must say when not to retrieve. “Search the company wiki” with no negative rule becomes a fuzzy database. “Search runbooks for procedures. Do not search for invoices, users, or tickets; those have lookup tools.” is a real description.

Auth is not optional seasoning. get_invoice already knows the caller. A wiki search that is not filtered by tenant will return a neighbor’s refund policy. The tenant-filter lesson is later. The idea starts here: tools inherit API auth; RAG must be given the same tenant on purpose.

Mixed workflows are still two systems

A common product is: look up the invoice, then retrieve the policy that applies to that status. The lookup returns status: open and cents: 4000. The retrieve query is then a policy question you wrote in code, like “refund timing for open invoices,” not the user’s original “what about INV-17.” You cite the policy chunk. You display the row from the tool. If you stuff the row into the wiki index instead, you lose not_found, you lose authz, and you lose a transaction log.

Auth is inherited on tools because the API already knows the caller. RAG must be given the same tenant and the same “may this user see this collection?” check. A public FAQ collection and an internal runbook collection are different filters. Support users may retrieve FAQ. Only on-call may retrieve the production runbook. That is not cosine. That is the same idea as get_invoice returning 403.

Eval the fork with two buckets of questions: (1) pasted ids must call a lookup in the later agent track; for this track, your retrieve tests should not treat an invoice id as a gold wiki chunk; (2) how-to questions must retrieve a policy chunk, not a random row dump. If your golden set only has FAQs, you will never catch the fuzzy-database bug.

When a wiki page mentions an id in an example, hybrid search will want that page for real ids. Pinning identifiers (later) can make this worse if the example is INV-17 in a tutorial. Tutorials should use obviously fake ids, or ingest should tag them collection=tutorial and default retrieve should skip that collection. This is still RAG vs tools: examples are prose; ledgers are rows.

Common mistakes

• Embedding the invoice table “so RAG can see money.” You built a slow, wrong database with no transactions.
• One collection named “everything.” Runbooks, tickets, and Slack dumps do not want the same retriever.
• Treating a high cosine as a found row. Cosine is not a primary key.
• Skipping not_found because the wiki mentioned a similar id.
• Using the user’s raw sentence as the only query after you already looked up a row. Rewrite the policy query from the row’s fields.

How agents use this

A support flow often needs both in sequence without being one soup: look up the invoice, then retrieve the refund policy that applies to that status. The lookup is exact. The policy is RAG. The citation is on the policy, not on the cents.

When you write tool descriptions (later tracks), spend as many words on when not to call retrieve as on when to call it. If the user pasted INV-17, retrieve is the wrong first action. If the user asked “what does OOM mean in our runner,” lookup is the wrong first action.

If you only remember one sentence: ids are tools; prose is RAG. Mixing them makes a slow, fuzzy database.