Hybrid Search

Vector search will rank “never refund in cash” next to “how do I refund in cash” because the words sit near each other in embedding space. That is the point of embeddings. It is also the bug when the user types an id.

Keyword search (sparse retrieval) will require tokens like INV-17 to appear, or at least to matter a lot. It is bad at paraphrase: “ran out of RAM” may miss a runbook that only says “OOM” unless you rewrite (later) or embed.

Hybrid search uses both and fuses the ranks (or the normalized scores). You want the id hit and the paraphrase hit, then a single top-k for the prompt.

Typical recipe:

1. Sparse / keyword score (BM25 in production, or a classroom token overlap)
2. Dense / vector cosine
3. Normalize both to a comparable range per query
4. final = alpha keyword + (1 - alpha) cosine
5. Take top-k from the fused list (then threshold if you calibrated one)

alpha is a knob. High when users type SKUs, error codes, and ticket ids. Low when they type vibes (“that memory thing”). Measure it on labeled questions. There is no universal 0.5.

Why keyword still wins some queries

• Identifiers: INV-17, usr_19, stack hashes, CVE numbers
• New terms the embedding model never saw (a product name from last week)
• Exact token presence as a start for negation (not a full solution: “never cash” still needs care)

You cannot add raw BM25 (unbounded, often 0 to 20+) to cosine (about -1 to 1) and call it math. A BM25 of 12.4 would always dominate. Min-max normalize per query, or skip scores and use Reciprocal Rank Fusion in the next lesson.

Min-max on a single query: if all keyword scores are 0, the normalized list is zeros (or a guard). If one chunk has the id, it becomes 1.0 on the keyword channel even if cosine liked a paraphrase better.

Two inverted lists, one prompt

In production, BM25 is an inverted index: term → documents. Cosine is a vector index. Hybrid means two retrieve calls, then fusion. Log both channels’ top hits. Many “RAG is dumb” tickets are “BM25 had the id in slot 1 and we fused it to slot 9.” That is a fusion bug, not a generation bug.

If you only log the fused list, you cannot see which channel saved you or buried you.

Run to execute this in your browser. Nothing is sent to a server.

Watch chunk a jump when alpha rises: INV-17 is a keyword event (the extra +2.0 on tokens with digits or dashes). Pure vectors (alpha=0) dilute it with chunk c’s “refund in cash.” Chunk c is a customer rumor, not policy. That is why hybrid exists.

This classroom uses bag-of-words cosine, not a neural embedder, so you can run it here. The fusion lesson is the same when vec comes from a vendor model: normalize, then mix, then log both.

Negation is still hard

Hybrid does not solve “never cash” vs “cash.” Keywords will retrieve both chunks that contain “cash.” You still need packing, citations, and a model that reads “never.” Do not advertise hybrid as a lawyer.

What to log and what to pin

Log four lists per query: keyword top ids, vector top ids, fused top ids, and any pinned id-matches. If the user typed INV-17 and keyword rank 1 is the policy chunk that contains it, but fused rank 1 is a paraphrase blog, fusion buried the token. Pinning means: chunks that contain an identifier token from the query occupy reserved slots in the packed block even if fused rank is poor. Pinning is not “always retrieve the first wiki hit for invoice.” It is “do not drop the exact token.”

Alpha is per product, not per request from the model. A model that can set alpha will set it to 1 after a page says “keywords only.” You set alpha in config. SKU-heavy catalogs sit high. Chatty FAQs sit lower. Re-measure when the corpus mix changes.

Min-max per query has a sharp edge: if every keyword score is 0, the channel is all zeros and fusion becomes pure cosine. That is correct. If one junk chunk has a tiny BM25 and others are 0, min-max makes junk look like 1.0. Guard: if max keyword score is below a small floor, treat the whole keyword channel as empty rather than stretching noise.

Classroom overlap and bag-of-words cosine are stand-ins. Production BM25 has document-length normalization. Production dense vectors come from the embedder in the last part. Fusion math does not care, as long as you do not add raw unbounded scores to cosine.

Common mistakes

• Adding raw BM25 to cosine.
• One alpha for ids and for chatty FAQs with no eval.
• Logging only fused ranks.
• Skipping keywords because “we have embeddings now.”

How agents use this

When retrieve is a library, hybrid is the default for support corpora that mix policies and identifiers. Log keyword top hits and vector top hits in the same trace line. If BM25 had INV-17 at rank 1 and fusion dropped it, fix fusion (or pin id-matches before mix) before you touch the prompt.

Pinning means: if a chunk contains an identifier token from the query, it cannot fall below a reserved slot. RRF (next) is another way to avoid scale fights. Pinning is a belt. Measure both.

Stay in this lane: search quality. The agent loop that chooses retrieve vs get_invoice is later. If the user pasted an id, a tool still wins. Hybrid is for the prose that contains ids, not for replacing the ledger.

Log both channels. Normalize before you mix. Pin identifier tokens if fusion buries them. Alpha lives in config, not in the model’s arguments. Keyword still wins SKUs, hashes, and last week’s product name.