MMR and Coverage

If three chunks are near-duplicate refunds, a reranker will put them 1–2–3 and you will waste the context window. The model will read the same sentence three times and sound sure. Sure is not coverage. The user asked about refunds and OOM. You delivered refunds, refunds, and refunds.

You want coverage of sub-questions, not three copies of the same sentence. Overlap in the source (heading split + 15% window overlap) makes near-duplicates likely. That overlap is good for not tearing facts. It is bad if you pack all of them.

MMR (maximal marginal relevance) picks the next chunk that is relevant to the query and unlike what you already picked. The usual shape:

score(candidate) = relevance(query, candidate) − lambda * similarity(candidate, chosen_set)

A cheap classroom version: relevance is word overlap with the query; similarity is word overlap with the union of chosen chunks; lambda (often written λ, here lambda) is how hard you punish duplicates. High lambda: more diversity, maybe you drop a still-useful second policy. Low lambda: greedy relevance, duplicates stay.

Dedup is the sibling of MMR

Before MMR, dedup by chunk id (trivial) and by near-duplicate text (normalize whitespace, maybe hash the first 200 characters). If two ids are the same bytes, keep one. MMR then handles paraphrases that are not exact copies.

Three copies make the model overconfident: “several sources say 5-7 days.” They were one source, chunked three times. Citations should not look like three independent witnesses.

MMR is not a substitute for packing limits. It chooses which chunks. Packing (next) chooses how many characters.

Run to execute this in your browser. Nothing is sent to a server.

Greedy overlap wants two refund paraphrases: they both overlap “refund INV-17 days.” MMR keeps one refund chunk and the OOM chunk — both parts of the query. That is coverage. Toggle lam toward 0 and MMR collapses toward greedy. Toggle toward 1 and it may pick “never cash” as the diverse refund sibling instead of OOM, depending on overlap with the first pick. That is the trade: diversity among refunds vs covering the OOM sub-question. Your lambda should be set by whether the query is multi-part, not by a blog post.

A production version uses cosine between chunk vectors for the duplicate term, not word sets. Same idea. Do not pull in a numeric library here; you already know cosine by hand from the embed lessons.

Multi-part queries

If query rewrite (later) splits “refunds and OOM” into two retrieves, merge with MMR or with a simple round-robin per hop so one hop cannot fill the window. If you do not split, MMR is the cheap joiner.

Duplicates, lambda, and false diversity

Near-duplicates come from overlap windows, from mirrored wiki pages, and from “same policy in FAQ and runbook.” Dedup exact bytes first (normalize whitespace). Then MMR for paraphrases. If you only MMR, exact copies still waste a slot until the duplicate term gets large.

Lambda is the diversity knob. Set it by whether queries are multi-part, not by a default blog value. A single-fact FAQ (“how many days?”) wants low lambda so the best refund chunk stays and a random OOM page does not sneak in as “diversity.” A two-part question wants higher lambda or two retrieves.

False diversity: two chunks that share the word “refund” but are timing vs never cash are not duplicates. Penalizing all overlap can drop the second policy. Use cosine between chunk vectors for the duplicate term if you have them, or overlap on content words after dropping the query words. The classroom word-set is a sketch.

Citations: if three packed chunks are the same paragraph, the UI should show one source. MMR is how the prompt avoids pretending there are three witnesses. Overconfidence from copies is a packing bug that looks like a model bug.

n in MMR should be the pack budget in chunks, not stage-1 k. If n equals k, you only reordered the pile. Diversity never dropped anyone.

Common mistakes

• MMR with n equal to k, so you still pack everything, just in a different order.
• Dedup only on id after re-chunking produced new ids for the same sentence.
• Penalizing all overlap so a heading word shared by two different sections drops a needed policy.

How agents use this

Dedup by chunk id and by near-duplicate text before stuffing the prompt. Three copies make the model overconfident. Coverage beats repetition.

When traces show three citations to the same paragraph, users think they have independent confirmation. Your UI should collapse them. MMR is how the prompt avoids the lie. The agent loop does not fix a packed pile of clones.

If working memory is already huge, retrieve fewer diverse chunks rather than more duplicates. Packing next: the budget.

Set lambda from query shape when you can: low for a single FAQ fact, higher when rewrite emitted two sub-queries or the user asked two things. After MMR, you should still be able to point at one refund span and one OOM span on a two-part question. If MMR dropped the only id-match, pin that id first, then diversity-rank the rest. Coverage is a retrieve job. Generation cannot invent the missing section if you never packed it.

Exact-byte dedup is cheap and should always run. Overlap windows will produce copies; that is expected. MMR is for paraphrases that hashing will not catch. Do not raise k to “get more coverage” if the extra rows are the same sentence. Coverage means different facts, not more tokens of the same fact. Three paraphrases of refunds will make the model sure and leave OOM out of the window. Sure is not coverage.