Vector Indexes

With 200 chunks, brute force cosine against every vector is the correct design. It is exact and testable. People skip it and install a distributed vector database for a help-center corpus that fits in RAM. Then they debug “ANN recall” before they debug chunking.

Brute force means: score the query against every document vector, sort, take top-k. Recall of the true nearest neighbor is 100% of whatever your embedding deserves. If gold is missing, the embedding or the chunk is wrong, not the index. Latency grows with corpus size. For 10k short chunks on one machine, Saturday is still a valid runtime.

With millions of vectors, brute force is too slow. You use an ANN (Approximate Nearest Neighbor) index. ANN does not guarantee the true nearest neighbor. It guarantees a speed/recall tradeoff you must measure. You only score a subset of vectors (buckets, clusters, a graph of neighbors). Faster. Some true neighbors never get scored.

Recall@k vs latency

Recall@k (retrieval sense) is the fraction of queries whose gold chunk appears in the top k. Report it next to latency. If you cannot quote both, you are collecting a dependency, not operating an index.

ANN papers quote recall of the true nearest neighbor. You care about gold chunk in top k, which also depends on chunking and embeddings. Still: an ANN that drops the true neighbor cannot retrieve gold if gold was that neighbor.

Tune ANN parameters the way you tune tau: on a labeled set. Graphs go stale when you add many vectors. Rebuild (or the vendor’s incremental insert that you have tested) is part of ingest. A graph built in January with inserts until September can quietly lose recall.

You might not need a new database

For many products, a vector column in the database you already run is enough until latency proves otherwise. Filter by tenant_id in SQL, then brute force the remaining vectors, or use the database’s ANN on that filtered set. Separate systems mean two auth stories and two backups.

When you do need ANN, treat it as a cache of geometry, not as a source of truth. You should still be able to brute-force a subsample and compare. The classroom demo below is a one-bit bucket: crude on purpose so you can see a miss.

Run to execute this in your browser. Nothing is sent to a server.

When overlap is below 5/5, you see approximation: the true neighbors lived in the other bucket. Production ANN is the same idea with better graphs. Never skip a brute-force baseline on a subsample. If subsample brute and ANN disagree on gold questions, you have an index problem, not a prompt problem.

“ANN scanned N of M” is the speed story. If N is still M, you did not approximate. If N is tiny and recall is 0.4, you approximated too hard.

Debug like an index, not like a chatbot

Ship a “search this id” debug endpoint that prints a chunk’s nearest neighbors. When a gold chunk never appears, ask: is it in the index at all? Which bucket? What is cosine to the query under brute force? ANN miss looks like a smart model that “didn’t read the doc.”

Rebuild when ingest adds a large batch. Measure recall@k in CI on a tiny golden set (later lesson). Do not wait for a customer to find the missing runbook.

What you are actually approximating

ANN families (graphs, clusters, trees) all skip some vectors. Skipping is the speed. The miss is a gold neighbor that lived on the other side of a partition, like the toy bucket. Production graphs are better than one bit, but they still miss, especially after many inserts without rebuild, and especially for queries that do not look like the rest of the graph.

Filters plus ANN is a second trap. If the index is global and you filter tenant after ANN, you may have scored a neighbor’s vectors and then dropped them, leaving this tenant with a thin, wrong shortlist. Prefer filter-then-ANN, or a per-tenant graph. The next lesson is tenants; the index must not fight it.

A vector column in the database you already run, with brute force on a few thousand rows, is boring and correct. Move to ANN when p95 latency on brute force exceeds your SLO on production-sized data, not on a blog post about millions of vectors.

Debug endpoint: given a chunk id, print its text, tenant, model name, and top neighbors under brute force on a subsample. When a user says “it never finds section 4.2,” look the section up by id first. Not in the index is ingest. In the index but not a neighbor of the query is embedding or chunking. Neighbor under brute, missing under ANN, is the index.

Rebuild cost is part of ingest SLO. A graph that takes six hours to rebuild needs a plan for daytime edits: dual-write, or accept lag. Dual-write without a checksum is how two versions of a refund policy both retrieve.

Common mistakes

• Distributed ANN for 800 FAQ chunks.
• No brute-force baseline, so every miss is “the model.”
• Never rebuilding the graph.
• Using ANN as a permission system (next lesson: it is not).

How agents use this

An ANN miss looks like a smart model that “didn’t read the doc.” Check retrieval before you rewrite the prompt. Rebuild graphs when you add many vectors. Ship a debug endpoint that prints neighbors for a chunk id.

The agent loop will call retrieve. Retrieve will call the index. If the index is approximate, your eval must include questions whose gold neighbors sit in the long tail of the graph. Otherwise you only test the easy cluster.

Brute force is the truth. ANN is a trade. Quote both numbers.