If a retrieval system hands the model the wrong three chunks, no amount of prompt work saves the answer. Yet retrieval is routinely the least measured part of the stack, because measuring generation feels more like the product.
Here is the smallest harness I have found genuinely useful, built twice now on client work.
The three things you have to have
- A fixed question set with known-good sources. Fifty questions is enough to start. Each one records which document IDs should come back, written by someone who knows the corpus.
- Rank-aware metrics, not hit rate. Whether the right chunk appeared in the top 20 is nearly useless when you pass 5 to the model.
- A stored run artefact. Every run writes a JSON file with config, per-question results, and a git SHA. Without this you cannot answer “did last Tuesday’s chunking change help.”
Metrics that earn their place
- Recall@k — did the relevant documents appear in the top k. Report at your actual k, and at 3x your k so you can see whether reranking has room to work.
- MRR — reciprocal rank of the first relevant hit. Cheap, and it moves when ordering improves.
- nDCG@k — the one to use when relevance is graded rather than binary.
I skip precision unless the corpus has near-duplicates, where it suddenly becomes the only metric that shows the problem.
def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
for i, doc_id in enumerate(retrieved, start=1):
if doc_id in relevant:
return 1.0 / i
return 0.0The configuration axes that actually moved numbers
On a 40,000-document corpus of engineering runbooks:
| Change | Recall@5 | MRR |
|---|---|---|
| Baseline: 512-token chunks, dense only | 0.61 | 0.44 |
| Chunk on headings, 200–900 tokens | 0.72 | 0.53 |
| Add BM25, reciprocal rank fusion | 0.84 | 0.66 |
| Cross-encoder rerank of top 50 | 0.89 | 0.79 |
Two observations. Hybrid search was the single biggest win and it is the least fashionable option on the list — a lot of runbook queries contain exact identifiers that dense embeddings blur. And chunking on document structure beat every fixed-size variant I tried, which cost nothing but an afternoon with the parser.
Retrieval failures are boring and fixable. Generation failures are interesting and mostly are not. Spend your time accordingly.
Building the question set without dying
The objection is always that nobody has time to hand-label fifty questions. Two shortcuts that worked:
- Mine real queries from logs, then label only which documents were right, which is far faster than writing questions from scratch.
- Generate candidate questions from documents with a model, then have a human keep or bin each one. Keeping is fast. Writing is slow.
Neither produces a perfect set. A rough set you actually run beats a perfect set you never build.
What I would add next
Per-question tracking over time, so a regression on one query type is visible rather than averaged away. Right now I get an aggregate table and have to go digging when it drops, which is exactly the mistake I wrote about in the Watchman p99 post — instrument the shape of the workload, not just the total.
The BEIR benchmark is a reasonable reference for metric implementations if you would rather not write your own.