Executive summary
Use hybrid retrieval, with a protected identifier fallback.
The recommended candidate improves overall retrieval while keeping the identifier segment under explicit review before release.
VerdictGO WITH CONDITIONS
Recommended implementation
ChunkingStructural · ~700 tokens
Embeddingvoyage-4
RetrievalHybrid BM25 + dense
RerankerEnabled · final top 5
Condition before release: validate the identifier fallback with held-out queries in the production stack. Offline retrieval evidence does not measure final answer quality or latency.
Current versus recommended
| Metric | Current | Recommended | Delta |
|---|---|---|---|
| Recall@1 | 48.6% | 67.1% | +18.5 pts |
| Recall@3 | 65.4% | 84.2% | +18.8 pts |
| Recall@5 | 74.2% | 91.8% | +17.6 pts |
| MRR | 59.8% | 76.4% | +16.6 pts |
| nDCG@5 | 63.1% | 80.3% | +17.2 pts |
Corpus health
Ready for a comparison, with two files to inspect.
- 86 documents parsed from 11 input files
- 182,400 sampled tokens and 534 retrieval chunks
- Two PDFs have sparse extracted text; OCR before the rerun
- Three duplicate source titles need clearer identifiers
Failure analysis
Exact identifiers drive the remaining release condition.
- Dense retrieval misses 7 identifier queries in the top five
- Hybrid retrieval recovers 6 of those failures
- One ambiguous source label still requires corpus cleanup
- No material regression appears in policy or procedural queries
Segment evidence and regressions
| Segment | Queries | Baseline | Candidate | Delta |
|---|---|---|---|---|
| Procedures | 16 | 69% | 94% | +25 pts |
| Policies | 12 | 75% | 92% | +17 pts |
| Identifiers | 10 | 80% | 90% | +10 pts |
| Short queries | 10 | 70% | 90% | +20 pts |
Tested alternatives
The strongest candidate was not selected on average alone.
- Fixed 800 + dense — current baseline
- Fixed 800 + hybrid — identifier recovery
- Structural 700 + hybrid — selected candidate
- Structural 700 + hybrid + reranker — no material retrieval coverage gain
Migration estimate
Plan the re-index, then measure production economics.
- Estimated 602 chunks for the recommended configuration
- Estimated raw vector storage: 7.0 MB
- Embedding estimate shown per configured model in the real report
- Excludes database overhead, requests, reranking and negotiated rates
Limitations and next step
Implement the candidate, then validate it on held-out queries.
RetrievalGate reports offline retrieval evidence. It does not prove answer correctness, production latency, access controls or the final user experience. A Full Audit includes one rerun to validate the implementation with fresh evidence.