Chunking
Fixed 400, 800 and 1,200 tokens; structural about 700.
RetrievalGate research - September 2026
On this SciFact slice, hybrid retrieval lost to dense search.
We evaluated 75 public scientific claims against a reproducible 99,981-token SciFact corpus. The best dense configuration reached 97.3% Recall@5; the tested hybrid variants did not improve it.
TL;DR
Changing retrieval strategy was not automatically an improvement.
Against the reconstructed baseline, structural 700-token chunks with text-embedding-3-large improved Recall@5 by 2.7 percentage points. BM25 and the tested hybrid variants were materially weaker. This is a result for this corpus and query set, not a leaderboard claim.
Question and corpus
Can a global score hide an important retrieval trade-off?
SciFact is a public BEIR scientific-claim retrieval dataset. We selected 263 documents (99,981 estimated tokens) and 75 test queries with one positive relevance reference per query. The fixture is deterministic: positive documents are retained first, then public distractors are selected by stable hash.
The query set contains 47 manually-labelled semantic claims and 28 numeric claims. RetrievalGate also assigned heuristic categories such as exact, identifier and short; these overlap and are diagnostic only.
Method
A controlled comparison, not a generic leaderboard.
Every candidate saw the same corpus, relevance evidence and query set. We changed one practical retrieval decision at a time.
Embeddings
OpenAI text-embedding-3-small and text-embedding-3-large.
Retrieval
BM25, dense search and reciprocal-rank-fusion hybrid.
Decision metrics
Recall at 1, 3 and 5, plus MRR and nDCG at 5.
No reranker in this run
It was deliberately excluded, so this result isolates first-stage retrieval.
Result
Dense retrieval won this comparison.
The best result came from a larger embedding model and structural chunks, without adding a hybrid layer.
Baseline
94.7%Recall at 5Dense / fixed 800 / embedding-3-smallBest tested
97.3%Recall at 5Dense / structural 700 / embedding-3-large+2.7 points vs baselineBest hybrid shown
86.7%Recall at 5Hybrid / fixed 800 / embedding-3-large-8.0 points vs baselineFinding
Adding hybrid retrieval would have lowered Recall at 5 on this slice. BM25 reached 77.3%; dense fixed 800 with embedding-3-large reached 96.0%. The result is specific to this corpus and evidence, which is exactly why the change needed testing.
What the average hides
Inspect categories before shipping.
The winning dense configuration improved numeric Recall@5 from 96.4% to 100.0%, and exact-category Recall@5 from 91.3% to 95.7%. But its heuristic multilingual_or_other category fell from 94.1% to 88.2% at Recall@3, despite an unchanged Recall@5 of 94.1%.
The corpus is English, so this is not evidence about multilingual retrieval. It is evidence that a global winning configuration can still move some query types lower in the ranking. That is why the segment label, sample size and rank cutoff belong beside the average.
Evidence boundary
Useful for a decision. Not a universal claim.
75 real queries
Enough for a directional comparison, not for a universal ranking.
One expected source
Each query used its first positive BEIR relevance judgment.
99,981-token slice
This is a public SciFact slice, not the complete corpus.
Validate before release
The same queries selected and measured candidates; use held-out evidence next.
Not measured
Reranking, API cost, serving latency and infrastructure cost were outside this run.
Make the same decision on your evidence
Test the retrieval change before you rebuild the index.
Start with a Free Snapshot, then use real queries to test the configurations that matter to your system.