The starting point: mixed query intent
The team has manuals, product tables and support procedures in one corpus. Most evaluation questions are conversational, so the average score is dominated by semantic retrieval. The smaller identifier segment contains the part numbers that support engineers rely on. Both groups need to be represented explicitly in the evaluation CSV.
The attractive candidate
Imagine a candidate that improves overall Recall@5 from 72% to 80%. On the surface, the eight-percentage-point increase looks ready to ship. But the identifier segment drops from 90% to 70%. A user searching a specific part now has a worse chance of finding the correct document, despite the improved average.
The finding that changes the decision
The segment comparison exposes the loss and the query regression list shows the actual affected questions. The report associates the failure with a likely lexical gap. That is a tested hypothesis to investigate, not proof that every identifier needs the same routing rule.
The next controlled experiment
The team tests hybrid search on the same chunks and embedding model, isolating the retrieval-mode change. If the identifiers recover without hurting other important segments, the candidate becomes more credible. If they do not, the team inspects extraction, chunk boundaries, exact source references and duplicate documents before adding more complexity.
What counts as a successful outcome
A useful audit can recommend a better configuration or prevent a harmful migration. The production change only goes ahead after the team reviews the implementation card, checks fresh queries and measures latency in its own application. A NO-GO that avoids a regression is a useful result too.