Build the simplest useful contract
Each CSV row needs query and relevant_id. Add segment for a business category such as billing, identifiers or policies. This version supports one relevant source per row. Use unique questions; repeated rows inflate the apparent sample size and are rejected. The audit caps the number of evaluated rows by tier.
Use stable source references
After uploading, expand Source ID in the private file list. Copy the full value into relevant_id. A unique filename such as handbook.md also works, but duplicate filenames are ambiguous. For tabular CSV data, use #row=17, where the header is line one. For JSON arrays, the parser identifies items with source fragments shown in the methodology.
Represent the difficult tail
Include identifiers, numeric questions, short requests, long procedural questions and each language your users need. The engine adds heuristic query categories and keeps your custom segment. A strong average from mostly easy questions is weak evidence about the errors that trigger support tickets.
Treat synthetic questions as a starting point
Without evaluation.csv, the engine derives questions from selected passages, using the configured question-generation provider or a deterministic fallback. Those questions are useful for initial diagnosis but can echo the source language and underrepresent real user behavior. Their synthetic origin stays visible in the evidence panel.
Keep fresh queries for validation
Use one set to explore and choose a candidate, then a separate held-out set for the final rerun. Current reports use the same questions for selection and displayed metrics; they do not claim an independent holdout score. Review annotation mistakes and duplicate documents before interpreting small differences as meaningful gains.