Beyond Benchmark Scores: How Synthetic and Authentic Query Distributions Diverge in RAG Evaluation
RAG systems are routinely evaluated using synthetic question sets generated from the target document corpus. While this practice provides a useful check on overall retrieval capability, relying exclusively on synthetic benchmarks can mislead under distribution shift and overstate deployment readiness. Synthetic generation spreads questions evenly across the corpus, formulating long, detailed queries; real users put most of their traffic on a few administrative and procedural topics in short queries, while also asking about matters the generator never covers at all. We demonstrate this gap on a university faculty information system, comparing 1,851 synthetic questions generated via Gemini Notebook against 322 authentic queries collected via a student survey. The synthetic and authentic query sets differ significantly: authentic queries average 6.8 words versus 15.7 for the synthetic ones, and draw from only 53 unique sources compared to 165. Consequently, configurations that appear highly effective on synthetic benchmarks experience a substantial performance drop on authentic queries. Importantly, optimizing on synthetic queries selected a higher-latency hybrid retriever. In our setting the sparse retrieval component benefited long synthetic questions but not short authentic ones, costing up to $8\times$ the latency of the fastest configuration we tested. We propose treating synthetic and authentic query sets as complementary extremes of the query-quality spectrum: synthetic data verifies maximum retrieval capacity under idealized conditions, while authentic queries test system robustness to the imprecise, underspecified inputs of real users.