Post-Generation Verification Dominates Retrieval Optimization: A 2^4 Factorial Ablation of RAG Pipeline Features

Abstract

Modern RAG pipelines stack many enhancement features, but these features are typically validated in isolation, leaving their interactions unmeasured. We run a 2^4 full factorial ablation of four pipeline features: section expansion (SE), agentic search (AS), completeness check (CC), and table-of-contents-guided retrieval (ToC). The design crosses 16 configurations, 24 queries spanning eight interaction types, and two cloud-class models (768 designed runs, 759 completed and scored) on five public documents (78-492 pages), and every answer is scored against a verified reference. On this corpus and task, post-generation verification dominates: CC is the strongest feature (d=+0.48, p<0.001), improving accuracy, completeness, and usefulness simultaneously, and CC alone (4.31/5) is ahead of every configuration without it, including the three-feature SE+AS+ToC (4.11). ToC yields a significant gain at zero additional LLM calls (d=+0.22); AS shows a small, unstable effect that does not survive correction for multiple comparisons, helping some queries and harming others; SE is neutral. These main effects hold when every response is re-scored by two alternate LLM judges. The highest-quality configuration roughly doubles baseline latency, producing a genuine quality-latency Pareto frontier of six configurations. Feature utility is strongly query-type dependent (CC's completeness effect reaches d=+0.83 on completeness-demanding queries), so single-query-type evaluations systematically mis-rank features. We conclude that verifying answers matters more than optimizing retrieval, and that factorial designs with diverse query types are necessary to evaluate RAG features.