Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.40167

Beyond $χ^2$: Ensemble null tests for reliable scientific discovery

Adrian E. Bayer·David N. Spergel
View original source ↗Read source document ↗Export metadata ↓

Abstract

Unrecognized systematic errors can survive standard checks and undermine scientific discoveries. Null tests and experimental controls can expose these failures, but averaging measurements or reducing them to scalar statistics can hide important structure. We propose ensemble null tests that compare multivariate distributions with references from observations or simulations. We demonstrate their benefits across science. In cosmic microwave background simulations, they detect systematic effects missed by conventional $χ^2$ tests. In Gaia data, they detect velocity asymmetry as sensitively as an informed search. In a CRISPR screen, they expose a control mismatch in DNA damage and reveal which cellular profiles drive it. As scientific datasets grow, ensemble testing turns repeated measurements into a broader search for unknown unknowns, strengthening the empirical foundation for scientific discovery.

Explore related subjects

stat.MEastro-ph.COastro-ph.IMphysics.data-anq-bio.QM

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Adrian E. Bayer, David N. Spergel. 2026-10-29. Beyond $χ^2$: Ensemble null tests for reliable scientific discovery. https://arxiv.org/abs/2610.40167

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

On the closed-loop Volterra method for analyzing and forecasting time series

This paper studies the closed-loop Volterra method for analyzing and predicting stationary time series. The method represents the dependence of a current observation on past observations through polynomial terms, including powers of individual lags and interactions between different lags. Since the number of coefficients increases rapidly with the memory length and polynomial degree, we use the theory of reproducing kernel Hilbert spaces to estimate these coefficients by regularized least squares through kernel evaluations. We establish conditions on marginal moments, temporal dependence, model complexity and regularization under which the prediction risk converges to the minimum mean squared prediction error as the sample size increases. We also consider data-driven selection of tuning parameters by a rolling-origin procedure based on one-step-ahead prediction errors, and show that the resulting selector asymptotically attains the prediction risk of the best candidate in the considered grid. Simulation studies and a real-data application examine the finite-sample performance of the method and its comparison with linear and nonlinear prediction methods.

stat.ME↗
arXiv2026

Boosting e-BH via conditional calibration

The e-BH procedure is an e-value-based multiple testing procedure that provably controls the false discovery rate (FDR) under any dependence structure between the e-values. Despite this appealing theoretical FDR control guarantee, the e-BH procedure often suffers from low power in practice. In this paper, we propose a general framework that boosts the power of e-BH without sacrificing its FDR control under arbitrary dependence. This is achieved by the technique of conditional calibration, where we take as input the e-values and calibrate them to be a set of "boosted e-values" that are guaranteed to be no less -- and are often more -- powerful than the original ones. Our general framework is explicitly instantiated in three classes of multiple testing problems: (1) testing under parametric models, (2) conditional independence testing under the model-X setting, and (3) model-free conformalized selection. Extensive numerical experiments show that our proposed method significantly improves the power of e-BH while continuing to control the FDR. We also demonstrate the effectiveness of our method through an application to an observational study dataset for identifying individuals whose counterfactuals satisfy certain properties.

stat.ME↗
arXiv2026

Differentially Private Covariate Balancing Causal Inference

Differential privacy is the leading mathematical framework for privacy protection, providing a probabilistic guarantee that safeguards individuals' private information when publishing statistics from a dataset. This guarantee is achieved by applying a randomized algorithm to the original data, which introduces unique challenges in data analysis by distorting inherent patterns. In particular, causal inference using observational data while preserving privacy is challenging because it requires a good covariate balance between treatment groups, but checking the covariate balance is challenging when privacy is a primary concern because releasing any statistic (e.g., t-statistic) for the balance check compromises privacy. Additionally, the performance of the privatized estimator critically depends on the choice of differential privacy mechanisms, and it remains unexplored how privacy-protecting causal inference could be conducted with observational data. In this article, we present a differentially private two-stage covariate balancing weighting estimator to infer causal effects from observational data. Our algorithm produces both point and interval estimators with statistical guarantees, such as covariate balance, rate optimality, and oracle budget allocation, under a given privacy budget.

stat.ME↗