The Quantification of Replicability

Ester Alongi; Gianmarco Altoè; Giovanni Parmigiani

Home/Archive/arXiv

arXiv · 2610.14407

The Quantification of Replicability

Ester Alongi·Gianmarco Altoè·Giovanni Parmigiani

View original source ↗Read source document ↗Export metadata ↓

Abstract

What does it mean to measure the replicability of scientific results? Is it merely determining whether results are consistent across studies? Much of the existing literature on replicability draws conclusions exclusively from the comparison of study results, in terms of observed estimates, uses a variety of different metrics, and ultimately collapses the comparison into a binary verdict of success or failure. This fails to differentiate between repeated errors and corroboration of the shared generative effect. We develop a unified Bayesian framework in which distinct but related notions of replicability are formally defined and jointly embedded within a single inferential structure. A single model answers four complementary questions: agreement among independent studies; consistency between the individual studies and the shared generative effect; agreement between an existing body of evidence and a new study from the same super-population; inference on the shared generative effect. In practice, we suggest quantifying replicability through posterior probabilities relative to regions of practical relevance. We illustrate these ideas in an eQTL (expression quantitative trait locus) analysis assessing the replicability of a regulatory effect across human ancestral groups.

Explore related subjects

stat.ME

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Ester Alongi, Gianmarco Altoè, Giovanni Parmigiani. 2026-10-09. The Quantification of Replicability. https://arxiv.org/abs/2610.14407

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Comparison of Bayesian methods for extrapolation of treatment effects: a simulation study

Extrapolating treatment effects from related studies is a promising strategy for designing and analyzing clinical trials in situations where achieving an adequate sample size is challenging. Bayesian methods are well-suited for this purpose, as they enable the synthesis of prior information through the use of prior distributions. While the operating characteristics of Bayesian approaches for borrowing data from control arms have been extensively studied, methods that borrow treatment effects--quantities derived from the comparison between two arms--remain less well understood. In this paper, we present the findings of an extensive simulation study designed to address this gap, across six clinically motivated case studies covering continuous, binary, recurrent-event, and time-to-event endpoints. We varied target sample size, treatment-effect drift, and endpoint-specific design assumptions. Power and estimation performance were compared using a frequentist analysis of the target data alone, calibrated to the same actual type I error rate. Our results provide insights into the strengths and limitations of existing methods in the context of confirmatory trials. At matched actual type I error rates, power differences were usually small, with occasional gains over the reference analysis and larger losses for some adaptive procedures. Borrowing reduced MSE when source and target effects were consistent, but increasing drift eroded this benefit and could substantially reduce interval coverage. Commensurate priors with appropriately chosen hyperpriors offered the most favorable balance between estimation gains and robustness to conflict. The conditional power prior was competitive for estimation at comparable type I error rates, despite its lack of adaptation. Test-then-pool procedures often performed poorly. No method dominated across all criteria.

stat.ME↗

arXiv2026

Efficiently Computed Marginal Likelihoods for Multivariate Mixture Models

We propose a new methodology for efficiently computing marginal likelihoods for univariate and multivariate mixture models. The estimator that this methodology proposes computes the marginal likelihood from Markov chain Monte Carlo (MCMC) samples, is consistent, asymptotically normal and of finite variance. In addition, it is invariant to label switching, does not require posterior samples from hidden allocation vectors, and is efficiently approximated, even for an arbitrarily high number of components. Its computational efficiency is based on an asymptotically optimal ordering of the parameter space, which can in turn be used to provide useful visualisations. We test it in simulation settings where the true marginal likelihood is available analytically. It performs well against state-of-the-art competitors, even in multivariate settings with a high number of components. We demonstrate its utility for inference and model selection on univariate and multivariate data sets.

stat.ME↗

arXiv2026

Asymmetric Space-Time Covariance Functions via Hierarchical Mixtures

This work is focused on constructing space-time covariance functions through a hierarchical mixture approach that can serve as building blocks for capturing complex dependency structures. This approach provides a unifying modeling framework that not only constructs a new class of asymmetric space-time covariance functions with closed-form expressions, but also provides corresponding space-time process representations, which further unify constructions for several existing space-time covariance models. This hierarchical mixture framework decomposes the complexity of model specification at different levels of hierarchy, for which parsimonious covariance models can be specified with simple mixing measures to yield flexible properties and closed forms. A characterization theorem is provided to study how the mixing measures determine the statistical properties of covariance functions. Several new covariance models resulting from this framework are discussed in terms of their practical usefulness. This hierarchical mixture approach is extended for nonstationary and multivariate settings. A theorem is also provided to construct valid asymmetric space-time covariance functions with arbitrary and possibly different degrees of smoothness and long-range dependence in space and in time. This generalized class also bridges a theoretical gap in using the Lagrangian reference framework. The superior performance of several new parsimonious covariance models over existing models is verified with the well-known Irish wind data and the U.S. air temperature data.

stat.ME↗