Value-of-Information Analysis for External Validation of Risk Prediction Models in Multicenter Studies and Systematic Reviews

Laure Wynants; Kim Zhipei Wang; Sabine Grimm; Andrea Gabrio; Andrew Vickers; Ewout Steyerberg; Ben Van Calster; Mohsen Sadatsafavi

arXiv · 2607.02321

Value-of-Information Analysis for External Validation of Risk Prediction Models in Multicenter Studies and Systematic Reviews

Laure Wynants·Kim Zhipei Wang·Sabine Grimm·Andrea Gabrio·Andrew Vickers·Ewout Steyerberg·Ben Van Calster·Mohsen Sadatsafavi

View original source ↗Read source document ↗Export metadata ↓

Abstract

External validation studies have finite sample sizes, creating uncertainty about whether a prediction model's Net Benefit (NB) exceeds default strategies' NB. The expected value of perfect information (EVPI) quantifies consequences of uncertainty. Current EVPI methods focus on single studies, ignoring between-center heterogeneity. We extend EVPI and expected value of partial perfect information (EVPPI) to account for between-cluster heterogeneity in multicenter studies and meta-analyses. We distinguish between the global and local optimal strategy and between observed and unobserved clusters. We define EVPIglobal, EVPIcluster_j, EVPIcluster, and EVPPIcluster,prevalence, implemented in the MetaNB R package, and illustrate them using a systematic review across 36 centers of the ADNEX model for ovarian cancer diagnosis. Assuming one global decision regarding ADNEX adoption, there is no need for further data to confirm ADNEX is superior overall (EVPIglobal 0). Meta-analysis borrows information across observed clusters, resulting in consistent local superiority of ADNEX and nonzero but typically lower EVPIcluster_j than when considering local data alone. There is 0.03 probability default strategies are superior in unobserved centers. Eliminating uncertainty on performance and prevalence in each (EVPIcluster) would gain 1134 net avoided false positives (FP) per year, assuming 350000 tumors annually with 20% malignancies. Determining only local prevalence with certainty (EVPPIcluster, prevalence) would gain net 158 avoided FP per year. EVPI extensions disentangle sources of uncertainty and quantify the need for further validation to determine the global or locally optimal strategy. Considering uncertainty and heterogeneity in clinical utility across clusters is essential to decide whether additional validation studies are warranted.

Explore related subjects

stat.AP stat.ME

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Laure Wynants, Kim Zhipei Wang, Sabine Grimm, Andrea Gabrio, Andrew Vickers, Ewout Steyerberg, Ben Van Calster, Mohsen Sadatsafavi. 2026-07-02. Value-of-Information Analysis for External Validation of Risk Prediction Models in Multicenter Studies and Systematic Reviews. https://arxiv.org/abs/2607.02321

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Privacy-Preserving Causal Meta-Mediation Analysis with Survival Outcomes

Privacy and data-governance constraints often prevent pooling individual-level data across studies, limiting the use of conventional approaches for causal media- tion analysis in multicenter settings. We propose a federated causal meta-mediation framework for right-censored time-to-event outcomes that enables collaborative es- timation without sharing individual-level data. Our framework targets natural indirect effects in a prespecified population by combining information on mediator and outcome mechanisms across distributed data sources. A site-by-site identifi- cation strategy further allows heterogeneity across data sources to be character- ized, with a variance decomposition separating outcome-related, mediator-related, and interaction components. We develop federated one-step and targeted maxi- mum likelihood estimators that accommodate data-adaptive and machine-learning methods for nuisance-function estimation. The finite-sample performance of the proposed estimators is evaluated through numerical simulations. To illustrate the practical utility of the framework, we apply it on data from the French National Health Data System to evaluate the role of methotrexate coprescription in explain- ing the effect of TNFi versus IL-12/23 inhibitor therapy on treatment persistence among psoriatic patients.

stat.AP↗

arXiv2026

Geospatial Foundation Models Capture Health-Relevant Dimensions of Place Beyond Conventional Social Risk Indices

Area-based social risk indices summarize residents' socioeconomic conditions but incompletely capture physical features of place that may affect health. We evaluated whether numerical representations of physical place produced by four geospatial foundation model families from 2022 satellite data explained residual variance in tract-level associations between the Area Deprivation Index, Social Deprivation Index, and Social Vulnerability Index with health outcomes. We used LightGBM to predict variables from the American Community Survey and 40 chronic disease and health-behavior outcomes from CDC PLACES across 82,646 census tracts in the contiguous United States, evaluating performance across 10 held-out states. Among survey variables, models were moderately predictive of some variables including housing type (R-squared up to 0.54) but weak for disability, unemployment, and income disparity. For health outcomes, models explained up to 54% of variance left unexplained by social risk indices, with the largest gains for annual checkups, arthritis, and high blood pressure. Mean total variance explained by geospatial foundation models across the 40 health-related outcomes increased from 0.31 in the smallest tract-size decile to 0.39 in the largest. Geospatial foundation models capture health-relevant features of place not represented by conventional social risk indices and may usefully augment them in epidemiological analyses.

stat.AP↗

arXiv2026

A spatiotemporal negative binomial model with dynamic dispersion: An application to Tuberculosis infections

Tuberculosis (TB) remains a critical public health concern in Brazil, characterized by pronounced spatial heterogeneity and fluctuating temporal volatility. In this paper, we study monthly TB notifications across 61 microregions of Sao Paulo state from 2001 to 2024. To do this, we introduce a negative binomial spatial integer-valued generalized autoregressive conditional heteroskedastic (INGARCH) model featuring jointly dynamic conditional means and time-varying dispersion. To capture inter-regional spillovers, we incorporate both discrete adjacency structures and a novel continuous distance-based formulation leveraging the Matern correlation function. Parameter estimation via conditional maximum likelihood employs a two-step profile-likelihood iterative scheme, demonstrating solid finite-sample performance in simulation studies. Applied to the Sao Paulo TB surveillance data, the framework substantially outperforms standard Poisson and fixed-dispersion spatiotemporal baselines in empirical fit and uncertainty quantification, maintaining nominal 95% predictive coverage across both dense metropolitan centers and rural microregions. Our results reveal marked spatial heterogeneity in baseline incidence, dynamic overdispersion driven by localized outbreaks, and short-range spatial interaction decay. By accurately modeling spatiotemporal volatility, the proposed methodology provides a robust statistical tool to support public health surveillance, policy-making, and resource allocation.

stat.AP↗