Searcharxiv⌕ Search

arXiv subjects

Mohsen Sadatsafavi

Publications and source records attributed to Mohsen Sadatsafavi.

At least 19 recordsLinked to original sources

Robust Decision Curve Analysis for Survival Risk Prediction Models in the Presence of Informative Censoring

Decision curve analysis (DCA) is recommended for evaluating the clinical utility of prediction models by quantifying net benefit (NB). Conventional NB estimators for survival outcomes rely on the assumption that censoring is non-informative, potentially yielding biased estimates and misleading decisions when this assumption is violated. We propose a robust estimator of survival NB using conditional inverse probability of censoring weighting (CIPCW). In simulation studies, we compared its performance against the conventional approach across censoring mechanisms, censoring model specifications, censoring rates, and decision thresholds, in terms of relative bias, relative root mean squared error (RMSE), and 90% confidence interval coverage. Under non-informative censoring, both estimators performed comparably, with relative bias below 1% and coverage close to the nominal level. Under informative censoring, conventional estimators underestimated the true NB by up to 58% with coverage as low as 1%, whereas the CIPCW estimator had a relative bias of at most 4%, coverage of at least 78% and consistently lower relative RMSE. As a case study, we illustrated its use in validating a breast cancer recurrence-free survival model to guide intensified adjuvant therapy in the German Breast Cancer Study Group (GBSG) cohort. Using the model was the best strategy at some clinically relevant thresholds, but accounting for potential informative censoring changed the best strategy to treating all patients, showing that informative censoring can change the conclusions drawn from DCA. When censoring may be informative, we recommend CIPCW-based NB estimation in DCA for survival outcomes.

stat.ME↗

Evaluation of clinical prediction models: Decomposing performance uncertainty into model development and validation components

There has been growing emphasis on acknowledging, quantifying, and communicating the uncertainty inherent in the output of clinical prediction models. However, allowing for model uncertainty when assessing the performance of an existing model in a new sample (i.e., external validation) has not been sufficiently explored. When validating a model that generates uncertain predictions, the uncertainty of a given performance metric is affected by both model uncertainty and the finiteness of the validation sample (validation uncertainty). For a given performance metric, we show that the total uncertainty of the performance estimate can be decomposed into the uncertainty due to the model and the uncertainty due to the validation. Also, we propose $M^2$, the fraction of the total variance due to model uncertainty, and $V^2$, the fraction of the total variance due to validation uncertainty. These measures serve as complementary scalar indices that can be implemented for common performance metrics and visualizations such as calibration plots using simple Monte Carlo algorithms. Critically, our case study shows that with the same model and the same validation sample, the contribution of uncertainty sources is highly variable across performance metrics ($M^2$ varying from 12% for the Brier score to 78% for the calibration slope). This indicates that stakeholders who prioritize different aspects of performance (e.g., calibration versus clinical utility) might arrive at different decisions on whether future efforts should be directed towards creating a more reliable model or procuring a larger validation sample. In conclusion, external validation studies can readily be amended to incorporate model uncertainty, and we recommend that such studies report total variance as well as the relative share of variance components for all performance metrics.

stat.ME↗

Bayesian additive regression trees for evaluating treatment benefit predictors using observational data

A treatment benefit predictor (TBP) is an algorithm that maps a patient's characteristics to their putative benefit from a given treatment, which can be used to inform treatment decisions. However, a TBP must be evaluated in the target population before being adopted for patient care. When only observational data are available, evaluating TBPs requires standard causal identification assumptions, as treatment assignment in such settings is not random. We obtain the posterior distributions of predictive performance measures to evaluate prespecified TBPs using observational data, by taking advantage of Bayesian additive regression trees (BART). We illustrate the evaluation of TBPs using selected measures and graphical visualizations: the concentration of benefit ($C_b$) index and the moderate calibration curve. Simulation studies of binary and continuous outcomes settings, including balanced and imbalanced treatment allocation in the binary setting, establish the validity of the proposed approach. In a case study, we use this approach to assess a TBP for systemic antibiotic therapy for patients with chronic obstructive pulmonary disease (COPD). We show that the constructed TBP does not in fact make calibrated predictions, because it both makes optimistic predictions of risks and exaggerates the risk reduction due to treatment. We conclude that flexible Bayesian approaches have the potential to assess TBPs, offering opportunities for flexible model specifications, adjustment for confounding, and uncertainty characterization.

stat.ME↗

Non-parametric assessment of the calibration of individualized treatment effects

An important aspect of the performance of algorithms that predict individualized treatment effects (ITE) is moderate calibration, i.e., the average treatment effect among individuals with predicted treatment effect of z being equal to z. The assessment of moderate calibration is challenging on two fronts: counterfactual responses are unobserved, and quantifying the conditional response function for models that generate continuous predicted values requires regularization. Perhaps because of these challenges, there is currently no inferential method for the null hypothesis that an ITE model is moderately calibrated in a population. In this work, we propose non-parametric methods for the assessment of moderate calibration of ITE models for binary outcomes using data from a randomized trial. These methods simultaneously resolve both challenges, resulting in novel graphical, numerical, and inferential methods for the assessment of moderate calibration. The key idea is to formulate a stochastic process for the cumulative prediction errors that obeys a functional central limit theorem, enabling the use of the properties of Brownian motion for asymptotic inference. We propose two approaches to construct this process from a sample: a conditional approach that relies on predicted risks (often an output of ITE models), and a marginal approach based on replacing the cumulative conditional moments with their marginal counterparts. Numerical simulations confirm the desirable properties of both approaches and their ability to detect miscalibration of different forms. We use a case study to provide suggestions on graphical presentation and the interpretation of results. Moderate calibration of predicted ITEs can be assessed without requiring regularization techniques or making assumptions about the functional form of treatment response. The accompanying cumulcalib R package implements this method.

stat.ME↗

Transportability of Prognostic Markers: Rethinking Common Practices through a Sufficient-Component-Cause Perspective

Transportability, the ability to maintain performance across populations, is a desirable property of markers of clinical outcomes. However, empirical findings indicate that markers often exhibit varying performances across populations. For prognostic markers that are advertised as predictive risk equations for an outcome of interest, oftentimes a form of updating is required when the equation is transported to populations with different outcome prevalences. Here, we revisit transportability of prognostic markers through the lens of the foundational framework of sufficient component causes (SCC). We argue that transporting a marker "as is" implicitly assumes predictive values are transportable, whereas conventional prevalence adjustment shifts the locus of transportability to accuracy metrics (sensitivity and specificity). Using a minimalist SCC framework that decomposes risk prediction into broad causal constituents, we show that both approaches rely on strong assumptions about the stability of cause distributions. An SCC framework instead invites making transparent assumptions about how different causes vary across populations, leading to different transportation methods. For example, in the absence of any external information other than outcome prevalence, an impartial perspective can assume all causes are responsible for change in prevalence, leading to a new form of marker transportation. Numerical experiments demonstrate that different transportability assumptions lead to varying degrees of information loss, depending on the distribution of causes across populations. An SCC perspective challenges common assumptions and practices for marker transportability, and results in novel transportability methods based on explicit assumptions on how different causes vary across populations.

stat.AP↗

Value-of-Information Analysis for External Validation of Risk Prediction Models in Multicenter Studies and Systematic Reviews

External validation studies have finite sample sizes, creating uncertainty about whether a prediction model's Net Benefit (NB) exceeds default strategies' NB. The expected value of perfect information (EVPI) quantifies consequences of uncertainty. Current EVPI methods focus on single studies, ignoring between-center heterogeneity. We extend EVPI and expected value of partial perfect information (EVPPI) to account for between-cluster heterogeneity in multicenter studies and meta-analyses. We distinguish between the global and local optimal strategy and between observed and unobserved clusters. We define EVPIglobal, EVPIcluster_j, EVPIcluster, and EVPPIcluster,prevalence, implemented in the MetaNB R package, and illustrate them using a systematic review across 36 centers of the ADNEX model for ovarian cancer diagnosis. Assuming one global decision regarding ADNEX adoption, there is no need for further data to confirm ADNEX is superior overall (EVPIglobal 0). Meta-analysis borrows information across observed clusters, resulting in consistent local superiority of ADNEX and nonzero but typically lower EVPIcluster_j than when considering local data alone. There is 0.03 probability default strategies are superior in unobserved centers. Eliminating uncertainty on performance and prevalence in each (EVPIcluster) would gain 1134 net avoided false positives (FP) per year, assuming 350000 tumors annually with 20% malignancies. Determining only local prevalence with certainty (EVPPIcluster, prevalence) would gain net 158 avoided FP per year. EVPI extensions disentangle sources of uncertainty and quantify the need for further validation to determine the global or locally optimal strategy. Considering uncertainty and heterogeneity in clinical utility across clusters is essential to decide whether additional validation studies are warranted.

stat.AP↗

Progression to the mean: A comparison of Bayesian clinical prediction models outputting the posterior mean versus conventional plug-in predictions

Clinical prediction models provide predictions for individuals, typically expressed as point estimates derived from a deterministic function, such as a logistic regression equation. Such 'plug-in' predictions hide inherent uncertainty. In contrast, Bayesian methods offer a coherent mechanism for uncertainty propagation, and allow the computation of the posterior mean as the measure of centrality of choice for clinical decision-making. However, Bayesian methods are not widely utilised in predictive analytics for healthcare. We investigated the feasibility and performance of a Bayesian adaptation of the commonly used frequentist framework for risk prediction modelling. We assessed (i) the use of shrinkage priors with complementary features (simplicity, user input, and automatic shrinkage) that enable Laplace/normal approximation of the posterior, and (ii) exact and approximate methods for efficient computation of the posterior mean. Using examples and simulations, we demonstrate that this Bayesian approach is feasible and improves predictive performance, while enabling uncertainty quantification with suitable coverage. In small-to-medium sample sizes, the gain in clinical utility by using the posterior mean over plug-in predictions was equivalent to the gain from using a noticeably larger sample size. Adapting the widely used parametric regression methods to an approximate Bayesian framework for prediction modelling is both pragmatic and clinically advantageous.

stat.ME↗

What Medicine Taught Us About Fairness and What It Missed: Lessons from Reconsidering Race-Specific Lung Function Reference Algorithms

Since 2019, medical societies have reconsidered race-specific clinical equations often in parallel to and largely independent from algorithmic fairness research. Focusing on lung function reference algorithms that affect medical care, insurance, and employment for hundreds of millions globally, we analyze the transition from race-specific GLI-2012 to race-averaged GLI-Global through a fairness lens. Drawing on historical context, citation analysis, and quantitative evaluation, we show (i) limited cross-citation between FAccT and clinical guideline revision efforts; (ii) that GLI-Global implicitly encodes assumptions about social determinants of health, behaving as if ~62% of the Black-White gap in FEV1 is exposure-related; and (iii) clinical validation studies operationalized a sufficiency-like fairness criterion long before its formalization in fairness literature, while neglecting foundational results such as the impossibility theorem has led to inefficiencies in clinical research. Overall, our analysis highlights the value of deeper, mutually beneficial engagement between medical and fairness communities and the public to accelerate progress toward equitable healthcare algorithms.

cs.CY↗

Evaluating Treatment Benefit Predictors using Observational Data: Contending with Identification and Confounding Bias

A treatment benefit predictor (TBP) is a function that maps patient characteristics to an estimate of the treatment benefit for that patient. Such predictors support optimizing individualized treatment decisions, which are central to precision medicine. However, evaluating the predictive performance of a TBP is challenging, as this often must be conducted in a sample where treatment assignment is not random. After briefly reviewing several metrics for evaluating TBPs, we show conceptually how to evaluate a pre-specified TBP using observational data from the target population, for a binary treatment decision at a single time point. We exemplify with a particular measure of discrimination (the concentration of benefit index) and a particular measure of calibration (the moderate calibration curve). The population-level definitions of these metrics involve the latent treatment benefit variable, but we show identification by re-expressing the respective estimands in terms of the distribution of observable data only. We also show that in the absence of full confounding control, bias propagates in a more complex manner than when targeting more commonly encountered estimands. We find the patterns of biases are often unpredictable, and general intuition about the direction of bias in causal effect estimates does not hold in the present context.

stat.ME↗

Sequential sample size calculations and learning curves safeguard the robust development of a clinical prediction model for individuals

When prospectively developing a new clinical prediction model (CPM), fixed sample size calculations are typically conducted before data collection based on sensible assumptions. But if the assumptions are inaccurate the actual sample size required to develop a reliable model may be very different. To safeguard against this, adaptive sample size approaches have been proposed, based on sequential evaluation of a models predictive performance. Aim: illustrate and extend sequential sample size calculations for CPM development by (i) proposing stopping rules based on minimising uncertainty (instability) and misclassification of individual-level predictions, and (ii) showcasing how it safeguards against inaccurate fixed sample size calculations. Using the sequential approach repeats the pre-defined model development strategy every time a chosen number (e.g., 100) of participants are recruited and adequately followed up. At each stage, CPM performance is evaluated using bootstrapping, leading to prediction and classification stability statistics and plots, alongside optimism-adjusted measures of calibration and discrimination. Our approach is illustrated for development of acute kidney injury using logistic regression CPMs. The fixed sample size calculation, based on perceived sensible assumptions suggests recruiting 342 patients to minimise overfitting; however, the sequential approach reveals that a much larger sample size of 1100 is required to minimise overfitting (targeting population-level stability). If the stopping rule criteria also target small uncertainty and misclassification probability of individual predictions, the sequential approach suggests an even larger sample size (n=1800). Our sequential sample size approach allows users to dynamically monitor individual-level prediction and classification instability and safeguard against using inaccurate assumptions.

stat.ME↗

Expected value of sample information calculations for risk prediction model development

Risk prediction models are often advertised as deterministic functions that map covariates to predicted risks. However, they are typically trained using finite samples, and as such, their predictions are inherently uncertain. This uncertainty has been addressed in terms of uncertainty around metrics of model performance (e.g., confidence intervals around c-statistic), as well as uncertainty or instability of predictions. Correspondingly, sample size calculations for model development studies target the precision of estimates of summary statistics and the stability of predictions. However, when evaluating the clinical utility of a model (as in Net Benefit (NB) calculations in decision curve analysis), statistical inference is less relevant. From a decision-theoretic perspective, the finite size of the sample results in utility loss due to the discrepancy between the fitted model and the correct model. From this perspective, procuring more development data is associated with an expected gain in the utility of using the model. In this work, we define the Expected Value of Sample Information (EVSI) as the expected gain in clinical utility, defined in NB terms, by procuring an additional development sample of a given size. We propose a bootstrap-based algorithm for EVSI computations and demonstrate its feasibility and face validity in a case study. We conclude that decision-theoretic metrics can complement classical inferential methods when designing studies aimed at developing risk prediction models.

stat.ME↗

Bayesian sample size calculations for external validation studies of risk prediction models

Contemporary sample size calculations for external validation of risk prediction models require users to specify fixed values of assumed model performance metrics alongside target precision levels (e.g., 95% CI widths). However, due to the finite samples of previous studies, our knowledge of true model performance in the target population is uncertain, and so choosing fixed values represents an incomplete picture. As well, for net benefit (NB) as a measure of clinical utility, the relevance of conventional precision-based inference is doubtful. In this work, we propose a general Bayesian framework for multi-criteria sample size considerations for prediction models for binary outcomes. For statistical metrics of performance (e.g., discrimination and calibration), we propose sample size rules that target desired expected precision or desired assurance probability that the precision criteria will be satisfied. For NB, we propose rules based on Optimality Assurance (the probability that the planned study correctly identifies the optimal strategy) and Value of Information (VoI) analysis. We showcase these developments in a case study on the validation of a risk prediction model for deterioration of hospitalized COVID-19 patients. Compared to the conventional sample size calculation methods, a Bayesian approach requires explicit quantification of uncertainty around model performance, and thereby enables flexible sample size rules based on expected precision, assurance probabilities, and VoI. In our case study, calculations based on VoI for NB suggest considerably lower sample sizes are needed than when focusing on precision of calibration metrics.

stat.AP↗

A general sample size framework for developing or updating a clinical prediction model

Aims: To propose a general sample size framework for developing or updating a clinical prediction model using any statistical or machine learning method, based on drawing samples from anticipated posterior distributions and targeting assurance in predictive performance. Methods: Users provide a reference model (eg, matching outcome incidence, predictor weights and c-statistic of previous models), and a (synthetic) dataset reflecting the joint distribution of candidate predictors in the target population. Then a fully simulation-based approach allows the impact of a chosen development sample size and modelling strategy to be examined. This generates thousands of models and, by applying each to the target population, leads to posterior distributions of individual predictions and model performance (degradation) metrics, to inform required sample size. To improve computation speed for penalised regression, we also propose a one-sample Bayesian analysis combining shrinkage priors with a likelihood decomposed into sample size and Fisher's information. Results: The framework is illustrated when developing pre-eclampsia prediction models using logistic regression (unpenalised, uniform shrinkage, lasso or ridge) and random forests. We show it encompasses existing sample size calculation criteria whilst providing model assurance probabilities, instability metrics and degradation statistics about calibration, discrimination, clinical utility, prediction error and fairness. Crucially, the required sample size depends on the users' key estimands and planned model development or updating approach. Conclusions: The framework generalises existing sample size proposals for model development by utilising anticipated posterior distributions conditional on a chosen sample size and development strategy. This informs the sample size required to target appropriate model performance.

stat.ME↗

Recent Advances, Applications and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2024 Symposium

The fourth Machine Learning for Health (ML4H) symposium was held in person on December 15th and 16th, 2024, in the traditional, ancestral, and unceded territories of the Musqueam, Squamish, and Tsleil-Waututh Nations in Vancouver, British Columbia, Canada. The symposium included research roundtable sessions to foster discussions between participants and senior researchers on timely and relevant topics for the ML4H community. The organization of the research roundtables at the conference involved 13 senior and 27 junior chairs across 13 tables. Each roundtable session included an invited senior chair (with substantial experience in the field), junior chairs (responsible for facilitating the discussion), and attendees from diverse backgrounds with an interest in the session's topic.

cs.LG↗

A decomposition of Fisher's information to inform sample size for developing fair and precise clinical prediction models -- Part 2: time-to-event outcomes

Background: When developing a clinical prediction model using time-to-event data, previous research focuses on the sample size to minimise overfitting and precisely estimate the overall risk. However, instability of individual-level risk estimates may still be large. Methods: We propose a decomposition of Fisher's information matrix to examine and calculate the sample size required for developing a model that aims for precise and fair risk estimates. We propose a six-step process which can be used before data collection or when an existing dataset is available. Steps (1) to (5) require researchers to specify the overall risk in the target population at a key time-point of interest; an assumed pragmatic 'core model' in the form of an exponential regression model; the (anticipated) joint distribution of core predictors included in that model; and the distribution of any censoring. Results: We derive closed-form solutions that decompose the variance of an individual's estimated event rate into Fisher's unit information matrix, predictor values and total sample size; this allows researchers to calculate and examine uncertainty distributions around individual risk estimates and misclassification probabilities for specified sample sizes. We provide an illustrative example in breast cancer and emphasise the importance of clinical context, including risk thresholds for decision making, and examine fairness concerns for pre- and post-menopausal women. Lastly, in two empirical evaluations, we provide reassurance that uncertainty interval widths based on our approach are close to using more flexible models. Conclusions: Our approach allows users to identify the (target) sample size required to develop a prediction model for time-to-event outcomes, via the pmstabilityss module. It aims to facilitate models with improved trust, reliability and fairness in individual-level predictions.

stat.ME↗

Identification of distributions for risks based on the first moment and c-statistic

We show that for any family of distributions with support on [0,1] with strictly monotonic cumulative distribution function that has no jumps and is quantile-identifiable (i.e., any two distinct quantiles identify the distribution), knowing the first moment and c-statistic is enough to identify the distribution. The derivations motivate numerical algorithms for mapping a given pair of expected value and c-statistic to the parameters of specified two-parameter distributions for probabilities. We implemented these algorithms in R and in a simulation study evaluated their numerical accuracy for common families of distributions for risks (beta, logit-normal, and probit-normal). An area of application for these developments is in risk prediction modeling (e.g., sample size calculations and Value of Information analysis), where one might need to estimate the parameters of the distribution of predicted risks from the reported summary statistics.

stat.ME↗

The expected value of sample information calculations for external validation of risk prediction models

In designing external validation studies of clinical prediction models, contemporary sample size calculation methods are based on the frequentist inferential paradigm. One of the widely reported metrics of model performance is net benefit (NB), and the relevance of conventional inference around NB as a measure of clinical utility is doubtful. Value of Information methodology quantifies the consequences of uncertainty in terms of its impact on clinical utility of decisions. We introduce the expected value of sample information (EVSI) for validation as the expected gain in NB from conducting an external validation study of a given size. We propose algorithms for EVSI computation, and in a case study demonstrate how EVSI changes as a function of the amount of current information and future study's sample size. Value of Information methodology provides a decision-theoretic lens to the process of planning a validation study of a risk prediction model and can complement conventional methods when designing such studies.

stat.AP↗

Non-parametric inference on calibration of predicted risks

Moderate calibration, the expected event probability among observations with predicted probability z being equal to z, is a desired property of risk prediction models. Current graphical and numerical techniques for evaluating moderate calibration of risk prediction models are mostly based on smoothing or grouping the data. As well, there is no widely accepted inferential method for the null hypothesis that a model is moderately calibrated. In this work, we discuss recently-developed, and propose novel, methods for the assessment of moderate calibration for binary responses. The methods are based on the limiting distributions of functions of standardized partial sums of prediction errors converging to the corresponding laws of Brownian motion. The novel method relies on well-known properties of the Brownian bridge which enables joint inference on mean and moderate calibration, leading to a unified "bridge" test for detecting miscalibration. Simulation studies indicate that the bridge test is more powerful, often substantially, than the alternative test. As a case study we consider a prediction model for short-term mortality after a heart attack, where we provide suggestions on graphical presentation and the interpretation of results. Moderate calibration can be assessed without requiring arbitrary grouping of data or using methods that require tuning of parameters. An accompanying R package implements this method (see https://github.com/resplab/cumulcalib/).

stat.ME↗