Estimating Stability of Individual-Specific Predictions from AI-based prediction models using Bootstrap and Cross-validation Resampling

Abstract

Clinical prediction models are typically developed in a single finite dataset, which is one of many possible datasets that could have been collected. Applying the same model development procedure in another sample will provide different predictions for an individual. The degree to which an individual's prediction varies across development sets is referred to as prediction instability and is not captured by average performance measures. Resampling of the training data offers a practical way to assess instability, but its ability to do so accurately and reliably for different model choices is unclear. In a simulation study, we considered non-parametric, parametric, and out-of-bag bootstrap methods and repeated 10-, 5-, and 2-fold cross-validation (CV) to estimate metrics of prediction instability for Elastic Net logistic regression and Random Forest. The same metrics estimated in independent development samples provided the reference. We assessed individual-level prediction variance, empirical 95% range width, and mean absolute differences from the original prediction and true outcome probability. Instability increased in smaller development samples, was greater for Random Forest, and varied across the probability range. Repeated 5- and 10-fold CV underestimated prediction instability across scenarios. Default non-parametric bootstrap more closely approximated independent sampling for Elastic Net but often overestimated instability for Random Forest, as did parametric bootstrap. Out-of-bag bootstrap and repeated 2-fold CV generally agreed most closely with independent sampling, although no procedure was uniformly best. Resampling procedures should not be used interchangeably, nor should less variable predictions be taken as evidence of greater stability. To estimate prediction instability, prefer procedures that predict on held-out data and limit overlap between training resamples.