A repeated k-fold cross-validation approach for evaluating the instability of clinical prediction models: an empirical comparison to the bootstrap approach
Bootstrap-based methods have been recommended for assessing prediction instability in clinical prediction models, but their performance relative to cross-validation (CV) remains unclear. We propose a CV-based approach for assessing prediction instability and compare it with a bootstrap-based approach in logistic regression and random forest models. We conducted a resampling-based empirical experiment using a clinical cohort of 19,418 emergency department patients. Development samples were generated under events-per-variable (EPV) scenarios of 10, 30, and 50, and results were compared with those from the full dataset. Models were evaluated using bootstrap validation and repeated 5-fold CV; nested CV was used for random forest tuning. Predictive performance was assessed using AUC, calibration slope, and calibration-in-the-large. Prediction instability was quantified using mean absolute prediction error (MAPE). For logistic regression, bootstrap validation and repeated 5-fold CV produced broadly similar discrimination and calibration, especially at higher EPV values. For random forest, apparent performance consistently overestimated empirical discrimination. Bootstrap validation and repeated 5-fold CV gave comparable discrimination, but repeated 5-fold CV produced calibration slope estimates closer to the empirical value. Prediction stability improved as EPV increased for both modelling approaches. At EPV 30, bootstrap-derived MAPE was higher than CV-derived MAPE for both logistic regression (median, 0.042 versus 0.020) and random forest (median, 0.077 versus 0.027). A CV-based approach can assess prediction instability while also providing internally validated performance. These findings support CV-based instability assessment as a practical alternative to bootstrap-based assessment, particularly when comparing instability across multiple modelling algorithms.