Hidden Data Challenges in Clinical Machine Learning

Abstract

Widely reused preprocessing pipelines for electronic health records can encode assumptions about when information became available, what clinical episode a label represents, and what missingness means. These assumptions are rarely audited systematically and may propagate across subsequent studies when preprocessing pipelines are reused or adapted. Using MIMIC-IV as a case study, we identify and quantify violations of three complementary forms of preprocessing alignment, i.e., temporal, semantic, and population alignment, which manifest as temporal leakage, feature-task misalignment, and biased imputation. We show how ambiguities in timestamp semantics, episode definitions, endpoint construction, modality selection, and missingness handling can alter the information available to a model and affect the interpretation of its predictions. Through analyses of MIMIC-IV data and empirical case studies, we demonstrate that such choices may remain largely invisible in aggregate predictive performance while changing the signals on which models rely. We derive practical recommendations for temporally valid, clinically aligned, and reproducible preprocessing of MIMIC-IV and related electronic health record datasets. The code for our experiments, including our proposed preprocessing, is available at: https://github.com/m2l-lab/mimic-iv-preprocess.