Transporting Predictions via Double Machine Learning: Predicting Partially Unobserved Students' Outcomes

Abstract

Educational policymakers often lack data on student outcomes where standardized tests were not administered. Machine learning can predict unobserved outcomes in target populations using source population data. However, covariate distribution differences between populations reduce model transportability, potentially decreasing predictive accuracy and introducing bias. We propose using double machine learning for covariate-shift weighted models. First, we estimate overlap scores -- the probability an observation belongs to the source dataset given covariates. Second, balancing weights, defined as density ratios of target-to-source membership probabilities, reweight individual observations' contributions to the loss function in target outcome prediction models. This downweights source observations less similar to the target population, allowing predictions to rely more on observations with greater overlap. Consequently, predictions become more transportable under covariate shift. We illustrate this framework using student standardized financial literacy scores (FLS) data. Using Bayesian Additive Regression Trees (BART), we predict missing FLS. We find minimal predictive performance differences between weighted and unweighted models, suggesting limited covariate shift in our setting. Nonetheless, our approach provides a principled framework for addressing covariate shift and is broadly applicable to predictive modeling in social and health sciences, where source-target population differences are common.