Searcharxiv⌕ Search

arXiv subjects

Youmi Suk

Publications and source records attributed to Youmi Suk.

7 recordsLinked to original sources

Causal Machine Learning for Multilevel Studies with Unmeasured Cluster-Level Covariates via Variational Autoencoders

Unmeasured confounders are a key obstacle to accurately estimating treatment effects in observational studies, and in multilevel data, they often arise at the cluster level. Flexible machine learning methods cannot directly resolve such unmeasured confounding. To address this, we propose the variational autoencoder-augmented R-learner (VAE-R), which leverages variational autoencoders (VAEs) to infer unmeasured cluster-level covariates from the within-cluster dependence of outcome and treatment residuals, and uses the resulting adjustments to estimate the conditional average treatment effect (CATE) and the average treatment effect (ATE). The VAE decoder is specified through an offset likelihood so that the latent variables capture only the cluster-level variation left unexplained by measured covariates. VAE-R accommodates unmeasured cluster-level confounders in observational studies and unmeasured cluster-level predictors of the outcome in multisite randomized trials. We also provide data-driven strategies for selecting the two hyperparameters of the VAE, the latent dimension size and the Kullback-Leibler weight. Our simulation study shows that VAE-R nearly eliminates the bias of the ordinary R-learner's CATE and ATE estimates in multilevel observational data and improves their precision in multisite randomized trials. Finally, we illustrate VAE-R with two datasets, one from a multilevel observational study and the other from a multisite randomized trial.

stat.ME↗

Equality, Equity, and Causality in Fairness Research: A Commentary on Cheng (2026)

This is an invited commentary on the Psychometrika focus article "Fairness Issues and Evaluation in Psychometrics and AI/ML: What Can We Learn from Each Field?" by Ying Cheng (2026, doi:10.1017/psy.2026.10110). Cheng offers a systematic comparison between long-standing test fairness and modern algorithmic fairness. Her mapping of the entire testing workflow onto the AI/ML fairness paradigm, rather than only the final selection stage, is a crucial contribution to interdisciplinary fairness research. This commentary extends her discussion by examining two conceptual issues: the distinction between equality and equity, and the role of causality in fairness research. The focus article and this commentary point to directions for future fairness research across the psychometrics and AI/ML communities.

stat.ME↗

Estimating Optimal Dynamic Treatment Regimes for Personalized Education: A Tutorial and Applications with Machine Learning

Optimal dynamic treatment regimes (ODTRs) are increasingly used to generate sequences of personalized recommendations that maximize a final outcome of interest. Although ODTRs are widely used in biostatistics and precision medicine, their applications in education remain limited, especially for longitudinal decision-making. This study translates ODTR methods for single- and multi-stage designs into individualized high school math course recommendations, providing step-by-step guidelines and demonstrations with data from the High School Longitudinal Study of 2009 (HSLS:09). We use Targeted Maximum Likelihood Estimation with ensemble machine learning to estimate ODTRs that maximize students' math achievement and college enrollment, imposing feasibility constraints based on domain knowledge and propensity-score thresholds to ensure recommendations are practically implementable. Results show that unconstrained regimes yield higher estimated values but infeasible recommendations, while feasible regimes produce realistic pathways with stronger empirical and expert-informed support. Finally, we discuss practical considerations for designing ODTRs in education.

stat.AP↗

Distilling Black-Box Machine Learning into a Small, Self-Explaining Language Model for Learning Analytics

Learning analytics increasingly relies on flexible machine learning (ML), but the model opacity and the burden of deployment prevent these tools from reaching educational practice. We propose a two-stage fine-tuning pipeline that distills a fitted black-box estimator and its post hoc interpretation (the mentor) into a small, open-weight large language model (LLM; the mentee) that returns an individual-level estimate and explains in natural language. The design is estimator-agnostic and paired with a faithfulness-first evaluation framework that audits every narration against the attribution it claims to describe. We design a simulation study that separates distillation loss from estimator loss by comparing an oracle mentor with a realistic ML mentor. Given an oracle signal, distillation with a two-billion-parameter LLM model is nearly lossless in recovering the effect surface (r > .90), perfectly ranking the important variables, and citing no spurious covariate. Under a realistic estimator, almost all remaining error originates upstream. We find that fluency is no evidence of correctness since narration quality is independent of signal quality, and decision quality collapses toward the majority action in severely imbalanced settings. Applied to a nationally representative dataset, the pipeline recovers the finding that advanced mathematics coursework benefits students least likely to enroll in four-year college the most, with 98.8% of narrations passing the audit and no fabricated quantities. The result is a single fine-tuned LLM that predicts and explains offline on a commodity laptop, so student records never leave the machine.

cs.HC↗

Identifying Causes of Test Unfairness: Manipulability and Separability

Differential item functioning (DIF) is a widely used statistical notion for identifying items that may disadvantage specific groups of test-takers. These groups are often defined by non-manipulable characteristics, e.g., gender, race/ethnicity, or English-language learner (ELL) status. While DIF can be framed as a causal fairness problem by treating group membership as the treatment variable, this invokes the long-standing controversy over the interpretation of causal effects for non-manipulable treatments. To better identify and interpret causal sources of DIF, this study leverages an interventionist approach using treatment decomposition proposed by Robins and Richardson (2010). Under this framework, we can decompose a non-manipulable treatment into intervening variables. For example, ELL status can be decomposed into English lexical proficiency and instructional English comprehension, each of which influences the outcome through different causal pathways. We formally define separable DIF effects associated with these decomposed components, depending on the absence or presence of item impact, and provide causal identification strategies for each effect. We then apply the framework to biased test items in the SAT and Regents exams. We also provide formal detection methods using causal machine learning methods, namely causal forests and Bayesian additive regression trees, and demonstrate their performance through a simulation study and a real-world application. Finally, we discuss the implications of adopting interventionist approaches in educational testing practices.

stat.ME↗

Generative AI-Based Monte Carlo Simulation for Method Evaluation Using Synthetic Multilevel Data

The role of AI-generated synthetic data has recently been expanded to support realistic Monte Carlo simulations. However, guidance is limited on generating data with multilevel structures and designing simulations based on such data. This study proposes a general framework for AI-based simulation studies to evaluate the predictive performance and parameter recovery of quantitative methods, specifically using multilevel data commonly observed in the social sciences. Our proposed six-stage workflow consists of (i) specifying a method and real data, (ii) training Generative AI with real data, (iii) assessing synthetic data quality, (iv) designing and conducting simulations, (v) evaluating method performance, and (vi) checking robustness. To enhance fidelity in multilevel data generation, we also introduce targeted modifications to diffusion models and Generative Adversarial Networks (GANs). Furthermore, we develop a systematic quality evaluation framework that assesses both within-table and between-table fidelity, and discuss how AI-based simulation designs should differ depending on whether the simulation's objective is predictive performance or parameter recovery. Finally, using empirical multilevel data and multilevel modeling methods, we demonstrate the utility of the proposed AI-based simulation framework. This approach leads to more accurate and honest evaluations of quantitative methods in the real world, unlike traditional simulation studies based on arbitrary simulated scenarios.

stat.ME↗

Separable Effects in Four-Arm and Two-Arm Designs

Robins and Richardson (2010) reformulated mediation analysis by decomposing treatments into multiple components and examining separable effects of each component. While this approach is increasingly popular, existing work has analyzed ``two-arm'' data, where components are strictly bundled and manipulated simultaneously. However, in practice, four-arm data where components are assigned independently are often available. For example, testing accommodations might strictly bundle extra time with a separate session or allow them to be assigned separately. To address this distinction, we propose a general framework for analyzing separable effects in four-arm and two-arm designs. This framework provides distinct identification and estimation strategies for each design. For estimation, we utilize efficient influence function estimators coupled with machine learning and cross-fitting techniques. Additionally, we introduce two falsification tests for key identification assumptions required in the two-arm design by leveraging four-arm data. We investigate the performance of the proposed estimators via a simulation study and demonstrate their application by studying the effect of extended time accommodations using data from the National Assessment of Educational Progress. Ultimately, this separable effects analysis enables practitioners to clearly communicate underlying mechanisms and derive informative policy recommendations.

stat.ME↗