Deep Neural Mixed Models for Nonlinear Longitudinal Prediction with Subject-Specific Heterogeneity

Abstract

Longitudinal health data often exhibit within-subject correlation, subject-specific heterogeneity, and complex nonlinear predictor--response relationships that may not be adequately captured by conventional mixed models. We propose a Deep Generalized Neural Network Mixed Model (DGNMM) that extends the GLMM framework by using a deep neural network to learn nonlinear population-level relationships while incorporating directly trainable, regularized subject-specific effects to capture individual heterogeneity. The model is estimated using prediction-oriented stochastic optimization with separate regularization of the neural-network parameters and subject-specific effects. This formulation allows us to investigate when increased nonlinear representation capacity provides additional predictive benefit in the presence of subject-specific heterogeneity. A simulation study varies population-level nonlinearity and subject-specific heterogeneity and compares DGNMM with GLMM, artificial neural network (ANN), Maity--Pal (MP), and GNMM alternatives for continuous and binary longitudinal outcomes. In the real-data application, DGNMM achieved the lowest MSPE for continuous depression and anxiety prediction, while alternative models remained competitive for binary outcomes. Exploratory subject-level analyses showed heterogeneous predictive gains across individuals, but the magnitude of the estimated subject-specific effects was not significantly associated with predictive gain. Overall, increased nonlinear representation capacity was most beneficial for continuous prediction under nonlinear response structures, while greater subject-specific heterogeneity did not systematically yield larger predictive gains.