Keyao Zhan | Searcharxiv

arXiv2025

Analyzing the Role of Permutation Invariance in Linear Mode Connectivity

It was empirically observed in Entezari et al. (2021) that when accounting for the permutation invariance of neural networks, there is likely no loss barrier along the linear interpolation between two SGD solutions -- a phenomenon known as linear mode connectivity (LMC) modulo permutation. This phenomenon has sparked significant attention due to both its theoretical interest and practical relevance in applications such as model merging. In this paper, we provide a fine-grained analysis of this phenomenon for two-layer ReLU networks under a teacher-student setup. We show that as the student network width $m$ increases, the LMC loss barrier modulo permutation exhibits a double descent behavior. Particularly, when $m$ is sufficiently large, the barrier decreases to zero at a rate $O(m^{-1/2})$. Notably, this rate does not suffer from the curse of dimensionality and demonstrates how substantial permutation can reduce the LMC loss barrier. Moreover, we observe a sharp transition in the sparsity of GD/SGD solutions when increasing the learning rate and investigate how this sparsity preference affects the LMC loss barrier modulo permutation. Experiments on both synthetic and MNIST datasets corroborate our theoretical predictions and reveal a similar trend for more complex network architectures.

stat.ML↗

arXiv2024

Domain Adaptation Optimized for Robustness in Mixture Populations

While domain adaptation methods address data shifts, most assume target populations align with at least one source population, neglecting mixtures that combine sources influenced by factors like demographics. Additional challenges in electronic health record (EHR)-based studies include unobserved outcomes and the need to explain population mixtures using broader clinical characteristics than those in standard risk models. To address these challenges under shifts in both covariate distributions and outcome models, we propose a novel framework: Domain Adaptation Optimized for Robustness in Mixture populations (DORM). Leveraging partially labeled source data, DORM constructs an initial target outcome model under a joint source-mixture assumption. To enhance generalizability to future target populations that may deviate from the joint source-mixture approximation, DORM incorporates a group adversarial learning step to derive a final estimate, optimizing its worst-case performance within a convex uncertainty set built around the initial target model. In addition, this robust domain adaptation procedure is assisted by high-dimensional surrogates that enhance transferability in EHR studies. When a small set of gold-standard or noisy labels is available from the target population, a tuning strategy is implemented to refine the uncertainty set, mitigating conservativeness and further improving performance for the specific target population. Statistical convergence and predictive accuracy of our method are quantified through asymptotic studies. Simulation and real-world studies demonstrate the out-performance of our method over existing approaches.

stat.ME↗