arXiv subjects

Sehyun Yun

Publications and source records attributed to Sehyun Yun.

3 recordsLinked to original sources

arXiv2026

ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-wise design fragments adaptation in three ways: capacity is split across narrow low-rank updates, gradient supervision becomes sparse and imbalanced under sparse routing, and execution is decomposed into many small GEMMs. We find that such expert-wise separation is often unnecessary, as subsets of LoRA adapters become functionally similar during fine-tuning, revealing redundancy among expert-specific adapters. Based on this redundancy, we propose ACE (Adapter Consolidation across Experts), which groups redundant experts and replaces their expert-specific adapters with group-shared higher-rank LoRA modules under the same PEFT budget. ACE further introduces grouped adapter execution, which consolidates fragmented expert-wise adapter computations into fewer, larger group-level GEMMs. Across evaluations covering 12 datasets and four MoE backbones, ACE achieves the highest observed mean accuracy among the parameter-matched PEFT methods on the three backbones with complete baseline coverage, while providing $1.31\times$ to $1.48\times$ wall-clock training speedup over expert-wise LoRA without increasing peak memory. Our code is available at https://github.com/UbiquitousAILab/ACE.

cs.LG↗

arXiv2026

EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning

Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation. Existing parameter-efficient fine-tuning (PEFT) methods such as LoRA ignore MoE routing dynamics, leading to suboptimal resource use. We propose EPnG, an adaptive prune-and-grow framework that reallocates LoRA capacity based on expert importance derived from router gate probabilities. EPnG prunes under-utilized experts and expands high-importance experts via rank growth with orthogonal initialization, while maintaining a fixed parameter budget. Across OLMoE and Qwen1.5-MoE, EPnG consistently outperforms LoRA under the same budget and achieves performance comparable to full fine-tuning while updating only 0.55%-0.72% of parameters (up to 140x-180x fewer). These results demonstrate that aligning PEFT with MoE routing yields a more effective and scalable fine-tuning strategy.

cs.LG↗

arXiv2026

Knowledge Integration in Differentiable Models: A Comparative Study of Data-Driven, Soft-Constrained, and Hard-Constrained Paradigms for Identification and Control of the Single Machine Infinite Bus System

Integrating domain knowledge into neural networks is a central challenge in scientific machine learning. Three paradigms have emerged -- data-driven (Neural Ordinary Differential Equations, NODEs), soft-constrained (Physics-Informed Neural Networks, PINNs), and hard-constrained (Differentiable Programming, DP) -- each encoding physical knowledge at different levels of structural commitment. However, how these strategies impact not only predictive accuracy but also downstream tasks such as control synthesis remains insufficiently understood. This paper presents a comparative study of NODEs, PINNs, and DP for dynamical system modeling, using the Single Machine Infinite Bus power system as a benchmark. We evaluate these paradigms across three tasks: trajectory prediction, parameter identification, and Linear Quadratic Regulator control synthesis. Our results yield three principal findings. First, knowledge representation determines generalization: NODE, which learns the system operator, enables robust extrapolation, whereas PINN, which approximates a solution map, restricts generalization to the training horizon. Second, hard-constrained formulations (DP) reduce learning to a low-dimensional physical parameter space, achieving faster and more reliable convergence than soft-constrained approaches. Third, knowledge fidelity propagates to control performance: DP produces controllers that closely match those obtained from true system parameters, while NODE provides a viable data-driven alternative by recovering control-relevant Jacobians with $3-4\%$ relative error and yielding LQR gains within $0.36\%$ of the ground truth. Based on these findings, we propose a practical decision framework for selecting knowledge integration strategies in neural modeling of dynamical systems.

cs.LG↗