arXiv subjects

Yaoshuai Ma

Publications and source records attributed to Yaoshuai Ma.

4 recordsLinked to original sources

arXiv2026

Muse: Representation Geometry of Muon Beyond Normalized Momentum

Muon-style optimizers apply a polar map to matrix momentum, but their updates also depend on the representation of each parameter block before orthogonalization. We study this representation choice as a form of optimizer geometry and introduce {\method}, a family of Muon-style optimizers that shares the same momentum rule and Newton--Schulz backend across native, nearest-square, skinny, and vector representations. Each Frobenius-isometric representation induces a distinct polar steepest-descent geometry, in which the shorter matrix dimension determines the number of supported singular channels, the pullback scaling, and the constants in stochastic nonconvex convergence bounds. In a teacher--student model, curvature collapse and an isotropic Marchenko--Pastur spectral profile connect early-stage dissipation to the represented nuclear-to-squared-Frobenius norm ratio. Pretraining experiments on LLaMA2-130M and LLaMA2-600M, together with fixed-momentum diagnostics, show that balanced non-native representations can match the performance of the native representation, whereas reducing the shorter dimension weakens the scaling and singular-channel support, leading to behavior that increasingly resembles normalized momentum.

cs.LG↗

arXiv2026

FAB: A First-Order AB-based Gradient Algorithm for Distributed Bilevel Optimization over Time-Varying Directed Graphs

Distributed optimization over time-varying directed graphs has shown promising performance in addressing challenges posed by complex communication constraints in real-world scenarios. In many practical settings, however, the direct application of distributed optimization algorithms encounters additional difficulties, most notably hyperparameter tuning, which our empirical observations suggest can be effectively mitigated by integrating bilevel optimization. Motivated by these findings, we study distributed bilevel optimization over time-varying directed networks, a problem that remains largely unexplored due to the compounded challenges arising from consensus bias in dynamic unbalanced communication and the nested optimization structure. In this work, we propose a fully first-order distributed gradient-based algorithm that integrates the Push-Pull (also known as AB) communication strategy with a value function-based penalty method and establish its non-asymptotic convergence properties. Notably, a simplified variant of our analysis framework for nonconvex single-level distributed optimization establishes a convergence rate for the Push-Pull algorithm, thereby resolving an open question concerning its convergence over time-varying directed graphs. Empirical evaluations across diverse tasks, including hyperparameter tuning, data hyper-cleaning, and reinforcement learning, validate the effectiveness and efficiency of the proposed algorithm.

math.OC↗

arXiv2026

SUN-DSBO: A Structured Unified Framework for Nonconvex Decentralized Stochastic Bilevel Optimization

Decentralized stochastic bilevel optimization (DSBO) is a powerful tool for various machine learning tasks, including decentralized meta-learning and hyperparameter tuning. Existing DSBO methods primarily address problems with strongly convex lower-level objective functions. However, nonconvex objective functions are increasingly prevalent in modern deep learning. In this work, we introduce SUN-DSBO, a Structured Unified framework for Nonconvex DSBO, in which both the upper- and lower-level objective functions may be nonconvex. Notably, SUN-DSBO offers the flexibility to incorporate decentralized stochastic gradient descent or various techniques for mitigating data heterogeneity, such as gradient tracking (GT). We demonstrate that SUN-DSBO-GT, an adaptation of the GT technique within our framework, achieves a linear speedup with respect to the number of agents. This is accomplished without relying on restrictive assumptions, such as gradient boundedness or any specific assumptions regarding gradient heterogeneity. Numerical experiments validate the effectiveness of our method.

math.OC↗

arXiv2021

FROG-measurement based phase retrieval for analytic signals

While frequency-resolved optical gating (FROG) is widely used in characterizing the ultrafast pulse in optics, analytic signals are often considered in time-frequency analysis and signal processing, especially when extracting instantaneous features of events. In this paper we examine the phase retrieval (PR) problem of analytic signals in $\Bbb{C}^N$ by their FROG measurements. After establishing the ambiguity of the FROG-PR of analytic signals, we found that the FROG-PR of analytic signals of even lengths is different from that of analytic signals of odd lengths, and it is also different from the case of $B$-bandlimited signals with $B \leq N/2$. The existing approach to bandlimited signals can be applied to analytic signals of odd lengths, but it does not apply to the even length case. With the help of two relaxed FROG-PR problems and a translation technique, we develop an approach to FROG-PR for the analytic signals of even lengths, and prove that in this case the generic analytic signals can be uniquely (up to the ambiguity) determined by their $(3N/2+1)$ FROG measurements.

eess.SP↗