arXiv subjects

Bohao Ma

Publications and source records attributed to Bohao Ma.

8 recordsLinked to original sources

arXiv2026

Stochastic Saddle Avoidance Beyond Unit Excitation and Smoothness: A Pathwise Lyapunov-Perron Framework

Unit excitation (UE) is a common assumption in stochastic saddle avoidance: the stochastic error must have a uniformly positive component along every direction, in expectation. This condition gives a direct way to rule out convergence to strict saddles, but it also oversimplifies the actual noise structure, and does not match many stochastic optimization regimes. In overparameterized or interpolation models, the noise may vanish near stationarity. In finite-sum problems, the stochastic gradient noise may lie in a low-dimensional, data-dependent subspace. In these (common) scenarios, UE is naturally not satisfied. In this paper, we prove an abstract almost sure avoidance theorem for stochastic recursions without UE. The theorem replaces UE-type requirements by verifiable pathwise conditions. In applications, these conditions follow, e.g., from local smoothness and finite-moment assumptions under standard i.i.d. sampling, or from the finite-sum structure under without-replacement sampling. Since the stochastically sampled maps generally do not share a fixed point, the celebrated center-stable manifold argument used in deterministic analyses is not directly applicable. Instead, we use a path-dependent change of variables together with a pathwise Lyapunov--Perron-based proof strategy. As applications, we obtain strict saddle avoidance for stochastic mirror descent (including SGD) and for random reshuffling. For nonsmooth composite objectives, we prove avoidance results for a proximal-type stochastic gradient method. Combining these insights with suitable iterate convergence guarantees, this allows establishing convergence to local minimizers of the original objective function.

math.OC↗

arXiv2026

Schattor: Schatten-family methods for deep learning optimization

Modern deep learning optimization features heterogeneous parameter structures, noisy gradients, and highly nonconvex landscapes, posing significant challenges for both algorithm design and theoretical analysis. Motivated by the limitations of SGD and the success of adaptive optimizers, we propose {\it Schattor}, a family of adaptive first-order methods based on Schatten norms. Schattor unifies SGD and the recently proposed matrix-variate adaptive optimizer Muon within a single Schatten-norm-based framework. We establish dimension-free stationarity guarantees for methods in the Schattor family for stochastic matrix optimization problems via a novel matrix martingale moment bound. We also develop multi-block extensions that adaptively balance block-wise optimization progress and prove dimension-free stationarity guarantees in this more general setting.

math.OC↗

arXiv2026

A Single-Loop Regularized Newton Method for Nonconvex-Strongly-Concave Minimax Optimization

For smooth nonconvex-strongly-concave minimax problems, existing second-order methods share a common double-loop structure where the inner maximization is solved to sufficiently high accuracy before each second-order step. First-order methods are often adopted in the inner loops for scalability, but they also undermine the condition-insensitivity of second-order methods, limiting these methods to instances with mild conditioning. To resolve this issue, we propose a novel single-loop framework based on an equivalent regularized minimization reformulation of the original problem. By deriving a new adaptive cubic-quadratic majorization to dynamically absorb the non-Lipschitz components of the reformulated Hessian, we establish a regularized Newton method with robust theoretical guarantees across multiple settings. For deterministic problems, our single-loop method matches the $\mathcal{O}(\varepsilon^{-1.5})$ global iteration complexity of the existing double-loop second-order methods, while automatically achieving a local superlinear rate that is unavailable in existing works due to the inner-loop bottleneck. For the stochastic setting, we achieve $\mathcal{O}(\varepsilon^{-3})$ gradient and $\mathcal{O}(\varepsilon^{-2})$ Hessian complexities by integrating a recursive variance reduction, strictly improving those of the double-loop methods by $\mathcal{O}(\varepsilon^{-0.5})$ factors. In both deterministic and stochastic experiments, our methods significantly outperform the benchmarks, offering substantial speedups over the double-loop methods even under mildly conditioned instances. As a byproduct of our analysis, we close a gap in stochastic second-order methods for nonconvex minimization, where the best known result contains a nontrivial technical issue.

math.OC↗

arXiv2026

Line-search and Adaptive Step Sizes for Nonconvex-strongly-concave Minimax Optimization

In this paper, we propose a novel reformulation of the smooth nonconvex-strongly-concave (NC-SC) minimax problems that casts the problem as a joint minimization. We show that our reformulation preserves not only first-order stationarity, but also global and local optimality, second-order stationarity, and the Kurdyka-{\L}ojasiewicz (KL) property, of the original NC-SC problem, which is substantially stronger than its nonsmooth counterpart in the literature. With these enhanced structures, we design a versatile parameter-free and nonmonotone line-search framework that does not require evaluating the inner maximization. Under mild conditions, global convergence rates can be obtained, and, with KL property, full sequence convergence with asymptotic rates is also established. In particular, we show our framework is compatible with the gradient descent-ascent (GDA) algorithm. By equipping GDA with Barzilai-Borwein (BB) step sizes and nonmonotone line-search, our method exhibits superior numerical performance against the compared benchmarks.

math.OC↗

arXiv2024

A KL-based Analysis Framework with Applications to Non-Descent Optimization Methods

We propose a novel analysis framework for non-descent-type optimization methodologies in nonconvex scenarios based on the Kurdyka-Lojasiewicz property. Our framework allows covering a broad class of algorithms, including those commonly employed in stochastic and distributed optimization. Specifically, it enables the analysis of first-order methods that lack a sufficient descent property and do not require access to full (deterministic) gradient information. We leverage this framework to establish, for the first time, iterate convergence and the corresponding rates for the decentralized gradient method and federated averaging under mild assumptions. Furthermore, based on the new analysis techniques, we show the convergence of the random reshuffling and stochastic gradient descent method without necessitating typical a priori bounded iterates assumptions.

math.OC↗

arXiv2024

Convergence of SGD with momentum in the nonconvex case: A time window-based analysis

The stochastic gradient descent method with momentum (SGDM) is a common approach for solving large-scale and stochastic optimization problems. Despite its popularity, the convergence behavior of SGDM remains less understood in nonconvex scenarios. This is primarily due to the absence of a sufficient descent property and challenges in simultaneously controlling the momentum and stochastic errors in an almost sure sense. To address these challenges, we investigate the behavior of SGDM over specific time windows, rather than examining the descent of consecutive iterates as in traditional studies. This time window-based approach simplifies the convergence analysis and enables us to establish the iterate convergence result for SGDM under the {\L}ojasiewicz property. We further provide local convergence rates which depend on the underlying {\L}ojasiewicz exponent and the utilized step size schemes.

math.OC↗

arXiv2024

Random Reshuffling with Momentum: Complexity Bounds and Last-iterate Convergence

Random reshuffling with momentum (RRM) corresponds to the SGD optimizer with the 'momentum' option enabled, as found in many machine learning libraries such as PyTorch and TensorFlow. Despite its widespread use, the convergence properties of RRM do not seem to be well understood. This work establishes new complexity bounds and asymptotic convergence guarantees for popular versions of RRM using stochastic heavy-ball momentum, Nesterov acceleration, and mini-batches in a general nonconvex setting. In particular, we prove that the base variant of RRM achieves the complexity $O(n^{-1/3}((1-\beta^n)T)^{-2/3})$, where $n$ denotes the number of samples, $\beta \in [0,1)$ is a momentum parameter, and $T$ is the total number of epochs. On the asymptotic side, we show that every accumulation point of the iterates $\{x^k\}_k$ generated by RRM is a stationary point of the problem. For definable objectives -- a broad and common class of functions including, e.g., semialgebraic, globally subanalytic, and log-exp functions -- we strengthen this subsequential result to last-iterate convergence to a single stationary point. Moreover, improved asymptotic complexity bounds are presented that are based on the additional geometric properties of definable functions.

math.OC↗

arXiv2022

Rethinking Quality of Experience for Metaverse Services: A Consumer-based Economics Perspective

The Metaverse is considered to be one prototype of the next-generation Internet, which contains people's expectations for the future world. However, the academic discussion of the Metaverse still mainly focused on the technical system design, and few research studied Metaverse challenges from the perspective of consumers, i.e., Metaverse users. One difficulty is that the analysis from the consumer's perspective requires interdisciplinary theoretical framework and quantifiable Quality of Experience (QoE) measurements. In this article, pioneering from consumers' point of view, we explore an interaction between Metaverse system design and consumer behaviors. Specifically, we rethink QoE and propose an framework that encompasses both the Metaverse service providers (MSPs) and consumer considerations. From the macro perspective, we introduce a joint optimization scheme that simultaneously considers the Metaverse system design, consumers' utility, and profitability of the MSPs. From the micro perspective, we advocate the Willingness-to-Pay (WTP) as an easy-to-implement measurement for future Metaverse system studies. To illustrate the usability of the proposed integrated framework, a use case, i.e., virtual traveling, is presented. We show that our framework can benefit the MSPs in offering competitive and economical service design to consumers while maximizing the profit.

cs.NI↗