Searcharxiv⌕ Search

arXiv subjects

Anthony Man-Cho So

Publications and source records attributed to Anthony Man-Cho So.

At least 19 recordsLinked to original sources
arXiv2026

A Stochastic Riemannian Alternating Descent Ascent Method for Nonsmooth Composite Expectation Optimization on Riemannian Manifolds

In this paper, we consider a class of Riemannian nonsmooth composite expectation optimization problems, which arises in various machine learning, signal processing, and statistics applications. Noting that these problems admit structured minimax reformulations, we propose an efficient algorithm, named stochastic Riemannian alternating descent ascent method with recursive momentum (StoRADA-RM), to tackle them. StoRADA-RM performs one or multiple Riemannian stochastic gradient descent steps and then a proximal gradient ascent step at each iteration. To compute the Riemannian stochastic gradient, we propose a vector transport-free recursive momentum estimator that requires only $O(1)$ stochastic gradient evaluations per iteration. We prove that StoRADA-RM returns an $ε$-Riemannian-stochastic-stationary point of a given problem in the said class in $O(ε^{-3})$ iterations while making $O(ε^{-3})$ calls to a stochastic first-order oracle (SFO). Both the iteration complexity and SFO complexity bounds are the best known in the literature for the said class of problems. The latter even matches the optimal lower bound for smooth nonconvex optimization with stochastic first-order algorithms. We then present numerical results on sparse principal component analysis and coordinate-independent sparse estimation to demonstrate the superior performance of our proposed method.

math.OC↗
arXiv2026

A Unified Framework for Iterate Convergence of Bregman Proximal Methods

Iterate convergence of Bregman proximal methods (BPMs) has long remained open, especially for nonconvex objectives. Recently, \citet{chen2026skl} made progress by establishing iterate convergence for a BPM via the so-called scaled Kurdyka-Łojasiewicz (SKŁ) property, but only for the Shannon entropy kernel and linearly constrained problems. In this paper, we develop a unified iterate convergence framework that applies to a broad group of kernels and composite objective functions. Our approach extends the analytical tools in \cite{chen2026skl}, in particular the SKŁ property, which plays a central role in ensuring convergence of the generated sequences. By introducing kernel-dependent parameterization functions, we show that the extended SKŁ property holds for all continuous subanalytic functions, particularly when the kernel has a closed domain. We then verify that the assumptions of the framework are satisfied by standard BPMs under mild regularity conditions, thereby establishing their iterate convergence for a wide range of objective functions. Furthermore, based on the parameterization functions, we show that the continuous-time BPM (mirror flow) converges to a stationary point for o-minimal definable objective functions, yielding the first trajectory convergence result for mirror flow without imposing convexity assumptions on the objective function or isolation assumptions on stationary points. Taken together, these discrete- and continuous-time convergence results provide a unified trajectory convergence theory for BPMs.

math.OC↗
arXiv2026

On the Iterate Convergence of Bregman Projected Gradient Method

The iterate convergence of \textit{Bregman projected gradient method} (BPGM) has remained a long-standing open problem, especially for the widely adopted Shannon entropy kernel. Existing convergence results are often limited, relying on Lipschitz continuity of the kernel's gradient or restrictive conditions on the objective function. In this paper, we develop a novel convergence analysis framework for the BPGM with the Shannon entropy kernel, yielding strong convergence results for a broad class of objective functions under linear constraints. The cornerstone of our framework is a new concept called \textit{scaled Kurdyka-Łojasiewicz} (SKŁ) property, which captures the local growth behavior of a function under the Bregman geometry. We show that the SKŁ property ensures the iterate convergence of BPGM and holds for all continuous subanalytical functions. Furthermore, we prove that the BPGM sequence exhibits linear convergence if the problem possesses an SKŁ exponent of $1/2$. We then furnish the examples of functions with the SKŁ exponent $1/2$ by proving that the SKŁ exponent $1/2$ is implied by the KŁ exponent $1/2$ under strict complementarity and local Lipschitz continuity of the objective's gradient. Building on these novel results, our work takes a first step towards resolving the open problem of BPGM iterate convergence.

math.OC↗
arXiv2026

On the hardness of deterministic second-order optimization of functions with Lipschitz gradients

We show that no deterministic zero-respecting algorithm (resp., (general) deterministic algorithm) can compute Goldstein approximate second-order stationary points of functions with Lipschitz continuous gradients within a finite number of (resp., no more than $n-3$ with $n$ being the input dimension) second-order oracle calls. This, among other consequences, shows that deterministic second-order weakly convex optimization is intractable.

math.OC↗
arXiv2026

On computing Goldstein approximate second-order stationary points of structured nonsmooth nonconvex programs

In this paper, we exhibit a randomized first-order algorithm to compute Goldstein approximate second-order stationary points of $L$-smooth functions, using tools from randomized smoothing. The algorithm has oracle complexity $\widetilde{O}({ n^2}/{\varepsilon^9}+{ n^3}/{\varepsilon^7})$, where $n=1,2,\ldots$ is the input dimension and $\varepsilon>0$ is the (common) tolerance. We also present extensions to weakly convex functions and applications to bilevel optimization.

math.OC↗
arXiv2026

Last-Iterate Convergence of Single-Loop Stochastic Methods for Constrained Convex-Concave Minimax Problems

In this paper, we study last-iterate convergence of stochastic first-order methods for constrained smooth convex--concave minimax optimization under the standard bounded-variance stochastic oracle. A fundamental challenge is that the last iterates of vanilla stochastic extragradient (S-EG) and stochastic optimistic gradient descent--ascent (S-OGDA) may fail to converge in the presence of stochastic gradient noise, even for simple bilinear problems. To overcome this difficulty, we introduce a simple perturbation framework that regularizes the original convex--concave problem into a strongly convex--strongly concave one. Applying S-EG and S-OGDA to the perturbed problem yields two simple single-loop methods, referred to as perturbed S-EG (PS-EG) and perturbed S-OGDA (PS-OGDA). We establish last-iterate convergence by first deriving convergence in terms of the squared distance to the saddle point of the perturbed problem and then translating this estimate into guarantees for the restricted primal--dual gap. Based on this framework, we establish two types of convergence guarantees. When the optimization horizon is known \emph{a priori}, both PS-EG and PS-OGDA achieve an $\mathcal{O}(T^{-1/4})$ last-iterate convergence rate for the restricted primal--dual gap, which coincides with the standard primal--dual gap on compact feasible domains. When the optimization horizon is unknown, we develop an anytime variant based on diminishing perturbations and diminishing stepsizes. For general closed convex feasible sets, both PS-EG and PS-OGDA achieve an $\mathcal{O}(T^{-1/5})$ last-iterate convergence rate for the restricted primal--dual gap. Furthermore, in the unconstrained setting, PS-EG admits a sharper $\mathcal{O}(T^{-1/4})$ anytime convergence rate in terms of the gradient norm.

math.OC↗
arXiv2026

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning

Can language models improve their reasoning performance without external rewards, using only their own sampled responses for training? We show that they can. We propose Self-evolving Post-Training (SePT), a simple post-training method that alternates between self-generation and training on self-generated responses. It repeatedly samples questions, uses the model itself to generate responses under a specified sampling temperature, and then trains the model on the self-generated data. In this self-training loop, we use an online data refresh mechanism, where each new batch is generated by the most recently updated model. Across six math reasoning benchmarks, SePT improves a strong no-training baseline, defined as the untuned base model evaluated at its best swept decoding temperature, on several tested models. Additional ablations demonstrate the importance of online data refresh and temperature dynamics. Overall, our results identify a practical regime where reasoning can be improved using self-generated supervision alone. Our code is available at https://github.com/ElementQiii/SePT.

cs.LG↗
arXiv2026

Learning Graph Topology with Functional Priors via Bilevel Optimization

Learning graph topology of complex networks is challenging due to limited data availability and imprecise data models. Different from prior works that focus on structural priors with explicit control on macroscopic properties such as sparsity, this paper proposes a novel functional prior approach for graph topology learning. We postulate that complex networks are inherently optimized to perform a certain task (e.g., social networks specialize at optimizing a welfare function, biological networks are resilient towards node/edge deletion), which can be incorporated as a regularizer to assist in graph learning. Mathematically, we formulate a bilevel optimization problem where the lower-level problem solves the associated task on a candidate graph topology and the upper-level problem trades off between data fitting and task performance. We design a two-timescale gradient descent (TTGD) algorithm and show that under verifiable conditions, it finds a stationary point to the bilevel graph learning problem with a sublinear convergence rate. We provide theoretical insights on the graph topology learned from the functional priors and show that the resulting regularizers subsume a broad class of graph filter regularizers, including polynomial graph regularizers as special cases. We show via extensive experiments on synthetic and real datasets that the proposed formulation gives rise to reliable estimates of graph topology, even with insufficient data.

eess.SP↗
arXiv2026

Dual Quaternion SE(3) Synchronization with Recovery Guarantees

Synchronization over the special Euclidean group SE(3) aims to recover absolute poses from noisy pairwise relative transformations and is a core primitive in robotics and 3D vision. Standard approaches often require multi-step heuristic procedures to recover valid poses, which are difficult to analyze and typically lack theoretical guarantees. This paper adopts a dual quaternion representation and formulates SE(3) synchronization directly over the unit dual quaternion. A two-stage algorithm is developed: A spectral initializer computed via the power method on a Hermitian dual quaternion measurement matrix, followed by a dual quaternion generalized power method (DQGPM) that enforces feasibility through per-iteration projection. The estimation error bounds are established for spectral estimators, and DQGPM is shown to admit a finite-iteration error bound and achieves linear error contraction up to an explicit noise-dependent threshold. Experiments on synthetic benchmarks and real-world multi-scan point-set registration demonstrate that the proposed pipeline improves both accuracy and efficiency over representative matrix-based methods.

math.OC↗
arXiv2026

Spurious Stationarity and Hardness Results for Bregman Proximal-Type Algorithms

Bregman proximal-type algorithms (BPs), such as mirror descent, have become popular tools in machine learning and data science for exploiting problem structures through non-Euclidean geometries. In this paper, we show that BPs can get trapped near a class of non-stationary points, which we term \emph{spurious stationary points}. Such stagnation can persist for any finite number of iterations if the gradient of the Bregman kernel is not Lipschitz continuous, even in convex problems. The root cause lies in a fundamental contrast in descent behavior between Euclidean and Bregman geometries: While Euclidean gradient descent ensures sufficient decrease near any non-stationary point, BPs may exhibit arbitrarily slow decrease around spurious stationary points. As a result, commonly used Bregman-based stationarity measure, such as relative change in terms of Bregman divergence, can vanish near spurious stationary points. This may misleadingly suggest convergence, even when the iterates remain far from any true stationary point. Our analysis further reveals that spurious stationary points are not pathological, but rather occur generically in a broad class of nonconvex problems with polyhedral constraints. Taken together, our findings reveal a serious blind spot in Bregman-based optimization methods and calls for new theoretical tools and algorithmic safeguards to ensure reliable convergence.

math.OC↗
arXiv2026

Primal-Dual Methods for Nonsmooth Nonconvex Optimization with Orthogonality Constraints

Recent advancements in data science have significantly elevated the importance of orthogonally constrained optimization problems. The Riemannian approach has become a popular technique for addressing these problems due to the advantageous computational and analytical properties of the Stiefel manifold. Nonetheless, the interplay of nonsmoothness alongside orthogonality constraints introduces substantial challenges to current Riemannian methods, including scalability, parallelizability, complicated subproblems, and cumulative numerical errors that threaten feasibility. In this paper, we take a retraction-free primal-dual approach and propose a linearized smoothing augmented Lagrangian method specifically designed for nonsmooth and nonconvex optimization with orthogonality constraints. Our proposed method is single-loop and free of subproblem solving. We establish its iteration complexity of $O(ε^{-3})$ for finding $ε$-KKT points, matching the best-known results in the Riemannian optimization literature. Additionally, by invoking the standard Kurdyka-Lojasiewicz (KL) property, we demonstrate asymptotic sequential convergence of the proposed algorithm. Numerical experiments on both smooth and nonsmooth orthogonal constrained problems demonstrate the superior computational efficiency and scalability of the proposed method compared with state-of-the-art algorithms.

math.OC↗
arXiv2026

A Riemannian Alternating Descent Ascent Algorithmic Framework for Nonconvex-Linear Minimax Problems on Riemannian Manifolds

In this paper, we consider a class of nonconvex-linear minimax problems on Riemannian manifolds, which find wide applications in machine learning and signal processing. For solving this class of problems, we develop a flexible Riemannian alternating descent ascent (RADA) algorithmic framework. Within this framework, we propose two easy-to-implement yet efficient algorithms that alternately perform one or multiple projected/Riemannian gradient descent steps and a proximal gradient ascent step at each iteration. We show that the proposed RADA algorithmic framework can find both an $\varepsilon$-Riemannian-game-stationary point and an $\varepsilon$-Riemannian-optimization-stationary point within $\mathcal{O}(\varepsilon^{-3})$ iterations, achieving the best-known iteration complexity. We also reveal intriguing similarities and differences between the algorithms developed within our proposed framework and existing algorithms, thus providing important insights into the improved efficiency of the former. Lastly, we present numerical results on sparse principal component analysis (PCA), fair PCA, and sparse spectral clustering to demonstrate the superior performance of the proposed algorithms.

math.OC↗
arXiv2026

Optimal Low-Dimensional Structures of ISAC Beamforming: Theory and Efficient Algorithms

Transmit beamforming design is a fundamental problem in integrated sensing and communication (ISAC) systems. Numerous methods have been proposed to jointly optimize key performance metrics such as the signal-to-interference-plus-noise ratio and Cramér-Rao bound. However, the computational complexity of these methods often grows rapidly with the number of transmit antennas at the base station (BS). To tackle this challenge, we prove a fundamental structural property of the ISAC beamforming problem, i.e., there exists an optimal solution exhibiting a low-dimensional structure. This leads to an equivalent reformulation of the problem with dimension related to the number of users rather than the number of BS antennas, thereby enabling the development of low-complexity algorithms. When applying the interior-point method to the reformulated problem, we achieve up to six orders of magnitude in complexity reduction when the number of antennas exceeds the number of users by an order of magnitude. To further reduce the complexity, we develop a balanced augmented Lagrangian method to solve the reformulated problem. The proposed algorithm maintains optimality while achieving a computational complexity that scales quartically with the number of users. Our simulation results demonstrate that the proposed R-BAL method can achieve a speedup of more than 10000$\times$ over the conventional IPM in massive MIMO scenarios.

eess.SP↗
arXiv2025

Gaussian Arimoto-Blahut Algorithm for Capacity Region Calculation of Gaussian Vector Broadcast Channels

This paper is concerned with the computation of the capacity region of a continuous, Gaussian vector broadcast channel (BC) with covariance matrix constraints. Since the decision variables of the corresponding optimization problem are Gaussian distributed, they can be characterized by a finite number of parameters. Consequently, we develop new Blahut-Arimoto (BA)-type algorithms that can compute the capacity without discretizing the channel. First, by exploiting projection and an approximation of the Lagrange multiplier, which are introduced to handle certain positive semidefinite constraints in the optimization formulation, we develop the Gaussian BA algorithm with projection (GBA-P). Then, we demonstrate that one of the subproblems arising from the alternating updates admits a closed-form solution. Based on this result, we propose the Gaussian BA algorithm with alternating updates (GBA-A) and establish its convergence guarantee. Furthermore, we extend the GBA-P algorithm to compute the capacity region of the Gaussian vector BC with both private and common messages. All the proposed algorithms are parameter-free. Lastly, we present numerical results to demonstrate the effectiveness of the proposed algorithms.

cs.IT↗
arXiv2025

Set Smoothness Unlocks Clarke Hyper-stationarity in Bilevel Optimization

Solving bilevel optimization (BLO) problems to global optimality is generally intractable. A common surrogate is to compute a hyper-stationary point -- a stationary point of the hyper-objective function obtained by minimizing or maximizing the upper-level objective over the lower-level solution set. Existing methods, however, either provide weak notions of stationarity or require restrictive assumptions to guarantee the smoothness of hyper-objective functions. In this paper, we eliminate these impractical assumptions and show that strong (Clarke) hyper-stationarity remains computable even when the hyper-objective is nonsmooth. Our key ingredient is a new structural property, called set smoothness, which captures the variational dependence of the lower-level solution set on the upper-level variable. We prove that this property holds for a broad class of BLO problems and ensures weak convexity (resp. concavity) of pessimistic (resp. optimistic) hyper-objective functions. Building on this foundation, we show that a zeroth-order algorithm that computes approximate Clarke hyper-stationary points with non-asymptotic convergence guarantees. To the best of our knowledge, this is the first computational guarantee for Clarke-type stationarity in nonsmooth BLO. Beyond this specific application, the set smoothness property emerges as a structural concept of independent interest, with potential to inform the analysis of broader classes of optimization and variational problems.

math.OC↗
arXiv2025

Efficient Single-Loop Stochastic Algorithms for Nonconvex-Concave Minimax Optimization

Nonconvex-concave (NC-C) finite-sum minimax problems have wide applications in signal processing and machine learning tasks. Conventional stochastic gradient algorithms, which rely on uniform sampling for gradient estimation, often suffer from slow convergence rates and require bounded variance assumptions. While variance reduction techniques can significantly improve the convergence of stochastic algorithms, the inherent nonsmooth nature of NC-C problems makes it challenging to design effective variance reduction techniques. To address this challenge, we develop a novel probabilistic variance reduction scheme and propose a single-loop stochastic gradient algorithm called the probabilistic variance-reduced smoothed gradient descent-ascent (PVR-SGDA) algorithm. The proposed PVR-SGDA algorithm achieves an iteration complexity of $\mathcal{O}(ε^{-4})$, surpassing the best-known rates of stochastic algorithms for NC-C minimax problems and matching the performance of state-of-the-art deterministic algorithms. Furthermore, to completely eliminate the need for full gradient computation and reduce the gradient complexity, we explore another variance reduction technique with auxiliary gradient trackers and propose a smoothed gradient descent-ascent algorithm without full gradient calculation, called ZeroSARAH-SGDA, for NC-C problems. The ZeroSARAH-SGDA algorithm achieves a comparable iteration complexity to PVR-SGDA, while reducing the gradient oracle calls at each iteration. Finally, we demonstrate the effectiveness of the proposed two algorithms through numerical simulations.

math.OC↗
arXiv2025

Accelerated Price Adjustment for Fisher Markets with Exact Recovery of Competitive Equilibrium

The canonical price-adjustment process, tâtonnement, typically fails to converge to the exact competitive equilibrium (CE) and requires a high iteration complexity of $\tilde{\mathcal{O}}(1/ε)$ to compute $ε$-CE prices in widely studied linear and quasi-linear Fisher markets. This paper proposes refined price-adjustment processes to overcome these limitations. By formulating the task of finding CE of a (quasi-)linear Fisher market as a strongly convex nonsmooth minimization problem, we develop a novel accelerated price-adjustment method (APM) that finds an $ε$-CE price in $\tilde{\mathcal{O}}(1/\sqrtε)$ lightweight iterations, which significantly improves upon the iteration complexities of tâtonnement methods. Furthermore, through our new formulation, we construct a recovery oracle that maps approximate CE prices to exact CE prices at a low computational cost. By coupling this recovery oracle with APM, we obtain an adaptive price-adjustment method whose iterates converge to CE prices in finite steps. To the best of our knowledge, this is the first convergence guarantee to exact CE for price-adjustment methods in linear and quasi-linear Fisher markets. Our developments pave the way for efficient lightweight computation of CE prices. We also present numerical results to demonstrate the fast convergence of the proposed methods and the efficient recovery of CE prices.

math.OC↗
arXiv2025

Doubly Smoothed Optimistic Gradients: A Universal Approach for Smooth Minimax Problems

Smooth minimax optimization problems play a central role in a wide range of applications, including machine learning, game theory, and operations research. However, existing algorithmic frameworks vary significantly depending on the problem structure -- whether it is convex-concave, nonconvex-concave, convex-nonconcave, or even nonconvex-nonconcave with additional regularity conditions. In particular, this diversity complicates the tuning of step-sizes since even verifying convexity (or concavity) assumptions is challenging and problem-dependent. We introduce a universal and single-loop algorithm, Doubly Smoothed Optimistic Gradient Descent Ascent (DS-OGDA), that applies to a broad class of smooth minimax problems. Specifically, this class includes convex-concave, nonconvex-concave, convex-nonconcave, and nonconvex-nonconcave minimax optimization problems satisfying a one-sided Kurdyka-Lojasiewicz (KL) property. DS-OGDA works with a universal single set of parameters for all problems in this class, eliminating the need for prior structural knowledge to determine step-sizes. Moreover, when a particular problem structure in our class is specified, DS-OGDA achieves optimal or best-known performance guarantees. Overall, our results provide a comprehensive and versatile framework for smooth minimax optimization, bridging the gap between convex and nonconvex problem structures and simplifying the choice of algorithmic strategies across diverse applications.

math.OC↗