arXiv subjects

Taoli Zheng

Publications and source records attributed to Taoli Zheng.

7 recordsLinked to original sources

arXiv2026

Optimal Deterministic First-Order Oracle Complexity for Nonconvex-Concave Minimax Optimization

We study the deterministic first-order oracle complexity of smooth nonconvex-concave minimax optimization over a bounded convex dual domain. Let $\ell$ denote the joint smoothness constant, $D_{\mathcal{Y}}$ the diameter of the dual domain, and $Δ$ the initial gap. We prove that every deterministic first-order algorithm requires $Ω(\ell^2D_{\mathcal{Y}}Δ/ε^3)$ oracle queries in the worst case to find an $ε$-optimization-stationary point whenever $ε\lesssim\min\{\ell D_{\mathcal{Y}},\sqrt{\ellΔ}\}$. We then develop Tracked-FOAM, a first-order method that attains a matching upper bound, removing the logarithmic factor from previous upper bounds. Together, these results establish the optimal dependence on all problem parameters in the stated regime.

math.OC↗

arXiv2026

Last-Iterate Convergence of Single-Loop Stochastic Methods for Constrained Convex-Concave Minimax Problems

In this paper, we study last-iterate convergence of stochastic first-order methods for constrained smooth convex--concave minimax optimization under the standard bounded-variance stochastic oracle. A fundamental challenge is that the last iterates of vanilla stochastic extragradient (S-EG) and stochastic optimistic gradient descent--ascent (S-OGDA) may fail to converge in the presence of stochastic gradient noise, even for simple bilinear problems. To overcome this difficulty, we introduce a simple perturbation framework that regularizes the original convex--concave problem into a strongly convex--strongly concave one. Applying S-EG and S-OGDA to the perturbed problem yields two simple single-loop methods, referred to as perturbed S-EG (PS-EG) and perturbed S-OGDA (PS-OGDA). We establish last-iterate convergence by first deriving convergence in terms of the squared distance to the saddle point of the perturbed problem and then translating this estimate into guarantees for the restricted primal--dual gap. Based on this framework, we establish two types of convergence guarantees. When the optimization horizon is known \emph{a priori}, both PS-EG and PS-OGDA achieve an $\mathcal{O}(T^{-1/4})$ last-iterate convergence rate for the restricted primal--dual gap, which coincides with the standard primal--dual gap on compact feasible domains. When the optimization horizon is unknown, we develop an anytime variant based on diminishing perturbations and diminishing stepsizes. For general closed convex feasible sets, both PS-EG and PS-OGDA achieve an $\mathcal{O}(T^{-1/5})$ last-iterate convergence rate for the restricted primal--dual gap. Furthermore, in the unconstrained setting, PS-EG admits a sharper $\mathcal{O}(T^{-1/4})$ anytime convergence rate in terms of the gradient norm.

math.OC↗

arXiv2025

Efficient Single-Loop Stochastic Algorithms for Nonconvex-Concave Minimax Optimization

Nonconvex-concave (NC-C) finite-sum minimax problems have wide applications in signal processing and machine learning tasks. Conventional stochastic gradient algorithms, which rely on uniform sampling for gradient estimation, often suffer from slow convergence rates and require bounded variance assumptions. While variance reduction techniques can significantly improve the convergence of stochastic algorithms, the inherent nonsmooth nature of NC-C problems makes it challenging to design effective variance reduction techniques. To address this challenge, we develop a novel probabilistic variance reduction scheme and propose a single-loop stochastic gradient algorithm called the probabilistic variance-reduced smoothed gradient descent-ascent (PVR-SGDA) algorithm. The proposed PVR-SGDA algorithm achieves an iteration complexity of $\mathcal{O}(ε^{-4})$, surpassing the best-known rates of stochastic algorithms for NC-C minimax problems and matching the performance of state-of-the-art deterministic algorithms. Furthermore, to completely eliminate the need for full gradient computation and reduce the gradient complexity, we explore another variance reduction technique with auxiliary gradient trackers and propose a smoothed gradient descent-ascent algorithm without full gradient calculation, called ZeroSARAH-SGDA, for NC-C problems. The ZeroSARAH-SGDA algorithm achieves a comparable iteration complexity to PVR-SGDA, while reducing the gradient oracle calls at each iteration. Finally, we demonstrate the effectiveness of the proposed two algorithms through numerical simulations.

math.OC↗

arXiv2025

Doubly Smoothed Optimistic Gradients: A Universal Approach for Smooth Minimax Problems

Smooth minimax optimization problems play a central role in a wide range of applications, including machine learning, game theory, and operations research. However, existing algorithmic frameworks vary significantly depending on the problem structure -- whether it is convex-concave, nonconvex-concave, convex-nonconcave, or even nonconvex-nonconcave with additional regularity conditions. In particular, this diversity complicates the tuning of step-sizes since even verifying convexity (or concavity) assumptions is challenging and problem-dependent. We introduce a universal and single-loop algorithm, Doubly Smoothed Optimistic Gradient Descent Ascent (DS-OGDA), that applies to a broad class of smooth minimax problems. Specifically, this class includes convex-concave, nonconvex-concave, convex-nonconcave, and nonconvex-nonconcave minimax optimization problems satisfying a one-sided Kurdyka-Lojasiewicz (KL) property. DS-OGDA works with a universal single set of parameters for all problems in this class, eliminating the need for prior structural knowledge to determine step-sizes. Moreover, when a particular problem structure in our class is specified, DS-OGDA achieves optimal or best-known performance guarantees. Overall, our results provide a comprehensive and versatile framework for smooth minimax optimization, bridging the gap between convex and nonconvex problem structures and simplifying the choice of algorithmic strategies across diverse applications.

math.OC↗

arXiv2023

Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization

Nonconvex-nonconcave minimax optimization has received intense attention over the last decade due to its broad applications in machine learning. Most existing algorithms rely on one-sided information, such as the convexity (resp. concavity) of the primal (resp. dual) functions, or other specific structures, such as the Polyak-Łojasiewicz (PŁ) and Kurdyka-Łojasiewicz (KŁ) conditions. However, verifying these regularity conditions is challenging in practice. To meet this challenge, we propose a novel universally applicable single-loop algorithm, the doubly smoothed gradient descent ascent method (DS-GDA), which naturally balances the primal and dual updates. That is, DS-GDA with the same hyperparameters is able to uniformly solve nonconvex-concave, convex-nonconcave, and nonconvex-nonconcave problems with one-sided KŁ properties, achieving convergence with $\mathcal{O}(ε^{-4})$ complexity. Sharper (even optimal) iteration complexity can be obtained when the KŁ exponent is known. Specifically, under the one-sided KŁ condition with exponent $θ\in(0,1)$, DS-GDA converges with an iteration complexity of $\mathcal{O}(ε^{-2\max\{2θ,1\}})$. They all match the corresponding best results in the literature. Moreover, we show that DS-GDA is practically applicable to general nonconvex-nonconcave problems even without any regularity conditions, such as the PŁ condition, KŁ condition, or weak Minty variational inequalities condition. For various challenging nonconvex-nonconcave examples in the literature, including ``Forsaken'', ``Bilinearly-coupled minimax'', ``Sixth-order polynomial'', and ``PolarGame'', the proposed DS-GDA can all get rid of limit cycles. To the best of our knowledge, this is the first first-order algorithm to achieve convergence on all of these formidable problems.

math.OC↗

arXiv2022

A Linearly Convergent Algorithm for Rotationally Invariant $\ell_1$-Norm Principal Component Analysis

To do dimensionality reduction on the datasets with outliers, the $\ell_1$-norm principal component analysis (L1-PCA) as a typical robust alternative of the conventional PCA has enjoyed great popularity over the past years. In this work, we consider a rotationally invariant L1-PCA, which is hardly studied in the literature. To tackle it, we propose a proximal alternating linearized minimization method with a nonlinear extrapolation for solving its two-block reformulation. Moreover, we show that the proposed method converges at least linearly to a limiting critical point of the reformulated problem. Such a point is proved to be a critical point of the original problem under a condition imposed on the step size. Finally, we conduct numerical experiments on both synthetic and real datasets to support our theoretical developments and demonstrate the efficacy of our approach.

math.OC↗

arXiv2021

Sparse High-Order Portfolios via Proximal DCA and SCA

In this paper, we aim at solving the cardinality constrained high-order portfolio optimization, i.e., mean-variance-skewness-kurtosis model with cardinality constraint (MVSKC). Optimization for the MVSKC model is of great difficulty in two parts. One is that the objective function is non-convex, the other is the combinational nature of the cardinality constraint, leading to non-convexity as well dis-continuity. Based on the observation that cardinality constraint has the difference-of-convex (DC) property, we transform the cardinality constraint into a penalty term and then propose three algorithms including the proximal difference of convex algorithm (pDCA), pDCA with extrapolation (pDCAe) and the successive convex approximation (SCA) to handle the resulting penalized MVSK (PMVSK) formulation. Moreover, theoretical convergence results of these algorithms are established respectively. Numerical experiments on the real datasets demonstrate the superiority of our proposed methods in obtaining high utility and sparse solutions as well as efficiency in terms of time usage.

q-fin.PM↗