arXiv subjects

Yuzixuan Zhu

Publications and source records attributed to Yuzixuan Zhu.

3 recordsLinked to original sources

arXiv2021

A New Primal-Dual Algorithm for a Class of Nonlinear Compositional Convex Optimization Problems

We develop a novel primal-dual algorithm to solve a class of nonsmooth and nonlinear compositional convex minimization problems, which covers many existing and brand-new models as special cases. Our approach relies on a combination of a new nonconvex potential function, Nesterov's accelerated scheme, and an adaptive parameter updating strategy. Our algorithm is single-loop and has low per-iteration complexity. Under only general convexity and mild assumptions, our algorithm achieves $\mathcal{O}(1/k)$ convergence rates through three different criteria: primal objective residual, dual objective residual, and primal-dual gap, where $k$ is the iteration counter. Our rates are both ergodic (i.e., on an averaging sequence) and non-ergodic (i.e., on the last-iterate sequence). These convergence rates can be accelerated up to $\mathcal{O}(1/k^2)$ if only one objective term is strongly convex (or equivalently, its conjugate is $L$-smooth). To the best of our knowledge, this is the first algorithm achieving optimal rates on the primal last-iterate sequence for nonlinear compositional convex minimization. As a by-product, we specify our algorithm to solve a general convex cone constrained program with both ergodic and non-ergodic rate guarantees. We test our algorithms and compare them with two recent methods on a binary classification and a convex-concave game model.

math.OC↗

arXiv2020

Non-Stationary First-Order Primal-Dual Algorithms with Faster Convergence Rates

In this paper, we propose two novel non-stationary first-order primal-dual algorithms to solve nonsmooth composite convex optimization problems. Unlike existing primal-dual schemes where the parameters are often fixed, our methods use pre-defined and dynamic sequences for parameters. We prove that our first algorithm can achieve $\mathcal{O}(1/k)$ convergence rate on the primal-dual gap, and primal and dual objective residuals, where $k$ is the iteration counter. Our rate is on the non-ergodic (i.e., the last iterate) sequence of the primal problem and on the ergodic (i.e., the averaging) sequence of the dual problem, which we call semi-ergodic rate. By modifying the step-size update rule, this rate can be boosted even faster on the primal objective residual. When the problem is strongly convex, we develop a second primal-dual algorithm that exhibits $\mathcal{O}(1/k^2)$ convergence rate on the same three types of guarantees. Again by modifying the step-size update rule, this rate becomes faster on the primal objective residual. Our primal-dual algorithms are the first ones to achieve such fast convergence rate guarantees under mild assumptions compared to existing works, to the best of our knowledge. As byproducts, we apply our algorithms to solve constrained convex optimization problems and prove the same convergence rates on both the objective residuals and the feasibility violation. We still obtain at least $\mathcal{O}(1/k^2)$ rates even when the problem is "semi-strongly" convex. We verify our theoretical results via two well-known numerical examples.

math.OC↗

arXiv2018

Augmented Lagrangian-Based Decomposition Methods with Non-Ergodic Optimal Rates

We develop two new variants of alternating direction methods of multipliers (ADMM) and two parallel primal-dual decomposition algorithms to solve a wide range class of constrained convex optimization problems. Our approach relies on a novel combination of the augmented Lagrangian framework, partial alternating/linearization scheme, Nesterov's acceleration technique, and adaptive strategy. The proposed algorithms have the following new features compared to existing ADMM variants. First, they have a Nesterov's acceleration step on the primal variables instead of the dual ones as in several ADMM variants. Second, they possess an optimal $O(\frac{1}{k})$-convergence rate guarantee in a non-ergodic sense without any smoothness or strong convexity-type assumption, where $k$ is the iteration counter. When one objective term is strongly convex, our algorithm achieves an optimal $O(\frac{1}{k^2})$-non-ergodic rate. Third, our methods have better per-iteration complexity than standard ADMM due to the linearization step in the second subproblem. Fourth, we provide a set of conditions to derive update rules for algorithmic parameters, and give a concrete update for these parameters as an example. Finally, when the objective function is separable, our methods can naturally be implemented in a parallel fashion. We also study two extensions of our methods and a connection to existing primal-dual methods. We verify our theoretical development via different numerical examples and compare our methods with some existing state-of-the-art algorithms.

math.OC↗