arXiv subjects

Daoli Zhu

Publications and source records attributed to Daoli Zhu.

17 recordsLinked to original sources

arXiv2025

New Classes of Non-monotone Variational Inequality Problems Solvable via Proximal Gradient on Smooth Gap Functions

In this paper, we study the local linear convergence behavior of proximal-gradient (PG) descent algorithm on a parameterized gap-function reformulation of a smooth but non-monotone variational inequality problem (VIP). The aim is to solve the non-monotone VI problem without assuming the existence of a Minty-type solution. We first introduce and study various error bound conditions for the gap functions in relation to the VI model. In particular, we show that uniform type error bounds imply level-set type error bounds for composite optimization, revealing a key hierarchical structure there. As a result, local linear convergence is established under some easy-verifiable conditions induced by level-set error bounds, the gradient Lipschitz condition and a suitable initialization condition. Furthermore, for non-monotone affine VIs we present a homotopy continuation scheme that achieves global convergence by dynamically tracing a solution path. Our numerical experiments show the efficacy of the proposed approach, leading to the solutions of a broad class of non-monotone VI problems resulting from the need to compute Nash equilibria, traffic controls, and the GAN models.

math.OC↗

arXiv2023

A Unified Analysis on the Subgradient Upper Bounds for the Subgradient Methods Minimizing Composite Nonconvex, Nonsmooth and Non-Lipschitz Functions

This paper presents a unified analysis for the proximal subgradient method (Prox-SubGrad) type approach to minimize an overall objective of $f(x)+r(x)$, subject to convex constraints, where both $f$ and $r$ are weakly convex, nonsmooth, and non-Lipschitz. Leveraging on the properties of the Moreau envelope of weakly convex functions, we are able to relate error-bound conditions, the growth conditions of the subgradients of the objective, and the behavior of the proximal subgradient iterates on some remarkably broad classes of objective functions. Various existing as well as new bounding conditions are studied, leading to novel iteration complexity results. The terrain of our exploration expands to stochastic proximal subgradient algorithms.

math.OC↗

arXiv2023

Randomized Coordinate Subgradient Method for Nonsmooth Composite Optimization

Coordinate-type subgradient methods for addressing nonsmooth optimization problems are relatively underexplored due to the set-valued nature of the subdifferential. In this work, our study focuses on nonsmooth composite optimization problems, encompassing a wide class of convex and weakly convex (nonconvex nonsmooth) problems. By utilizing the chain rule of the composite structure properly, we introduce the Randomized Coordinate Subgradient method (RCS) for tackling this problem class. To the best of our knowledge, this is the first coordinate subgradient method for solving general nonsmooth composite optimization problems. In theory, we consider the linearly bounded subgradients assumption for the objective function, which is more general than the traditional Lipschitz continuity assumption, to account for practical scenarios. We then conduct convergence analysis for RCS in both convex and weakly convex cases based on this generalized Lipschitz-type assumption. Specifically, we establish the $\widetilde{\mathcal{O}}$$(1/\sqrt{k})$ convergence rate in expectation and the $\tilde o(1/\sqrt{k})$ almost sure asymptotic convergence rate in terms of the suboptimality gap when $f$ is convex. For the case when $f$ is weakly convex and its subdifferential satisfies the global metric subregularity property, we derive the $\mathcal{O}(\varepsilon^{-4})$ iteration complexity in expectation. We also establish an asymptotic convergence result. To justify the global metric subregularity property utilized in the analysis, we establish this error bound condition for the concrete (real-valued) robust phase retrieval problem. We also provide a convergence lemma and the relationship between the global metric subregularity properties of a weakly convex function and its Moreau envelope. Finally, we conduct several experiments to demonstrate the possible superiority of RCS over the subgradient method.

math.OC↗

arXiv2023

An Augmented Lagrangian Approach to Conically Constrained Non-monotone Variational Inequality Problems

In this paper we consider a non-monotone (mixed) variational inequality model with (nonlinear) convex conic constraints. Through developing an equivalent Lagrangian function-like primal-dual saddle-point system for the VI model in question, we introduce an augmented Lagrangian primal-dual method, to be called ALAVI in the current paper, for solving a general constrained VI model. Under an assumption, to be called the primal-dual variational coherence condition in the paper, we prove the convergence of ALAVI. Next, we show that many existing generalized monotonicity properties are sufficient -- though by no means necessary -- to imply the above mentioned coherence condition, thus are sufficient to ensure convergence of ALAVI. Under that assumption, we further show that ALAVI has in fact an $o(1/\sqrt{k})$ global rate of convergence where $k$ is the iteration count. By introducing a new gap function, this rate further improves to be $O(1/k)$ if the mapping is monotone. Finally, we show that under a metric subregularity condition, even if the VI model may be non-monotone the local convergence rate of ALAVI improves to be linear. Numerical experiments on some randomly generated highly nonlinear and non-monotone VI problems show practical efficacy of the newly proposed method.

math.OC↗

arXiv2023

Revisiting Subgradient Method: Complexity and Convergence Beyond Lipschitz Continuity

The subgradient method is one of the most fundamental algorithmic schemes for nonsmooth optimization. The existing complexity and convergence results for this method are mainly derived for Lipschitz continuous objective functions. In this work, we first extend the typical iteration complexity results for the subgradient method to cover non-Lipschitz convex and weakly convex minimization. Specifically, for the convex case, we can drive the suboptimality gap to below $\varepsilon$ in $\mathcal{O}( \varepsilon^{-2} )$ iterations; for the weakly convex case, we can drive the gradient norm of the Moreau envelope of the objective function to below $\varepsilon$ in $\mathcal{O}( \varepsilon^{-4} )$ iterations. Then, we provide convergence results for the subgradient method in the non-Lipschitz setting when proper diminishing rules on the step size are used. In particular, when $f$ is convex, we establish an $\mathcal{O}(\log(k)/\sqrt{k})$ rate of convergence in terms of the suboptimality gap, where $k$ represents the iteration count. With an additional quadratic growth property, the rate is improved to $\mathcal{O}(1/k)$ in terms of the squared distance to the optimal solution set. When $f$ is weakly convex, asymptotic convergence is established. Our results neither require any modification to the subgradient method nor impose any growth condition on the subgradients, while our analysis is surprisingly simple. To further illustrate the wide applicability of our framework, we extend the aforementioned iteration complexity results to cover the truncated subgradient, the stochastic subgradient, and the proximal subgradient methods for non-Lipschitz convex / weakly convex objective functions.

math.OC↗

arXiv2020

Linear Convergence of Randomized Primal-Dual Coordinate Method for Large-scale Linear Constrained Convex Programming

Linear constrained convex programming has many practical applications, including support vector machine and machine learning portfolio problems. We propose the randomized primal-dual coordinate (RPDC) method, a randomized coordinate extension of the first-order primal-dual method by Cohen and Zhu, 1984 and Zhao and Zhu, 2019, to solve linear constrained convex programming. We randomly choose a block of variables based on a uniform distribution, linearize, and apply a Bregman-like function (core function) to the selected block to obtain simple parallel primal-dual decomposition. We then establish almost surely convergence and expected O(1/t) convergence rate, and expected linear convergence under global strong metric subregularity. Finally, we discuss implementation details for the randomized primal-dual coordinate approach and present numerical experiments on support vector machine and machine learning portfolio problems to verify the linear convergence.

math.OC↗

arXiv2020

Level-set Subdifferential Error Bounds and Linear Convergence of Variable Bregman Proximal Gradient Method

In this work, we develop a level-set subdifferential error bound condition aiming towards convergence rate analysis of a variable Bregman proximal gradient (VBPG) method for a broad class of nonsmooth and nonconvex optimization problems. It is proved that the aforementioned condition guarantees linear convergence of VBPG, and is weaker than Kurdyka-Lojasiewicz property, weak metric subregularity and Bregman proximal error bound. Along the way, we are able to derive a number of verifiable conditions for level-set subdifferential error bounds to hold, and necessary conditions and sufficient conditions for linear convergence relative to a level set for nonsmooth and nonconvex optimization problems. The newly established results not only enable us to show that any accumulation point of the sequence generated by VBPG is at least a critical point of the limiting subdifferential or even acritical point of the proximal subdifferential with a fixed Bregman function in each iteration, but also provide a fresh perspective that allows us to explore inner-connections among many known sufficient conditions for linear convergence of various first-order methods.

math.OC↗

arXiv2020

A First-Order Primal-Dual Method for Nonconvex Constrained Optimization Based On the Augmented Lagrangian

Nonlinearly constrained nonconvex and nonsmooth optimization models play an increasingly important role in machine learning, statistics and data analytics. In this paper, based on the augmented Lagrangian function we introduce a flexible first-order primal-dual method, to be called nonconvex auxiliary problem principle of augmented Lagrangian (NAPP-AL), for solving a class of nonlinearly constrained nonconvex and nonsmooth optimization problems. We demonstrate that NAPP-AL converges to a stationary solution at the rate of o(1/\sqrt{k}), where k is the number of iterations. Moreover, under an additional error bound condition (to be called VP-EB in the paper), we further show that the convergence rate is in fact linear. Finally, we show that the famous Kurdyka- Lojasiewicz property and the metric subregularity imply the afore-mentioned VP-EB condition.

math.OC↗

arXiv2019

First-Order Primal-Dual Method for Nonlinear Convex Cone Programming

Nonlinear Convex Cone Programming (NCCP) problems are important and have many practical applications. In this paper, we introduces a flexible first-order primal-dual algorithm called the Variant Auxiliary Problem Principle (VAPP) for solving NCCP problems when the objective function and constraints are smooth and may be nonsmooth. Each iteration of VAPP generates a nonlinear approximation to the primal problem of an augmented Lagrangian method. The approximation incorporates both linearization and a variable distance-like function, and then the iterations of VAPP provide one decomposition property for NCCP. Motivated by recent applications in big data analysis, there has been an explosive growth in interest in the convergence rate analysis of parallel computing algorithms for large scale optimization problem. This paper proposes an iteration-based error bound and linear convergence of VAPP. Some verifiable sufficient conditions of this error bound are also discussed. For the general convex case (without error bound), we establish $O(1/t)$ convergence rate for primal suboptimality, feasibility and dual suboptimality. By adaptively setting in parameters at different iterations, we show an $O(1/t^2)$ rate for the strongly convex case. We further present Forward-Backward Splitting (FBS) formulation of VAPP method and establish the connection between VAPP and other primal-dual splitting methods. Finally, we discuss some issues in the implementation of VAPP.

math.OC↗

arXiv2019

A Variational Approach on Level sets and Linear Convergence of Variable Bregman Proximal Gradient Method for Nonconvex Optimization Problems

We develop a new variational approach on level sets aiming towards convergence rate analysis of a variable Bregman proximal gradient (VBPG) method for a broad class of nonsmooth and nonconvex optimization problems. With this new approach, we are able to extend the concepts of Bregman proximal mapping and their corresponding Bregman proximal envelops, Bregman proximal gap function to nonconvex setting. Properties of these mappings and functions are examined. An aim of this work is to provide a solid foundation on which further design and analysis of VBPG for more general nonconvex optimization problems are possible. Another aim is to provide a unified theory on linear convergence of VBPG with a particular interest towards proximal gradient methods. Centrol to our analysis for achieving the above goals is an error bound in terms of level sets and subdifferentials (level-set subdifferential error bound) along with its links to other level-set error bounds. As a consequence, we have established a number of positive results. These newly established results not only enable us to show that any accumulation of the sequence generated by VBPG is at least a critical point of the limiting subdifferential or even a critical point of the proximal subdifferential with a fixed Bregman function in each iteration, but also provide a fresh perspective that allows us to explore inner-connections among many known sufficient conditions for linear convergence of various first-order methods. Along the way, we are able to derive a number of verifiable conditions for level-set error bounds to hold, obtain linear convergence of VBPG, and derive necessary conditions and sufficient conditions for linear convergence relative to a level set for nonsmooth and nonconvex optimization problems.

math.OC↗

arXiv2019

Linear Convergence of Variable Bregman Stochastic Coordinate Descent Method for Nonsmooth Nonconvex Optimization by Level-set Variational Analysis

Large-scale nonconvex and nonsmooth problems have attracted considerable attention in the fields of compress sensing, big data optimization and machine learning. Exploring effective methods is still the main challenge of today's research. Stochastic coordinate descent type methods have been widely used to solve large-scale optimization problems. In this paper, we derive the convergence of variable Bregman stochastic coordinate descent (VBSCD) method for a broad class of nonsmooth and nonconvex optimization problems, i.e., any accumulation of the sequence generated by VBSCD is almost surely a critical point. Moreover, we develop a new variational approach on level sets that aim towards the convergence rate analysis. If the level-set subdifferential error bound holds, we derive a linear rate of convergence for the expected values of the objective function and expected values of random variables generated by VBSCD.

math.OC↗

arXiv2019

Stochastic Primal-Dual Coordinate Method with Large Step Size for Composite Optimization with Composite Cone-constraints

We introduce a stochastic coordinate extension of the first-order primal-dual method studied by Cohen and Zhu (1984) and Zhao and Zhu (2018) to solve Composite Optimization with Composite Cone-constraints (COCC). In this method, we randomly choose a block of variables based on the uniform distribution. The linearization and Bregman-like function (core function) to that randomly selected block allow us to get simple parallel primal-dual decomposition for COCC. We obtain almost surely convergence and O(1/t) expected convergence rate in this work. The high probability complexity bound is also derived in this paper.

math.OC↗

arXiv2018

Stochastic Primal-Dual Coordinate Method for Nonlinear Convex Cone Programs

Block coordinate descent (BCD) methods and their variants have been widely used in coping with large-scale nonconstrained optimization problems in many fields such as imaging processing, machine learning, compress sensing and so on. For problem with coupling constraints, Nonlinear convex cone programs (NCCP) are important problems with many practical applications, but these problems are hard to solve by using existing block coordinate type methods. This paper introduces a stochastic primal-dual coordinate (SPDC) method for solving large-scale NCCP. In this method, we randomly choose a block of variables based on the uniform distribution. The linearization and Bregman-like function (core function) to that randomly selected block allow us to get simple parallel primal-dual decomposition for NCCP. The sequence generated by our algorithm is proved almost surely converge to an optimal solution of primal problem. Two types of convergence rate with different probability (almost surely and expected) are also obtained. The probability complexity bound is also derived in this paper.

math.OC↗

arXiv2017

Auxiliary Problem Principle of augmented Lagrangian with Varying Core Functions for Large-Scale Structured Convex Problems

The auxiliary problem principle of augmented Lagrangian (APP-AL), proposed by Cohen and Zhu (1984), aims to find the solution of a constrained optimization problem through a sequence of auxiliary problems involving augmented Lagrangian. The merits of this approach are two folds. First, the core function is usually separable, which makes the subproblems at each step decomposable and particularly attractive for parallel computing. Second, the choice of the core function is quite flexible. Consequently, by carefully specifying this function, APP-AL may reduce to some standard optimization algorithms. In this paper, we pursue enhancing such flexibility by allowing the core function to be non-identical at each step of the algorithm, and name it varying auxiliary problem principle (VAPP-AL). Depending on the problem structure, the varying core functions in VAPP-AL can be adapted to design new flexible and suitable algorithm for parallel and distributed computing. The convergence and O(1/t) convergence rate of VAPP-AL for convex problem with coupling objective and constraints is proved. Moreover, if this function is specialized to be quadratic, an o(1/t) convergence rate can be established. Interestingly, the new VAPP framework can cover several variants of Jacobian type augmented Lagrangian decomposition methods as special cases. Furthermore, our technique works for the convex problem with nonseparable objective and multi-blocks coupled linear constraints, which usually can not be handled by ADMM.

math.OC↗

arXiv2008

Empirical analysis of the worldwide maritime transportation network

In this paper we present an empirical study of the worldwide maritime transportation network (WMN) in which the nodes are ports and links are container liners connecting the ports. Using the different representation of network topology namely the space $L$ and $P$, we study the statistical properties of WMN including degree distribution, degree correlations, weight distribution, strength distribution, average shortest path length, line length distribution and centrality measures. We find that WMN is a small-world network with power law behavior. Important nodes are identified based on different centrality measures. Through analyzing weighted cluster coefficient and weighted average nearest neighbors degree, we reveal the hierarchy structure and "rich-club" phenomenon in the network.

physics.soc-ph↗

arXiv2007

Cost-driven weighted networks evolution

Inspired by studies on airline networks we propose a general model for weighted networks in which topological growth and weight dynamics are both determined by cost adversarial mechanism. Since transportation networks are designed and operated with objectives to reduce cost, the theory of cost in micro-economics plays a critical role in the evolution. We assume vertices and edges are given cost functions according to economics of scale and diseconomics of scale (congestion effect). With different cost functions the model produces broad distribution of networks. The model reproduces key properties of real airline networks: truncated degree distributions, nonlinear strength degree correlations, hierarchy structures, and particulary the disassortative and assortative behavior observed in different airline networks. The result suggests that the interplay between economics of scale and diseconomics of scale is a key ingredient in order to understand the underlying driving factor of the real-world weighted networks.

physics.soc-ph↗

arXiv2007

A weighted network evolution model based on passenger behavior

This paper presents an evolution model of weighted networks in which the structural growth and weight dynamics are driven by human behavior, i.e. passenger route choice behavior. Transportation networks grow due to people's increasing travel demand and the pattern of growth is determined by their route choice behavior. In airline networks passengers often transfer from a third airport instead of flying directly to the destination, which contributes to the hubs formation and finally the scale-free statistical property. In this model we assume at each time step there emerges a new node with m travel destinations. Then the new node either connects destination directly with the probability p or transfers from a third node with the probability 1-p. The analytical result shows degree and strength both obey power-law distribution with the exponent between 2.33 and 3 depending on p. The weights also obey power-law distribution. The clustering coefficient, degree assortatively coefficient and degree-strength correlation are all dependent on the probability p. This model can also be used in social networks.

physics.soc-ph↗