arXiv subjects

Shuzhong Zhang

Publications and source records attributed to Shuzhong Zhang.

At least 37 records · Page 2Linked to original sources

arXiv2024

Novel Optimization Techniques for Parameter Estimation

In this paper, we introduce a new optimization algorithm that is well suited for solving parameter estimation problems. We call our new method cubic regularized Newton with affine scaling (CRNAS). In contrast to so-called first-order methods which rely solely on the gradient of the objective function, our method utilizes the Hessian of the objective. As a result it is able to focus on points satisfying the second-order optimality conditions, as opposed to first-order methods that simply converge to critical points. This is an important feature in parameter estimation problems where the objective function is often non-convex and as a result there can be many critical points making it is near impossible to identify the global minimum. An important feature of parameter estimation in mathematical models of biological systems is that the parameters are constrained by either physical constraints or prior knowledge. We use an affine scaling approach to handle a wide class of constraints. We establish that CRNAS identifies a point satisfying $ε$-approximate second-order optimality conditions within $O(ε^{-3/2})$ iterations. Finally, we compare CRNAS with MATLAB's optimization solver fmincon on three different test problems. These test problems all feature mixtures of heterogeneous populations, a problem setting that CRNAS is particularly well-suited for. Our numerical simulations show CRNAS has favorable performance, performing comparable if not better than fmincon in accuracy and computational cost for most of our examples.

math.OC↗

arXiv2023

Linearly-Convergent FISTA Variant for Composite Optimization with Duality

Many large-scale optimization problems can be expressed as composite optimization models. Accelerated first-order methods such as the fast iterative shrinkage-thresholding algorithm (FISTA) have proven effective for numerous large composite models. In this paper, we present a new variation of FISTA, to be called C-FISTA, which obtains global linear convergence for a broader class of composite models than many of the latest FISTA variants. We demonstrate the versatility and effectiveness of C-FISTA by showing C-FISTA outperforms current first-order solvers on both group Lasso and group logistic regression models. Furthermore, we utilize Fenchel duality to prove C-FISTA provides global linear convergence for a large class of convex models without the loss of global linear convergence.

math.OC↗

arXiv2023

Spectrally Constrained Optimization

We investigate how to solve smooth matrix optimization problems with general linear inequality constraints on the eigenvalues of a symmetric matrix. We present solution methods to obtain exact global minima for linear objective functions, i.e., $F(X) = \langle C, X \rangle$, and perform exact projections onto the eigenvalue constraint set. Two first-order algorithms are developed to obtain first-order stationary points for general non-convex objective functions. Both methods are proven to converge sublinearly when the constraint set is convex. Numerical experiments demonstrate the applicability of both the model and the methods.

math.OC↗

arXiv2023

An Augmented Lagrangian Approach to Conically Constrained Non-monotone Variational Inequality Problems

In this paper we consider a non-monotone (mixed) variational inequality model with (nonlinear) convex conic constraints. Through developing an equivalent Lagrangian function-like primal-dual saddle-point system for the VI model in question, we introduce an augmented Lagrangian primal-dual method, to be called ALAVI in the current paper, for solving a general constrained VI model. Under an assumption, to be called the primal-dual variational coherence condition in the paper, we prove the convergence of ALAVI. Next, we show that many existing generalized monotonicity properties are sufficient -- though by no means necessary -- to imply the above mentioned coherence condition, thus are sufficient to ensure convergence of ALAVI. Under that assumption, we further show that ALAVI has in fact an $o(1/\sqrt{k})$ global rate of convergence where $k$ is the iteration count. By introducing a new gap function, this rate further improves to be $O(1/k)$ if the mapping is monotone. Finally, we show that under a metric subregularity condition, even if the VI model may be non-monotone the local convergence rate of ALAVI improves to be linear. Numerical experiments on some randomly generated highly nonlinear and non-monotone VI problems show practical efficacy of the newly proposed method.

math.OC↗

arXiv2023

Beyond Monotone Variational Inequalities: Solution Methods and Iteration Complexities

In this paper, we discuss variational inequality (VI) problems without monotonicity from the perspective of convergence of projection-type algorithms. In particular, we identify existing conditions as well as present new conditions that are sufficient to guarantee convergence. The first half of the paper focuses on the case where a Minty solution exists (also known as Minty condition), which is a common assumption in the recent developments for non-monotone VI. The second half explores alternative sufficient conditions that are different from the existing ones such as monotonicity or Minty condition, using an algorithm-based approach. Through examples and convergence analysis, we show that these conditions are capable of characterizing different classes of VI problems where the algorithms are guaranteed to converge.

math.OC↗

arXiv2023

Primal-Dual First-Order Methods for Affinely Constrained Multi-Block Saddle Point Problems

We consider the convex-concave saddle point problem $\min_{\mathbf{x}}\max_{\mathbf{y}}Φ(\mathbf{x},\mathbf{y})$, where the decision variables $\mathbf{x}$ and/or $\mathbf{y}$ subject to a multi-block structure and affine coupling constraints, and $Φ(\mathbf{x},\mathbf{y})$ possesses certain separable structure. Although the minimization counterpart of such problem has been widely studied under the topics of ADMM, this minimax problem is rarely investigated. In this paper, a convenient notion of $ε$-saddle point is proposed, under which the convergence rate of several proposed algorithms are analyzed. When only one of $\mathbf{x}$ and $\mathbf{y}$ has multiple blocks and affine constraint, several natural extensions of ADMM are proposed to solve the problem. Depending on the number of blocks and the level of smoothness, $\mathcal{O}(1/T)$ or $\mathcal{O}(1/\sqrt{T})$ convergence rates are derived for our algorithms. When both $\mathbf{x}$ and $\mathbf{y}$ have multiple blocks and affine constraints, a new algorithm called ExtraGradient Method of Multipliers (EGMM) is proposed. Under desirable smoothness condition, an $\mathcal{O}(1/T)$ rate of convergence can be guaranteed regardless of the number of blocks in $\mathbf{x}$ and $\mathbf{y}$. In depth comparison between EGMM (fully primal-dual method) and ADMM (approximate dual method) is made over the multi-block optimization problems to illustrate the advantage of the EGMM.

math.OC↗

arXiv2022

Minimax Problems with Coupled Linear Constraints: Computational Complexity, Duality and Solution Methods

In this work we study a special minimax problem where there are linear constraints that couple both the minimization and maximization decision variables. The problem is a generalization of the traditional saddle point problem (which does not have the coupling constraint), and it finds applications in wireless communication, game theory, transportation, just to name a few. We show that the considered problem is challenging, in the sense that it violates the classical max-min inequality, and that it is NP-hard even under very strong assumptions (e.g., when the objective is strongly convex-strongly concave). We then develop a duality theory for it, and analyze conditions under which the duality gap becomes zero. Finally, we study a class of stationary solutions defined based on the dual problem, and evaluate their practical performance in an application on adversarial attacks on network flow problems.

math.OC↗

arXiv2022

An Approximation-Based Regularized Extra-Gradient Method for Monotone Variational Inequalities

In this paper, we propose a general extra-gradient scheme for solving monotone variational inequalities (VI), referred to here as Approximation-based Regularized Extra-gradient method (ARE). The first step of ARE solves a VI subproblem with an approximation operator satisfying a $p^{th}$-order Lipschitz bound with respect to the original mapping, further coupled with the gradient of a $(p+1)^{th}$-order regularization. The optimal global convergence is guaranteed by including an additional extra-gradient step, while a $p^{th}$-order superlinear local convergence is shown to hold if the VI is strongly monotone. The proposed ARE is inclusive and general, in the sense that a variety of solution methods can be formulated within this framework as different manifestations of approximations, and their iteration complexities would follow through in a unified fashion. The ARE framework relates to the first-order methods, while opening up possibilities to developing higher-order methods specifically for structured problems that guarantee the optimal iteration complexity bounds.

math.OC↗

arXiv2022

Cubic-Regularized Newton for Spectral Constrained Matrix Optimization and its Application to Fairness

Matrix functions are utilized to rewrite smooth spectral constrained matrix optimization problems as smooth unconstrained problems over the set of symmetric matrices which are then solved via the cubic-regularized Newton method. A second-order chain rule identity for matrix functions is proven to compute the higher-order derivatives to implement cubic-regularized Newton, and a new convergence analysis is provided for cubic-regularized Newton for matrix vector spaces. We demonstrate the applicability of our approach by conducting numerical experiments on both synthetic and real datasets. In our experiments, we formulate a new model for estimating fair and robust covariance matrices in the spirit of the Tyler's M-estimator (TME) model and demonstrate its advantage.

math.OC↗

arXiv2022

A Gradient Complexity Analysis for Minimizing the Sum of Strongly Convex Functions with Varying Condition Numbers

A popular approach to minimize a finite-sum of convex functions is stochastic gradient descent (SGD) and its variants. Fundamental research questions associated with SGD include: (i) To find a lower bound on the number of times that the gradient oracle of each individual function must be assessed in order to find an $ε$-minimizer of the overall objective; (ii) To design algorithms which guarantee to find an $ε$-minimizer of the overall objective in expectation at no more than a certain number of times (in terms of $1/ε$) that the gradient oracle of each functions needs to be assessed (i.e., upper bound). If these two bounds are at the same order of magnitude, then the algorithms may be called optimal. Most existing results along this line of research typically assume that the functions in the objective share the same condition number. In this paper, the first model we study is the problem of minimizing the sum of finitely many strongly convex functions whose condition numbers are all different. We propose an SGD method for this model and show that it is optimal in gradient computations, up to a logarithmic factor. We then consider a constrained separate block optimization model, and present lower and upper bounds for its gradient computation complexity. Next, we propose to solve the Fenchel dual of the constrained block optimization model via the SGD we introduced earlier, and show that it yields a lower iteration complexity than solving the original model by the ADMM-type approach. Finally, we extend the analysis to the general composite convex optimization model, and obtain gradient-computation complexity results under certain conditions.

math.OC↗

arXiv2022

Projectively and weakly simultaneously diagonalizable matrices and their applications

Characterizing simultaneously diagonalizable (SD) matrices has been receiving considerable attention in the recent decades due to its wide applications and its role in matrix analysis. However, the notion of SD matrices is arguably still restrictive for wider applications. In this paper, we consider two error measures related to the simultaneous diagonalization of matrices, and propose several new variants of SD thereof; in particular, TWSD, TWSD-B, T_{m,n}-SD (SDO), DWSD and D_{m,n}-SD (SDO). Those are all weaker forms of SD. We derive various sufficient and/or necessary conditions of them under different assumptions, and show the relationships between these new notions. Finally, we discuss the applications of these new notions in, e.g., quadratically constrained quadratic programming (QCQP) and independent component analysis (ICA).

math.NA↗

arXiv2022

Accelerating Adaptive Cubic Regularization of Newton's Method via Random Sampling

In this paper, we consider an unconstrained optimization model where the objective is a sum of a large number of possibly nonconvex functions, though overall the objective is assumed to be smooth and convex. Our bid to solving such model uses the framework of cubic regularization of Newton's method. As well known, the crux in cubic regularization is its utilization of the Hessian information, which may be computationally expensive for large-scale problems. To tackle this, we resort to approximating the Hessian matrix via sub-sampling. In particular, we propose to compute an approximated Hessian matrix by either \textit{uniformly}\/ or \textit{non-uniformly}\/ sub-sampling the components of the objective. Based upon such sampling strategy, we develop accelerated adaptive cubic regularization approaches and provide theoretical guarantees on global iteration complexity of $O(ε^{-1/3})$ with high probability, which matches that of the original accelerated cubic regularization methods \cite{Jiang-2017-Unified} using the \textit{full}\/ Hessian information. Interestingly, we show that in the worst case scenario our algorithm still achieves an $O\left(\log(ε^{-1})ε^{-5/6}\right)$ iteration complexity bound. The performances of the proposed methods on the regularized logistic regression problems show a clear effect of acceleration in terms of the epoch counts on several real data sets.

math.OC↗

arXiv2021

New First-Order Algorithms for Stochastic Variational Inequalities

In this paper, we propose two new solution schemes to solve the stochastic strongly monotone variational inequality problems: the stochastic extra-point solution scheme and the stochastic extra-momentum solution scheme. The first one is a general scheme based on updating the iterative sequence and an auxiliary extra-point sequence. In the case of deterministic VI model, this approach includes several state-of-the-art first-order methods as its special cases. The second scheme combines two momentum-based directions: the so-called heavy-ball direction and the optimism direction, where only one projection per iteration is required in its updating process. We show that, if the variance of the stochastic oracle is appropriately controlled, then both schemes can be made to achieve optimal iteration complexity of $\mathcal{O}\left(κ\ln\left(\frac{1}ε\right)\right)$ to reach an $ε$-solution for a strongly monotone VI problem with condition number $κ$. We show that these methods can be readily incorporated in a zeroth-order approach to solve stochastic minimax saddle-point problems, where only noisy and biased samples of the objective can be obtained, with a total sample complexity of $\mathcal{O}\left(\frac{κ^2}ε\ln\left(\frac{1}ε\right)\right)$

math.OC↗

arXiv2021

On Lower Iteration Complexity Bounds for the Saddle Point Problems

In this paper, we study the lower iteration complexity bounds for finding the saddle point of a strongly convex and strongly concave saddle point problem: $\min_x\max_yF(x,y)$. We restrict the classes of algorithms in our investigation to be either pure first-order methods or methods using proximal mappings. The existing lower bound result for this type of problems is obtained via the framework of strongly monotone variational inequality problems, which corresponds to the case where the gradient Lipschitz constants ($L_x, L_y$ and $L_{xy}$) and strong convexity/concavity constants ($μ_x$ and $μ_y$) are uniform with respect to variables $x$ and $y$. However, specific to the min-max saddle point problem these parameters are naturally different. Therefore, one is led to finding the best possible lower iteration complexity bounds, specific to the min-max saddle point models. In this paper we present the following results. For the class of pure first-order algorithms, our lower iteration complexity bound is $Ω\left(\sqrt{\frac{L_x}{μ_x}+\frac{L_{xy}^2}{μ_xμ_y}+\frac{L_y}{μ_y}}\cdot\ln\left(\frac{1}ε\right)\right)$, where the term $\frac{L_{xy}^2}{μ_xμ_y}$ explains how the coupling influences the iteration complexity. Under several special parameter regimes, this lower bound has been achieved by corresponding optimal algorithms. However, whether or not the bound under the general parameter regime is optimal remains open. Additionally, for the special case of bilinear coupling problems, given the availability of certain proximal operators, a lower bound of $Ω\left(\sqrt{\frac{L_{xy}^2}{μ_xμ_y}+1}\cdot\ln(\frac{1}ε)\right)$ is established in this paper, and optimal algorithms have already been developed in the literature.

math.OC↗

arXiv2021

A Unifying Framework of Accelerated First-Order Approach to Strongly Monotone Variational Inequalities

In this paper, we propose a unifying framework incorporating several momentum-related search directions for solving strongly monotone variational inequalities. The specific combinations of the search directions in the framework are made to guarantee the optimal iteration complexity bound of $\mathcal{O}\left(κ\ln(1/ε)\right)$ to reach an $ε$-solution, where $κ$ is the condition number. This framework provides the flexibility for algorithm designers to train -- among different parameter combinations -- the one that best suits the structure of the problem class at hand. The proposed framework includes the following iterative points and directions as its constituents: the extra-gradient, the optimistic gradient descent ascent (OGDA) direction (aka "optimism"), the "heavy-ball" direction, and Nesterov's extrapolation points. As a result, all the afore-mentioned methods become the special cases under the general scheme of extra points. We also specialize this approach to strongly convex minimization, and show that a similar extra-point approach achieves the optimal iteration complexity bound of $\mathcal{O}(\sqrtκ\ln(1/ε))$ for this class of problems.

math.OC↗

arXiv2020

Distributionally Robust Newsvendor with Moment Constraints

This paper expands the work on distributionally robust newsvendor to incorporate moment constraints. The use of Wasserstein distance as the ambiguity measure is preserved. The infinite dimensional primal problem is formulated; problem of moments duality is invoked to derive the simpler finite dimensional dual problem. An important research question is: How does distributional ambiguity affect the optimal order quantity and the corresponding profits/costs? To investigate this, some theory is developed and a case study in auto sales is performed. We conclude with some comments on directions for further research.

q-fin.MF↗

arXiv2020

Tight Bounds for a Class of Data-Driven Distributionally Robust Risk Measures

This paper expands the notion of robust moment problems to incorporate distributional ambiguity using Wasserstein distance as the ambiguity measure. The classical Chebyshev-Cantelli (zeroth partial moment) inequalities, Scarf and Lo (first partial moment) bounds, and semideviation (second partial moment) in one dimension are investigated. The infinite dimensional primal problems are formulated and the simpler finite dimensional dual problems are derived. A principal motivating question is how does data-driven distributional ambiguity affect the moment bounds. Towards answering this question, some theory is developed and computational experiments are conducted for specific problem instances in inventory control and portfolio management. Finally some open questions and suggestions for future research are discussed.

math.OC↗

arXiv2020

Cubic Regularized Newton Method for Saddle Point Models: a Global and Local Convergence Analysis

In this paper, we propose a cubic regularized Newton (CRN) method for solving convex-concave saddle point problems (SPP). At each iteration, a cubic regularized saddle point subproblem is constructed and solved, which provides a search direction for the iterate. With properly chosen stepsizes, the method is shown to converge to the saddle point with global linear and local superlinear convergence rates, if the saddle point function is gradient Lipschitz and strongly-convex-strongly-concave. In the case that the function is merely convex-concave, we propose a homotopy continuation (or path-following) method. Under a Lipschitz-type error bound condition, we present an iteration complexity bound of $\mathcal{O}\left(\ln \left(1/ε\right)\right)$ to reach an $ε$-solution through a homotopy continuation approach, and the iteration complexity bound becomes $\mathcal{O}\left(\left(1/ε\right)^{\frac{1-θ}{θ^2}}\right)$ under a Hölderian-type error bound condition involving a parameter $θ$ ($0<θ<1$).

math.OC↗