arXiv subjects

Shenglong Zhou

Publications and source records attributed to Shenglong Zhou.

48 records · Page 3Linked to original sources

arXiv2021

Sparse SVM for Sufficient Data Reduction

Kernel-based methods for support vector machines (SVM) have shown highly advantageous performance in various applications. However, they may incur prohibitive computational costs for large-scale sample datasets. Therefore, data reduction (reducing the number of support vectors) appears to be necessary, which gives rise to the topic of the sparse SVM. Motivated by this problem, the sparsity constrained kernel SVM optimization has been considered in this paper in order to control the number of support vectors. Based on the established optimality conditions associated with the stationary equations, a Newton-type method is developed to handle the sparsity constrained optimization. This method is found to enjoy the one-step convergence property if the starting point is chosen to be close to a local region of a stationary point, thereby leading to a super-high computational speed. Numerical comparisons with several powerful solvers demonstrate that the proposed method performs exceptionally well, particularly for large-scale datasets in terms of a much lower number of support vectors and shorter computational time.

math.OC↗

arXiv2021

Support Vector Machine Classifier via $L_{0/1}$ Soft-Margin Loss

Support vector machine (SVM) has attracted great attentions for the last two decades due to its extensive applications, and thus numerous optimization models have been proposed. To distinguish all of them, in this paper, we introduce a new model equipped with an $L_{0/1}$ soft-margin loss (dubbed as $L_{0/1}$-SVM) which well captures the nature of the binary classification. Many of the existing convex/non-convex soft-margin losses can be viewed as a surrogate of the $L_{0/1}$ soft-margin loss. Despite the discrete nature of $L_{0/1}$, we manage to establish the existence of global minimizer of the new model as well as revealing the relationship among its minimizers and KKT/P-stationary points. These theoretical properties allow us to take advantage of the alternating direction method of multipliers. In addition, the $L_{0/1}$-support vector operator is introduced as a filter to prevent outliers from being support vectors during the training process. Hence, the method is expected to be relatively robust. Finally, numerical experiments demonstrate that our proposed method generates better performance in terms of much shorter computational time with much fewer number of support vectors when against with some other leading methods in areas of SVM. When the data size gets bigger, its advantage becomes more evident.

math.OC↗

arXiv2020

BOLIB: Bilevel Optimization LIBrary of test problems

This chapter presents the Bilevel Optimization LIBrary of the test problems (BOLIB for short), which contains a collection of test problems, with continuous variables, to help support the development of numerical solvers for bilevel optimization. The library contains 173 examples with 138 nonlinear, 24 linear, and 11 simple bilevel optimization problems. This BOLIB collection is probably the largest bilevel optimization library of test problems. Moreover, as the library is computationenabled with the MATLAB m-files of all the examples, it provides a uniform basis for testing and comparing algorithms. The library, together with all the related codes, is freely available at biopt.github.io/bolib.

math.OC↗

arXiv2020

Theoretical and numerical comparison of the Karush-Kuhn-Tucker and value function reformulations in bilevel optimization

The Karush-Kuhn-Tucker and value function (lower-level value function, to be precise) reformulations are the most common single-level transformations of the bilevel optimization problem. So far, these reformulations have either been studied independently or as a joint optimization problem in an attempt to take advantage of the best properties from each model. To the best of our knowledge, these reformulations have not yet been compared in the existing literature. This paper is a first attempt towards establishing whether one of these reformulations is best at solving a given class of the optimistic bilevel optimization problem. We design a comparison framework, which seems fair, considering the theoretical properties of these reformulations. This work reveals that although none of the models seems to particularly dominate the other from the theoretical point of view, the value function reformulation seems to numerically outperform the Karush-Kuhn-Tucker reformulation on a Newton-type algorithm. The computational experiments here are mostly based on test problems from the Bilevel Optimization LIBrary (BOLIB).

math.OC↗

arXiv2020

Newton method for $\ell_0$-regularized optimization

As a tractable approach, regularization is frequently adopted in sparse optimization. This gives rise to the regularized optimization, aiming at minimizing the $\ell_0$ norm or its continuous surrogates that characterize the sparsity. From the continuity of surrogates to the discreteness of $\ell_0$ norm, the most challenging model is the $\ell_0$-regularized optimization. To conquer this hardness, there is a vast body of work on developing numerically effective methods. However, most of them only enjoy that either the (sub)sequence converges to a stationary point from the deterministic optimization perspective or the distance between each iterate and any given sparse reference point is bounded by an error bound in the sense of probability. In this paper, we develop a Newton-type method for the $\ell_0$-regularized optimization and prove that the generated sequence converges to a stationary point globally and quadratically under the standard assumptions, theoretically explaining that our method is able to perform surprisingly well.

math.OC↗

arXiv2020

Global and Quadratic Convergence of Newton Hard-Thresholding Pursuit

Algorithms based on the hard thresholding principle have been well studied with sounding theoretical guarantees in the compressed sensing and more general sparsity-constrained optimization. It is widely observed in existing empirical studies that when a restricted Newton step was used (as the debiasing step), the hard-thresholding algorithms tend to meet halting conditions in a significantly low number of iterations and are very efficient. Hence, the thus obtained Newton hard-thresholding algorithms call for stronger theoretical guarantees than for their simple hard-thresholding counterparts. This paper provides a theoretical justification for the use of the restricted Newton step. We build our theory and algorithm, Newton Hard-Thresholding Pursuit (NHTP), for the sparsity-constrained optimization. Our main result shows that NHTP is quadratically convergent under the standard assumption of restricted strong convexity and smoothness. We also establish its global convergence to a stationary point under a weaker assumption. In the special case of the compressive sensing, NHTP effectively reduces to some of the existing hard-thresholding algorithms with a Newton step. Consequently, our fast convergence result justifies why those algorithms perform better than without the Newton step. The efficiency of NHTP was demonstrated on both synthetic and real data in compressed sensing and sparse logistic regression.

math.OC↗

arXiv2020

Newton Hard Thresholding Pursuit for Sparse LCP via A New Merit Function

Solutions to the linear complementarity problem (LCP) are naturally sparse in many applications such as bimatrix games and portfolio section problems. Despite that it gives rise to the hardness, sparsity makes optimization faster and enables relatively large scale computation. Motivated by this, we take the sparse LCP into consideration, investigating the existence and boundedness of its solution set as well as introducing a new merit function, which allows us to convert the problem into a sparsity constrained optimization. The function turns out to be continuously differentiable and twice continuously differentiable for some chosen parameters. Interestingly, it is also convex if the involved matrix is positive semidefinite. We then explore the relationship between the solution set to the sparse LCP and stationary points of the sparsity constrained optimization. Finally, Newton hard thresholding pursuit is adopted to solve the sparsity constrained model. Numerical experiments demonstrate that the problem can be efficiently solved through the new merit function.

math.OC↗

arXiv2019

Semismooth Newton-type method for bilevel optimization: Global convergence and extensive numerical experiments

We consider the standard optimistic bilevel optimization problem, in particular upper- and lower-level constraints can be coupled. By means of the lower-level value function, the problem is transformed into a single-level optimization problem with a penalization of the value function constraint. For treating the latter problem, we develop a framework that does not rely on the direct computation of the lower-level value function or its derivatives. For each penalty parameter, the framework leads to a semismooth system of equations. This allows us to extend the semismooth Newton method to bilevel optimization. Besides global convergence properties of the method, we focus on achieving local superlinear convergence to a solution of the semismooth system. To this end, we formulate an appropriate CD-regularity assumption and derive suffcient conditions so that it is fulfilled. Moreover, we develop conditions to guarantee that a solution of the semismooth system is a local solution of the bilevel optimization problem. Extensive numerical experiments on $124$ examples of nonlinear bilevel optimization problems from the literature show that this approach exhibits a remarkable performance, where only a few penalty parameters need to be considered.

math.OC↗

arXiv2014

Sparse and Low-Rank Covariance Matrices Estimation

This paper aims at achieving a simultaneously sparse and low-rank estimator from the semidefinite population covariance matrices. We first benefit from a convex optimization which develops $l_1$-norm penalty to encourage the sparsity and nuclear norm to favor the low-rank property. For the proposed estimator, we then prove that with large probability, the Frobenious norm of the estimation rate can be of order $O(\sqrt{s(\log{r})/n})$ under a mild case, where $s$ and $r$ denote the number of sparse entries and the rank of the population covariance respectively, $n$ notes the sample capacity. Finally an efficient alternating direction method of multipliers with global convergence is proposed to tackle this problem, and meantime merits of the approach are also illustrated by practicing numerical simulations.

math.ST↗

arXiv2014

Gradient Support Projection Algorithm for Affine Feasibility Problem with Sparsity and Nonnegativity

Let $A$ be a real $M \times N$ measurement matrix and $b\in \mathbb{R}^M$ be an observations vector. The affine feasibility problem with sparsity and nonnegativity ($AFP_{SN}$ for short) is to find a sparse and nonnegative vector $x\in \mathbb{R}^N$ with $Ax=b$ if such $x$ exists. In this paper, we focus on establishment of optimization approach to solving the $AFP_{SN}$. By discussing tangent cone and normal cone of sparse constraint, we give the first necessary optimality conditions, $α$-Stability, T-Stability and N-Stability, and the second necessary and sufficient optimality conditions for the related minimization problems with the $AFP_{SN}$. By adopting Armijo-type stepsize rule, we present a framework of gradient support projection algorithm for the $AFP_{SN}$ and prove its full convergence when matrix $A$ is $s$-regular. By doing some numerical experiments, we show the excellent performance of the new algorithm for the $AFP_{SN}$ without and with noise.

math.OC↗

arXiv2014

Exact Recovery for Sparse Signal via Weighted $l_1$ Minimization

Numerical experiments in literature on compressed sensing have indicated that the reweighted $l_1$ minimization performs exceptionally well in recovering sparse signal. In this paper, we develop exact recovery conditions and algorithm for sparse signal via weighted $l_1$ minimization from the insight of the classical NSP (null space property) and RIC (restricted isometry constant) bound. We first introduce the concept of WNSP (weighted null space property) and reveal that it is a necessary and sufficient condition for exact recovery. We then prove that the RIC bound by weighted $l_1$ minimization is $δ_{ak}<\sqrt{\frac{a-1}{a-1+γ^2}}$, where $a>1$, $0<γ\leq1$ is determined by an optimization problem over the null space. When $γ< 1$ this bound is greater than $\sqrt{\frac{a-1}{a}}$ from $l_1$ minimization. In addition, we also establish the bound on $δ_k$ and show that it can be larger than the sharp one 1/3 via $l_1$ minimization and also greater than 0.4343 via weighted $l_1$ minimization under some mild cases. Finally, we achieve a modified iterative reweighted $l_1$ minimization (MIRL1) algorithm based on our selection principle of weight, and the numerical experiments demonstrate that our algorithm behaves much better than $l_1$ minimization and iterative reweighted $l_1$ minimization (IRL1) algorithm.

cs.IT↗

arXiv2013

New RIC Bounds via l_q-minimization with 0<q<=1 in Compressed Sensing

The restricted isometry constants (RICs) play an important role in exact recovery theory of sparse signals via l_q(0 =4/3 to guarantee the exact recovery of k sparse signals through the l_1 minimization. This paper aims to establish new RICs bounds via l_q(0 1, (ii)several sufficient conditions can be derived, such as for any 0 =2, for any 1/2 =1, (iii) the bound on δ_k is given as well for any 0 =2) is even or δ_k<0.3203 when k(>=3) is odd.

cs.IT↗