arXiv subjects

Naihua Xiu

Publications and source records attributed to Naihua Xiu.

31 records · Page 2Linked to original sources

arXiv2020

Newton method for $\ell_0$-regularized optimization

As a tractable approach, regularization is frequently adopted in sparse optimization. This gives rise to the regularized optimization, aiming at minimizing the $\ell_0$ norm or its continuous surrogates that characterize the sparsity. From the continuity of surrogates to the discreteness of $\ell_0$ norm, the most challenging model is the $\ell_0$-regularized optimization. To conquer this hardness, there is a vast body of work on developing numerically effective methods. However, most of them only enjoy that either the (sub)sequence converges to a stationary point from the deterministic optimization perspective or the distance between each iterate and any given sparse reference point is bounded by an error bound in the sense of probability. In this paper, we develop a Newton-type method for the $\ell_0$-regularized optimization and prove that the generated sequence converges to a stationary point globally and quadratically under the standard assumptions, theoretically explaining that our method is able to perform surprisingly well.

math.OC↗

arXiv2020

Global and Quadratic Convergence of Newton Hard-Thresholding Pursuit

Algorithms based on the hard thresholding principle have been well studied with sounding theoretical guarantees in the compressed sensing and more general sparsity-constrained optimization. It is widely observed in existing empirical studies that when a restricted Newton step was used (as the debiasing step), the hard-thresholding algorithms tend to meet halting conditions in a significantly low number of iterations and are very efficient. Hence, the thus obtained Newton hard-thresholding algorithms call for stronger theoretical guarantees than for their simple hard-thresholding counterparts. This paper provides a theoretical justification for the use of the restricted Newton step. We build our theory and algorithm, Newton Hard-Thresholding Pursuit (NHTP), for the sparsity-constrained optimization. Our main result shows that NHTP is quadratically convergent under the standard assumption of restricted strong convexity and smoothness. We also establish its global convergence to a stationary point under a weaker assumption. In the special case of the compressive sensing, NHTP effectively reduces to some of the existing hard-thresholding algorithms with a Newton step. Consequently, our fast convergence result justifies why those algorithms perform better than without the Newton step. The efficiency of NHTP was demonstrated on both synthetic and real data in compressed sensing and sparse logistic regression.

math.OC↗

arXiv2019

Low-Rank Matrix Optimization Over Affine Set

The low-rank matrix optimization with affine set (rank-MOA) is to minimize a continuously differentiable function over a low-rank set intersecting with an affine set. Under some suitable assumptions, the intersection rule of the Fréchet normal cone to the feasible set of the rank-MOA is established. This allows us to propose the first-order optimality conditions for the rank-MOA in terms of the so-called F-stationary point and the $α$-stationary point. Furthermore, the second-order optimality analysis, including the necessary condition and the sufficient one, is proposed as well. All these results will enrich the theory of low-rank matrix optimization and give potential clues to designing efficient numerical algorithms for seeking low-rank solutions. Meanwhile, we illustrate our proposed optimality analysis for several specific applications of the rank-MOA including the Hankel matrix approximation problem in system identification and the low-rank representation on linear manifold in signal processing.

math.SP↗

arXiv2018

A Lagrangian Dual Based Approach to Sparse Linear Programming

A sparse linear programming (SLP) problem is a linear programming problem equipped with a sparsity (or cardinality) constraint, which is nonconvex and discontinuous theoretically and generally NP-hard computationally due to the combinatorial property involved. By rewriting the sparsity constraint into a disjunctive form, we present an explicit formula of its Lagrangian dual in terms of an unconstrained piecewise-linear convex programming problem which admits a strong duality. A semi-proximal alternating direction method of multipliers (sPADMM) is then proposed to solve this dual problem by taking advantage of the efficient computation of the proximal mapping of the vector Ky-Fan norm function. Based on the optimal solution of the dual problem, we design a dual-primal algorithm for pursuing a global solution of the original SLP problem. Numerical results illustrate that our proposed algorithm is promising especially for large-scale problems.

math.OC↗

arXiv2018

Solving the OSCAR and SLOPE Models Using a Semismooth Newton-Based Augmented Lagrangian Method

The octagonal shrinkage and clustering algorithm for regression (OSCAR), equipped with the $\ell_1$-norm and a pair-wise $\ell_{\infty}$-norm regularizer, is a useful tool for feature selection and grouping in high-dimensional data analysis. The computational challenge posed by OSCAR, for high dimensional and/or large sample size data, has not yet been well resolved due to the non-smoothness and inseparability of the regularizer involved. In this paper, we successfully resolve this numerical challenge by proposing a sparse semismooth Newton-based augmented Lagrangian method to solve the more general SLOPE (the sorted L-one penalized estimation) model. By appropriately exploiting the inherent sparse and low-rank property of the generalized Jacobian of the semismooth Newton system in the augmented Lagrangian subproblem, we show how the computational complexity can be substantially reduced. Our algorithm presents a notable advantage in the high-dimensional statistical regression settings. Numerical experiments are conducted on real data sets, and the results demonstrate that our algorithm is far superior, in both speed and robustness, than the existing state-of-the-art algorithms based on first-order iterative schemes, including the widely used accelerated proximal gradient (APG) method and the alternating direction method of multipliers (ADMM).

math.OC↗

arXiv2016

Z-tensors and complementarity problems

Tensors are multidimensional analogs of matrices. In this paper, based on degree-theoretic ideas, we study homogeneous nonlinear complementarity problems induced by tensors. By specializing this to $Z$-tensors (which are tensors with non-positive off-diagonal entries), we describe various equivalent conditions for a $Z$-tensor to have the global solvability property. We show by an example that the global solvability need not imply unique solvability and provide a sufficient and easily checkable condition for unique solvability.

math.OC↗

arXiv2015

The Sparsest Solutions to $Z$-Tensor Complementarity Problems

Finding the sparsest solutions to a tensor complementarity problem is generally NP-hard due to the nonconvexity and noncontinuity of the involved $\ell_0$ norm. In this paper, a special type of tensor complementarity problems with $Z$-tensors has been considered. Under some mild conditions, we show that to pursuit the sparsest solutions is equivalent to solving polynomial programming with a linear objective function. The involved conditions guarantee the desired exact relaxation and also allow to achieve a global optimal solution to the relaxed nonconvex polynomial programming problem. Particularly, in comparison to existing exact relaxation conditions, such as RIP-type ones, our proposed conditions are easy to verify.

math.SP↗

arXiv2014

Sparse and Low-Rank Covariance Matrices Estimation

This paper aims at achieving a simultaneously sparse and low-rank estimator from the semidefinite population covariance matrices. We first benefit from a convex optimization which develops $l_1$-norm penalty to encourage the sparsity and nuclear norm to favor the low-rank property. For the proposed estimator, we then prove that with large probability, the Frobenious norm of the estimation rate can be of order $O(\sqrt{s(\log{r})/n})$ under a mild case, where $s$ and $r$ denote the number of sparse entries and the rank of the population covariance respectively, $n$ notes the sample capacity. Finally an efficient alternating direction method of multipliers with global convergence is proposed to tackle this problem, and meantime merits of the approach are also illustrated by practicing numerical simulations.

math.ST↗

arXiv2014

Gradient Support Projection Algorithm for Affine Feasibility Problem with Sparsity and Nonnegativity

Let $A$ be a real $M \times N$ measurement matrix and $b\in \mathbb{R}^M$ be an observations vector. The affine feasibility problem with sparsity and nonnegativity ($AFP_{SN}$ for short) is to find a sparse and nonnegative vector $x\in \mathbb{R}^N$ with $Ax=b$ if such $x$ exists. In this paper, we focus on establishment of optimization approach to solving the $AFP_{SN}$. By discussing tangent cone and normal cone of sparse constraint, we give the first necessary optimality conditions, $α$-Stability, T-Stability and N-Stability, and the second necessary and sufficient optimality conditions for the related minimization problems with the $AFP_{SN}$. By adopting Armijo-type stepsize rule, we present a framework of gradient support projection algorithm for the $AFP_{SN}$ and prove its full convergence when matrix $A$ is $s$-regular. By doing some numerical experiments, we show the excellent performance of the new algorithm for the $AFP_{SN}$ without and with noise.

math.OC↗

arXiv2014

Exact Recovery for Sparse Signal via Weighted $l_1$ Minimization

Numerical experiments in literature on compressed sensing have indicated that the reweighted $l_1$ minimization performs exceptionally well in recovering sparse signal. In this paper, we develop exact recovery conditions and algorithm for sparse signal via weighted $l_1$ minimization from the insight of the classical NSP (null space property) and RIC (restricted isometry constant) bound. We first introduce the concept of WNSP (weighted null space property) and reveal that it is a necessary and sufficient condition for exact recovery. We then prove that the RIC bound by weighted $l_1$ minimization is $δ_{ak}<\sqrt{\frac{a-1}{a-1+γ^2}}$, where $a>1$, $0<γ\leq1$ is determined by an optimization problem over the null space. When $γ< 1$ this bound is greater than $\sqrt{\frac{a-1}{a}}$ from $l_1$ minimization. In addition, we also establish the bound on $δ_k$ and show that it can be larger than the sharp one 1/3 via $l_1$ minimization and also greater than 0.4343 via weighted $l_1$ minimization under some mild cases. Finally, we achieve a modified iterative reweighted $l_1$ minimization (MIRL1) algorithm based on our selection principle of weight, and the numerical experiments demonstrate that our algorithm behaves much better than $l_1$ minimization and iterative reweighted $l_1$ minimization (IRL1) algorithm.

cs.IT↗

arXiv2013

New RIC Bounds via l_q-minimization with 0<q<=1 in Compressed Sensing

The restricted isometry constants (RICs) play an important role in exact recovery theory of sparse signals via l_q(0 =4/3 to guarantee the exact recovery of k sparse signals through the l_1 minimization. This paper aims to establish new RICs bounds via l_q(0 1, (ii)several sufficient conditions can be derived, such as for any 0 =2, for any 1/2 =1, (iii) the bound on δ_k is given as well for any 0 =2) is even or δ_k<0.3203 when k(>=3) is odd.

cs.IT↗

arXiv2011

Sufficient Conditions for Low-rank Matrix Recovery, Translated from Sparse Signal Recovery

The low-rank matrix recovery (LMR) is a rank minimization problem subject to linear equality constraints, and it arises in many fields such as signal and image processing, statistics, computer vision, system identification and control. This class of optimization problems is $\N¶$-hard and a popular approach replaces the rank function with the nuclear norm of the matrix variable. In this paper, we extend the concept of $s$-goodness for a sensing matrix in sparse signal recovery (proposed by Juditsky and Nemirovski [Math Program, 2011]) to linear transformations in LMR. Then, we give characterizations of $s$-goodness in the context of LMR. Using the two characteristic $s$-goodness constants, $γ_s$ and $\hatγ_s$, of a linear transformation, not only do we derive necessary and sufficient conditions for a linear transformation to be $s$-good, but also provide sufficient conditions for exact and stable $s$-rank matrix recovery via the nuclear norm minimization under mild assumptions. Moreover, we give computable upper bounds for one of the $s$-goodness characteristics which leads to verifiable sufficient conditions for exact low-rank matrix recovery.

cs.IT↗

arXiv2010

Homogeneous Cone Complementarity Problems and $P$ Properties

We consider existence and uniqueness properties of a solution to homogeneous cone complementarity problem (HCCP). Employing the $T$-algebraic characterization of homogeneous cones, we generalize the $P, P_0, R_0$ properties for a nonlinear function associated with the standard nonlinear complementarity problem to the setting of HCCP. We prove that if a continuous function has either the order-$P_0$ and $R_0$, or the $P_0$ and $R_0$ properties then all the associated HCCPs have solutions. In particular, if a continuous function has the trace-$P$ property then the associated HCCP has a unique solution (if any); if it has the uniform-trace-$P$ property then the associated HCCP has the global uniqueness (of the solution) property (GUS). We present a necessary condition for a nonlinear transformation to have the GUS property. Moreover, we establish a global error bound for the HCCP with the uniform-trace-$P$ property. Finally, we study the HCCP with the relaxation transformation on a $T$-algebra and automorphism invariant properties for homogeneous cone linear complementarity problem.

math.OC↗