arXiv subjects

Guanyi Wang

Publications and source records attributed to Guanyi Wang.

24 records · Page 2Linked to original sources

arXiv2020

Orthant Based Proximal Stochastic Gradient Method for $\ell_1$-Regularized Optimization

Sparsity-inducing regularization problems are ubiquitous in machine learning applications, ranging from feature selection to model compression. In this paper, we present a novel stochastic method -- Orthant Based Proximal Stochastic Gradient Method (OBProx-SG) -- to solve perhaps the most popular instance, i.e., the l1-regularized problem. The OBProx-SG method contains two steps: (i) a proximal stochastic gradient step to predict a support cover of the solution; and (ii) an orthant step to aggressively enhance the sparsity level via orthant face projection. Compared to the state-of-the-art methods, e.g., Prox-SG, RDA and Prox-SVRG, the OBProx-SG not only converges to the global optimal solutions (in convex scenario) or the stationary points (in non-convex scenario), but also promotes the sparsity of the solutions substantially. Particularly, on a large number of convex problems, OBProx-SG outperforms the existing methods comprehensively in the aspect of sparsity exploration and objective values. Moreover, the experiments on non-convex deep neural networks, e.g., MobileNetV1 and ResNet18, further demonstrate its superiority by achieving the solutions of much higher sparsity without sacrificing generalization accuracy.

math.OC↗

arXiv2019

An Approximation Algorithm for training One-Node ReLU Neural Network

Training a one-node neural network with ReLU activation function (One-Node-ReLU) is a fundamental optimization problem in deep learning. In this paper, we begin with proving the NP-hardness of training One-Node-ReLU. We then present an approximation algorithm to solve One-Node-ReLU whose running time is $\mathcal{O}(n^k)$ where $n$ is the number of samples, $k$ is a predefined integral constant. Except $k$, this algorithm does not require pre-processing or tuning of parameters. We analyze the performance of this algorithm under various regimes. First, given any arbitrary set of training sample data set, we show that the algorithm guarantees a $\frac{n}{k}$-approximation for training One-Node-ReLU problem. As a consequence, in the realizable case (i.e. when the training error is zero), this approximation algorithm achieves the global optimal solution for the One-Node-ReLU problem. Second, we assume that the training sample data is obtained from an underlying one-node neural network with ReLU activation function, where the output is perturbed by a Gaussian noise. In this regime, we show that the same approximation algorithm guarantees a much better asymptotic approximation ratio which is independent of the number of samples $n$. Finally, we conduct extensive empirical studies and arrive at two conclusions. One, the approximation algorithm together with some heuristic performs better than gradient descent algorithm. Two, the solution of the approximation algorithm can be used as starting point for gradient descent -- a combination that works significantly better than gradient descent.

math.OC↗

arXiv2017

Sparse principal component analysis and its $l_1$-relaxation

Principal component analysis (PCA) is one of the most widely used dimensionality reduction methods in scientific data analysis. In many applications, for additional interpretability, it is desirable for the factor loadings to be sparse, that is, we solve PCA with an additional cardinality (l0) constraint. The resulting optimization problem is called the sparse principal component analysis (SPCA). One popular approach to achieve sparsity is to replace the l0 constraint by an l1 constraint. In this paper, we prove that, independent of the data, the optimal objective function value of the problem with l0 constraint is within a constant factor of the the optimal objective function value of the problem with l1 constraint. To the best of our knowledge, this is the first formal relationship established between the l0 and the l1 constraint version of the problem.

math.OC↗

arXiv2017

The Strength of Multi-row Aggregation Cuts for Sign-pattern Integer Programs

In this paper, we study the strength of aggregation cuts for sign-pattern integer programs (IPs). Sign-pattern IPs are a generalization of packing IPs and are of the form $\{x\in \mathbb{Z}^n_+\ | \ Ax\le b\}$ where for a given column $j$, $A_{ij}$ is either non-negative for all $i$ or non-positive for all $i$. Our first result is that the aggregation closure for such sign-pattern IPs can be 2-approximated by the original 1-row closure. This generalizes a result for packing IPs. On the other hand, unlike in the case of packing IPs, we show that the multi-row aggregation closure cannot be well approximated by the original multi-row closure. Therefore for these classes of integer programs general aggregated multi-row cutting planes can perform significantly better than just looking at cuts from multiple original constraints.

math.OC↗

arXiv2017

Mixed-integer linear representability, disjunctions, and Chvatal functions --- modeling implications

Jeroslow and Lowe gave an exact geometric characterization of subsets of $\mathbb{R}^n$ that are projections of mixed-integer linear sets, also known as MILP-representable or MILP-R sets. We give an alternate algebraic characterization by showing that a set is MILP-R {\em if and only if} the set can be described as the intersection of finitely many {\em affine Chvatal inequalities} in continuous variables (termed AC sets). These inequalities are a modification of a concept introduced by Blair and Jeroslow. Unlike the case for linear inequalities, allowing for integer variables in Chvatal inequalities and projection does not enhance modeling power. We show that the MILP-R sets are still precisely those sets that are modeled as affine Chvatal inequalites with integer variables. Furthermore, the projection of a set defined by affine Chvatal inequalites with integer variables is still an MILP-R set. We give a sequential variable elimination scheme that, when applied to a MILP-R set yields the AC set characterization. This is related to the elimination scheme of Williams and Williams-Hooker, who describe projections of integer sets using \emph{disjunctions} of affine Chvatal systems. We show that disjunctions are unnecessary by showing how to find the affine Chvatal inequalities that cannot be discovered by the Williams-Hooker scheme. This allows us to answer a long-standing open question due to Ryan (1991) on designing an elimination scheme to represent finitely-generated integral monoids as a system of Chvatal inequalities \emph{without} disjunctions. Finally, our work can be seen as a generalization of the approach of Blair and Jeroslow, and Schrijver for constructing consistency testers for integer programs to general AC sets.

math.OC↗

arXiv2016

Mixed-integer linear representability, disjunctions, and variable elimination

Jeroslow and Lowe gave an exact geometric characterization of subsets of $\mathbb{R}^n$ that are projections of mixed-integer linear sets, a.k.a MILP-representable sets. We give an alternate algebraic characterization by showing that a set is MILP-representable {\em if and only if} the set can be described as the intersection of finitely many {\em affine Chvátal inequalities}. These inequalities are a modification of a concept introduced by Blair and Jeroslow. This gives a sequential variable elimination scheme that, when applied to the MILP representation of a set, explicitly gives the affine Chvátal inequalities characterizing the set. This is related to the elimination scheme of Wiliams and Williams-Hooker, who describe projections of integer sets using \emph{disjunctions} of Chvátal systems. Our scheme extends their work in two ways. First, we show that disjunctions are unnecessary, by showing how to find the affine Chvátal inequalities that cannot be discovered by the Williams-Hooker scheme. Second, disjunctions of Chvátal systems can give sets that are \emph{not} projections of mixed-integer linear sets; so the Williams-Hooker approach does not give an exact characterization of MILP representability.

math.OC↗