arXiv subjects

Naihua Xiu

Publications and source records attributed to Naihua Xiu.

At least 19 recordsLinked to original sources

arXiv2026

A Proximal Point Semismooth Newton Algorithm for Structural Composite Cardinality Optimization

Many image-processing problems can be formulated as composite cardinality optimization (CCOP), whose objective is the sum of two convex terms and a cardinality function composed with a linear operator. The composite cardinality term creates major algorithmic challenges: the separability of the cardinality function is lost and convergence analysis often requires surjectivity-type assumptions on the linear operators. To overcome these challenges, we consider the stationary dual formulation of CCOP, which has more favorable structure consisting of two convex terms and a separable cardinality function. In this paper, we develop an efficient proximal point algorithm (PPA) to solve the stationary dual problem. The efficiency of our PPA stems from two aspects. Firstly, the key step of its subproblem solver minimizes a smooth convex function over a low-dimensional subspace by the classic semismooth Newton algorithm (SNA), which has global convergence and local superlinear rate under suitable conditions. Secondly, implementable inexact criteria are proposed for early termination of the SNA subroutine. These components form the basic framework of our inexact PPA. Under suitable conditions, it enjoys global convergence and local linear convergence rate. In particular, we provide examples in which the convergence assumptions are automatically satisfied. Finally, the SNA subroutine is incorporated into our inexact PPA to solve jump-sparse signal recovery and computed tomography (CT) image restoration. Numerical results demonstrate the time efficiency and solution accuracy of our proposed method.

math.OC↗

arXiv2026

On the Stationary Duality of Structural Composite Cardinality Optimization

Simple cardinality refers to counting nonzero elements of an independent variable satisfying certain properties. Composite cardinality is a simple counting process composited with an affine mapping, and is therefore more complicated than the simple cardinality. We study the composite cardinality optimization problem (CCOP) with structures covering a wide range of applications. Through the use of the stationary duality, we reduce the composite counting to simple counting, and thereby obtain a dual formulation of CCOP. For both primal and dual problems, we investigate the sufficient conditions for the existence of global solutions. Those conditions are validated on representative examples from existing literature. We then show that local solutions of the primal and dual problems are equivalent to their stationary points. This result further helps us establish a one-to-one correspondences between primal and dual local solutions. We also demonstrate that the correspondence holds for a pair of global solutions to the primal and dual problems, provided that the dual weighted parameters are appropriately selected. The reported theoretical results lay foundation for developing numerical algorithms for CCOP in future.

math.OC↗

arXiv2026

0/1 Constrained Optimization Solving Sample Average Approximation for Chance Constrained Programming

Sample average approximation (SAA) is a tractable approach for dealing with chance constrained programming, a challenging stochastic optimization problem. The constraint of SAA is characterized by the $0/1$ loss function which results in considerable complexities in devising numerical algorithms. Most existing methods have been devised based on reformulations of SAA, such as binary integer programming or relaxed problems. However, the development of viable methods to directly tackle SAA remains elusive, let alone providing theoretical guarantees. In this paper, we investigate a general $0/1$ constrained optimization, providing a new way to address SAA rather than its reformulations. Specifically, starting with deriving the Bouligand tangent and Fr$\acute{e}$chet normal cones of the $0/1$ constraint, we establish several optimality conditions. One of them can be equivalently expressed by a system of equations, enabling the development of a semismooth Newton-type algorithm. The algorithm demonstrates a locally superlinear or quadratic convergence rate under standard assumptions, along with nice numerical performance compared to several leading solvers.

math.OC↗

arXiv2026

Local Duality for Sparse Support Vector Machines

Due to the rise of cardinality minimization in optimization, sparse support vector machines (SSVMs) have attracted much attention lately and show certain empirical advantages over convex SVMs. A common way to derive an SSVM is to add a cardinality function such as $\ell_0$-norm to the dual problem of a convex SVM. However, this process lacks theoretical justification. This paper fills the gap by developing a local duality theory for such an SSVM formulation and exploring its relationship with the hinge-loss SVM (hSVM) and the ramp-loss SVM (rSVM). In particular, we prove that the derived SSVM is exactly the dual problem of the 0/1-loss SVM, and the linear representer theorem holds for their local solutions. The local solution of SSVM also provides guidelines on selecting hyperparameters of hSVM and rSVM. {Under specific conditions, we show that a sequence of global solutions of hSVM converges to a local solution of 0/1-loss SVM. Moreover, a local minimizer of 0/1-loss SVM is a local minimizer of rSVM.} This explains why a local solution induced by SSVM outperforms hSVM and rSVM in the prior empirical study. We further conduct numerical tests on real datasets and demonstrate potential advantages of SSVM by working with locally nice solutions proposed in this paper.

cs.LG↗

arXiv2025

Composite Optimization with Indicator Functions: Stationary Duality and a Semismooth Newton Method

Indicator functions of taking values of zero or one are essential to numerous applications in machine learning and statistics. The corresponding primal optimization model has been researched in several recent works. However, its dual problem is a more challenging topic that has not been well addressed. One possible reason is that the Fenchel conjugate of any indicator function is finite only at the origin. This work aims to explore the dual optimization for the sum of a strongly convex function and a composite term with indicator functions on positive intervals. For the first time, a dual problem is constructed by extending the classic conjugate subgradient property to the indicator function. This extension further helps us establish the equivalence between the primal and dual solutions. The dual problem turns out to be a sparse optimization with a $\ell_0$ regularizer and a nonnegative constraint. The proximal operator of the sparse regularizer is used to identify a dual subspace to implement gradient and/or semismooth Newton iteration with low computational complexity. This gives rise to a dual Newton-type method with both global convergence and local superlinear (or quadratic) convergence rate under mild conditions. Finally, when applied to AUC maximization and sparse multi-label classification, our dual Newton method demonstrates satisfactory performance on computational speed and accuracy.

math.OC↗

arXiv2023

0/1 Deep Neural Networks via Block Coordinate Descent

The step function is one of the simplest and most natural activation functions for deep neural networks (DNNs). As it counts 1 for positive variables and 0 for others, its intrinsic characteristics (e.g., discontinuity and no viable information of subgradients) impede its development for several decades. Even if there is an impressive body of work on designing DNNs with continuous activation functions that can be deemed as surrogates of the step function, it is still in the possession of some advantageous properties, such as complete robustness to outliers and being capable of attaining the best learning-theoretic guarantee of predictive accuracy. Hence, in this paper, we aim to train DNNs with the step function used as an activation function (dubbed as 0/1 DNNs). We first reformulate 0/1 DNNs as an unconstrained optimization problem and then solve it by a block coordinate descend (BCD) method. Moreover, we acquire closed-form solutions for sub-problems of BCD as well as its convergence properties. Furthermore, we also integrate $\ell_{2,0}$-regularization into 0/1 DNN to accelerate the training process and compress the network scale. As a result, the proposed algorithm has a high performance on classifying MNIST and Fashion-MNIST datasets. As a result, the proposed algorithm has a desirable performance on classifying MNIST, FashionMNIST, Cifar10, and Cifar100 datasets.

cs.LG↗

arXiv2023

Sparse SVM with Hard-Margin Loss: a Newton-Augmented Lagrangian Method in Reduced Dimensions

The hard margin loss function has been at the core of the support vector machine (SVM) research from the very beginning due to its generalization capability.On the other hand, the cardinality constraint has been widely used for feature selection, leading to sparse solutions. This paper studies the sparse SVM with the hard-margin loss (SSVM-HM) that integrates the virtues of both worlds. However, SSVM-HM is one of the most challenging models to solve. In this paper, we cast the problem as a composite optimization with the cardinality constraint. We characterize its local minimizers in terms of {\rm P}-stationarity that well captures the combinatorial structure of the problem. We then propose an inexact proximal augmented Lagrangian method (iPAL). The different parts of the inexactness measurements from the {\rm P}-stationarity are controlled at different scales in a way that the generated sequence converges both globally and at a linear rate. This matches the best convergence theory for composite optimization. To make iPAL practically efficient, we propose a gradient-Newton method in a subspace for the iPAL subproblem. This is accomplished by detecting active samples and features with the help of the proximal operator of the hard margin loss and the projection of cardinality constraint. Extensive numerical results on both simulated and real datasets demonstrate that the proposed method is fast, produces sparse solution of high accuracy, and can lead to effective reduction on active samples and features when compared with several leading solvers.

math.OC↗

arXiv2023

iNALM: An inexact Newton Augmented Lagrangian Method for Zero-One Composite Optimization

Zero-One Composite Optimization (0/1-COP) is a prototype of nonsmooth, nonconvex optimization problems and it has attracted much attention recently. The augmented Lagrangian Method (ALM) has stood out as a leading methodology for such problems. The main purpose of this paper is to extend the classical theory of ALM from smooth problems to 0/1-COP. We propose, for the first time, second-order optimality conditions for 0/1-COP. In particular, under a second-order sufficient condition (SOSC), we prove the R-linear convergence rate of the proposed ALM. In order to identify the subspace used in SOSC, we employ the proximal operator of the 0/1-loss function, leading to an active-set identification technique. Built around this identification process, we design practical stopping criteria for any algorithm to be used for the subproblem of ALM. We justify that Newton's method is an ideal candidate for the subproblem and it enjoys both global and local quadratic convergence. Those considerations result in an inexact Newton ALM (iNALM). The method of iNALM is unique in the sense that it is active-set based, it is inexact (hence more practical), and SOSC plays an important role in its R-linear convergence analysis. The numerical results on both simulated and real datasets show the fast running speed and high accuracy of iNALM when compared with several leading solvers.

math.OC↗

arXiv2022

Optimality Analysis and Block Sparse Algorithm for Complex Compressed Sensing

Recently, many new challenges in Compressed Sensing (CS), such as block sparsity, arose. In this paper, we present a new algorithm for solving CS with block sparse constraints (BSC) in complex fields. Firstly, based on block sparsity characteristics, we propose a new model to deal with CS with BSC and analyze the properties of the functions involved in this model. We then present a new $τ$-stationary point and analyze corresponding first-order sufficient and necessary conditions. That ensures we to further develop a block Newton hard-thresholding pursuit (BNHTP) algorithm for efficiently solving CS with BSC. Finally, preliminary numerical experiments demonstrate that the BNHTP algorithm has superior performance in terms of recovery accuracy and calculation time when compared with the classical AMP algorithm.

cs.IT↗

arXiv2021

Heaviside Set Constrained Optimization: Optimality and Newton Method

Data in the real world frequently involve binary status: truth or falsehood, positiveness or negativeness, similarity or dissimilarity, spam or non-spam, and to name a few, with applications into the regression, classification problems and so on. To characterize the binary status, one of the ideal functions is the Heaviside step function that returns one for one status and zero for the other. Hence, it is of dis-continuity. Because of this, the conventional approaches to deal with the binary status tremendously benefit from its continuous surrogates. In this paper, we target the Heaviside step function directly and study the Heaviside set constrained optimization: calculating the tangent and normal cones of the feasible set, establishing several first-order sufficient and necessary optimality conditions, as well as developing a Newton type method that enjoys locally quadratic convergence and excellent numerical performance.

math.OC↗

arXiv2021

Quadratic Convergence of Smoothing Newton's Method for 0/1 Loss Optimization

It has been widely recognized that the 0/1 loss function is one of the most natural choices for modelling classification errors, and it has a wide range of applications including support vector machines and 1-bit compressed sensing. Due to the combinatorial nature of the 0/1 loss function, methods based on convex relaxations or smoothing approximations have dominated the existing research and are often able to provide approximate solutions of good quality. However, those methods are not optimizing the 0/1 loss function directly and hence no optimality has been established for the original problem. This paper aims to study the optimality conditions of the 0/1 function minimization, and for the first time to develop Newton's method that directly optimizes the 0/1 function with a local quadratic convergence under reasonable conditions. Extensive numerical experiments demonstrate its superior performance as one would expect from Newton-type methods.ions. Extensive numerical experiments demonstrate its superior performance as one would expect from Newton-type methods.

math.OC↗

arXiv2021

Global Convergence of Inexact Augmented Lagrangian Method for Zero-One Composite Optimization

We consider the problem of minimizing the sum of a smooth function and a composition of a zero-one loss function with a linear operator, namely zero-one composite optimization problem (0/1-COP). It is a versatile model including the support vector machine (SVM), multi-label classification (MLC), maximum rank correlation (MRC) and so on. However, due to the nonconvexity, discontinuity and NP-hardness of the 0/1-COP, it is intractable to design a globally convergent algorithm and the work attempting to solve it directly is scarce. In this paper, we first define and characterize the proximal stationarity to derive the minimum and the strongly exact penalization of the Lyapunov function, which is a variant of the augmented Lagrangian function for the 0/1-COP. Based on this, we propose an inexact augmented Lagrangian method (IALM) for solving 0/1-COP, where the subproblem is solved by the zero-one Bregman alternating linearized minimization (0/1-BALM) algorithm with low computational complexity. Under some suitable assumptions, we prove that the whole sequence generated by the IALM converges to the local minimizer of 0/1-COP. As a direct application, we obtain the global convergence of IALM under the assumption that the data matrix is full row rank for solving the SVM, MLC and MRC.

math.OC↗

arXiv2021

Computing One-bit Compressive Sensing via Double-Sparsity Constrained Optimization

One-bit compressive sensing gains its popularity in signal processing and communications due to its low storage costs and low hardware complexity. However, it has been a challenging task to recover the signal only by exploiting the one-bit (the sign) information. In this paper, we appropriately formulate the one-bit compressive sensing into a double-sparsity constrained optimization problem. The first-order optimality conditions for this nonconvex and discontinuous problem are established via the newly introduced $τ$-stationarity, based on which, a gradient projection subspace pursuit (\texttt{GPSP}) algorithm is developed. It is proven that \texttt{GPSP} can converge globally and terminate within finite steps. Numerical experiments have demonstrated its excellent performance in terms of a high order of accuracy with a fast computational speed.

math.OC↗

arXiv2021

An Extended Newton-type Algorithm for $\ell_2$-Regularized Sparse Logistic Regression and Its Efficiency for Classifying Large-scale Datasets

Sparse logistic regression, as an effective tool of classification, has been developed tremendously in recent two decades, from its origination the $\ell_1$-regularized version to the sparsity constrained models. This paper is carried out on the sparsity constrained logistic regression by the Newton method. We begin with establishing its first-order optimality condition associated with a $τ$-stationary point. This point can be equivalently interpreted as a system of equations which is then efficiently solved by the Newton method. The method has a considerably low computational complexity and enjoys global and quadratic convergence properties. Numerical experiments on random and real data demonstrate its superior performance when against seven state-of-the-art solvers.

math.OC↗

arXiv2021

Single and Union Non-parallel Support Vector Machine Frameworks

Considering the classification problem, we summarize the nonparallel support vector machines with the nonparallel hyperplanes to two types of frameworks. The first type constructs the hyperplanes separately. It solves a series of small optimization problems to obtain a series of hyperplanes, but is hard to measure the loss of each sample. The other type constructs all the hyperplanes simultaneously, and it solves one big optimization problem with the ascertained loss of each sample. We give the characteristics of each framework and compare them carefully. In addition, based on the second framework, we construct a max-min distance-based nonparallel support vector machine for multiclass classification problem, called NSVM. It constructs hyperplanes with large distance margin by solving an optimization problem. Experimental results on benchmark data sets show the advantages of our NSVM.

cs.LG↗

arXiv2021

A Lagrange-Newton Algorithm for Sparse Nonlinear Programming

The sparse nonlinear programming (SNP) problem has wide applications in signal and image processing, machine learning, pattern recognition, finance and management, etc. However, the computational challenge posed by SNP has not yet been well resolved due to the nonconvex and discontinuous $\ell_0$-norm involved. In this paper, we resolve this numerical challenge by developing a fast Newton-type algorithm. As a theoretical cornerstone, we establish a first-order optimality condition for SNP based on the concept of strong $β$-Lagrangian stationarity via the Lagrangian function, and reformulate it as a system of nonlinear equations called the Lagrangian equations. The nonsingularity of the corresponding Jacobian is discussed, based on which the Lagrange-Newton algorithm (LNA) is then proposed. Under mild conditions, we establish the locally quadratic convergence and the iterative complexity estimation of LNA. To further demonstrate the efficiency and superiority of our proposed algorithm, we apply LNA to solve two specific application problems arising from compressed sensing and sparse high-order portfolio selection, in which significant benefits accrue from the restricted Newton step in LNA.

math.OC↗

arXiv2021

Support Vector Machine Classifier via $L_{0/1}$ Soft-Margin Loss

Support vector machine (SVM) has attracted great attentions for the last two decades due to its extensive applications, and thus numerous optimization models have been proposed. To distinguish all of them, in this paper, we introduce a new model equipped with an $L_{0/1}$ soft-margin loss (dubbed as $L_{0/1}$-SVM) which well captures the nature of the binary classification. Many of the existing convex/non-convex soft-margin losses can be viewed as a surrogate of the $L_{0/1}$ soft-margin loss. Despite the discrete nature of $L_{0/1}$, we manage to establish the existence of global minimizer of the new model as well as revealing the relationship among its minimizers and KKT/P-stationary points. These theoretical properties allow us to take advantage of the alternating direction method of multipliers. In addition, the $L_{0/1}$-support vector operator is introduced as a filter to prevent outliers from being support vectors during the training process. Hence, the method is expected to be relatively robust. Finally, numerical experiments demonstrate that our proposed method generates better performance in terms of much shorter computational time with much fewer number of support vectors when against with some other leading methods in areas of SVM. When the data size gets bigger, its advantage becomes more evident.

math.OC↗

arXiv2020

Proximal Operator and Optimality Conditions for Ramp Loss SVM

Support vector machines with ramp loss (dubbed as $L_r$-SVM) have attracted wide attention due to the boundedness of ramp loss. However, the corresponding optimization problem is non-convex and the given Karush-Kuhn-Tucker (KKT) conditions are only the necessary conditions. To enrich the optimality theory of $L_r$-SVM and go deep into its statistical nature, we first introduce and analyze the proximal operator for ramp loss, and then establish a stronger optimality conditions: P-stationarity, which is proved to be the first-order necessary and sufficient conditions for local minimizer of $L_r$-SVM. Finally, we define the $L_r$ support vectors based on the concept of P-stationary point, and show that all $L_r$ support vectors fall into the support hyperplanes, which possesses the same feature as the one of hard margin SVM.

math.OC↗