arXiv subjects

Wenxun Xing

Publications and source records attributed to Wenxun Xing.

11 recordsLinked to original sources

arXiv2026

Stochastic versus Deterministic in Stochastic Gradient Descent

This paper theoretically reanalyzes the convergence of the mini-batch stochastic gradient descent (SGD) for a structured minimization problem involving a finite-sum function with its gradient being stochastically approximated, and an independent term with its gradient being deterministically computed. Rather than collapsing this problem into a standard finite-sum formulation and treating all components uniformly, we study it from a stochastic versus deterministic viewpoint and focus on how these two gradient computations affect mini-batch SGD differently. The step size, the convergence rate, and the radius of the convergence region depend asymmetrically on the characteristics of the two components, which shows the distinct impacts of stochastic approximation versus deterministic computation in the mini-batch SGD. Based on this, we show that our analysis yields a faster convergence rate and a smaller radius of the convergence region. Moreover, an even better convergence rate can be obtained when the independent term endows the objective function with sufficient strong convexity. Also, the convergence rate of our algorithm in expectation approaches that of the classic gradient descent when the batch size increases. Numerical experiments are conducted to support the theoretical analysis as well.

math.OC↗

arXiv2025

Depth-first directional search for nonconvex optimization

Random search methods are widely used for global optimization due to their theoretical generality and implementation simplicity. This paper proposes a depth-first directional search (DFDS) algorithm for globally solving nonconvex optimization problems. Motivated by the penetrating beam of a searchlight, DFDS performs a complete stepping line search along each sampled direction before proceeding to the next, contrasting with existing directional search methods that prioritize broad exploratory coverage. We establish the convergence and computational complexity of DFDS through a novel geometric framework that models the success probability of finding a global optimizer as the surface area of a spherical cap. Numerical experiments on benchmark problems demonstrate that DFDS achieves significantly higher accuracy in locating the global optimum compared to other random search methods under the same function evaluation budget.

math.OC↗

arXiv2025

Stable gradient-adjusted root mean square propagation on least squares problem

Root mean square propagation (abbreviated as RMSProp) is a first-order stochastic algorithm used in machine learning widely. In this paper, a stable gradient-adjusted RMSProp (abbreviated as SGA-RMSProp) with mini-batch stochastic gradient is proposed, and its properties are studied on the linear least squares problem. R-linear convergence of the algorithm is established on the consistent linear least squares problem. The algorithm is also proved to converge R-linearly to a neighborhood of the minimizer for the inconsistent case, with the region of the neighborhood being controlled by the batch size. Furthermore, numerical experiments are conducted to compare the performances of SGA-RMSProp, stochastic gradient descent (abbreviated as SGD), and the original RMSProp with different batch sizes. The faster initial convergence rate of SGA-RMSProp is observed through numerical experiments and an adaptive strategy for switching from SGA-RMSProp to SGD is proposed, which combines the benefits of these two algorithms.

math.OC↗

arXiv2025

Globalized distributionally robust chance-constrained support vector machine based on core sets

Support vector machine (SVM) is a well known binary linear classification model in supervised learning. This paper proposes a globalized distributionally robust chance-constrained (GDRC) SVM model based on core sets to address uncertainties in the dataset and provide a robust classifier. The globalization means that we focus on the uncertainty in the sample population rather than the small perturbations around each sample point. The uncertainty is mainly specified by the confidence region of the first- and second-order moments. The core sets are constructed to capture some small regions near the potential classification hyperplane, which helps improve the classification quality via the expected distance constraint of the random vector to core sets. We obtain the equivalent semi-definite programming reformulation of the GDRC SVM model under some appropriate assumptions. To deal with the large-scale problem, an approximation approach based on principal component analysis is applied to the GDRC SVM. The numerical experiments are presented to illustrate the effectiveness and advantage of our model.

math.OC↗

arXiv2025

ADMM Algorithms for Residual Network Training: Convergence Analysis and Parallel Implementation

We propose both serial and parallel proximal (linearized) alternating direction method of multipliers (ADMM) algorithms for training residual neural networks. In contrast to backpropagation-based approaches, our methods inherently mitigate the exploding gradient issue and are well-suited for parallel and distributed training through regional updates. Theoretically, we prove that the proposed algorithms converge at an R-linear (sublinear) rate for both the iteration points and the objective function values. These results hold without imposing stringent constraints on network width, depth, or training data size. Furthermore, we theoretically analyze our parallel/distributed ADMM algorithms, highlighting their reduced time complexity and lower per-node memory consumption. To facilitate practical deployment, we develop a control protocol for parallel ADMM implementation using Python's multiprocessing and interprocess communication. Experimental results validate the proposed ADMM algorithms, demonstrating rapid and stable convergence, improved performance, and high computational efficiency. Finally, we highlight the improved scalability and efficiency achieved by our parallel ADMM training strategy.

cs.LG↗

arXiv2024

Globalized distributionally robust optimization with multi core sets

It is essential to capture the true probability distribution of uncertain data in the distributionally robust optimization (DRO). The uncertain data presents multimodality in numerous application scenarios, in the sense that the probability density function of the uncertain data has two or more modes (local maximums). In this paper, we propose a globalized distributionally robust optimization framework with multiple core sets (MGDRO) to handle the multimodal data. This framework captures the multimodal structure via a penalty function composed of the minimum distances from the random vector to all core sets. Under some assumptions, the MGDRO model can be reformulated as tractable semi-definite programs for both moment-based and metric-based ambiguity sets. We applied the MGDRO models to a multi-product newswendor problem with multimodal demands. The numerical results turn out that the MGDRO models outperform traditional DRO models and other multimodal models greatly.

math.OC↗

arXiv2023

Semidefinite Programming Approximation for a Matrix Optimization Problem over an Uncertain Linear System

A matrix optimization problem over an uncertain linear system on finite horizon (abbreviated as MOPUL) is studied, in which the uncertain transition matrix is regarded as a decision variable. This problem is in general NP-hard. By using the given reference values of system outputs at each stage, we develop a polynomial-time solvable semidefinite programming (SDP) approximation model for the problem. The upper bound of the cumulative error between reference outputs and the optimal outputs of the approximation model is theoretically analyzed. Two special cases associated with specific applications are considered. The quality of the SDP approximate solutions in terms of feasibility and optimality is also analyzed. Results of numerical experiments are presented to show the influences of perturbed noises at reference outputs and control levels on the performance of SDP approximation.

math.OC↗

arXiv2023

Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methods

Training deep neural networks (DNNs) is an important and challenging optimization problem in machine learning due to its non-convexity and non-separable structure. The alternating minimization (AM) approaches split the composition structure of DNNs and have drawn great interest in the deep learning and optimization communities. In this paper, we propose a unified framework for analyzing the convergence rate of AM-type network training methods. Our analysis is based on the non-monotone $j$-step sufficient decrease conditions and the Kurdyka-Lojasiewicz (KL) property, which relaxes the requirement of designing descent algorithms. We show the detailed local convergence rate if the KL exponent $θ$ varies in $[0,1)$. Moreover, the local R-linear convergence is discussed under a stronger $j$-step sufficient decrease condition.

cs.LG↗

arXiv2022

Globalized distributionally robust optimization based on samples

It is known that the set of perturbed data is key in robust optimization (RO) modelling. Distributionally robust optimization (DRO) is a methodology used for optimization problems affected by random parameters with uncertain probability distribution. In terms of the information of the perturbed data, it is essential to estimate an appropriate support set of the probability distribution in formulating DRO models. In this paper, we introduce two globalized distributionally robust optimization (GDRO) models which choose a core set based on data and a sample space containing the core set to balance the degree of robustness and conservatism at the same time. The degree of conservatism can be controlled by the expected distance of random parameters from the core set. Under some assumptions, we further reformulate several GDRO models into tractable semi-definite programs. In addition, numerical experiments are provided showing the relationship between the optimal objective values of the GDRO models and the size of the sample space and the core set.

math.OC↗

arXiv2014

Double Well Potential Function and Its Optimization in the n-dimensional Real Space - Part II

In contrast to taking the dual approach for finding a global minimum solution of a double well potential function, in Part II of the paper, we characterize a local minimizer, local maximizer, and global minimizer directly from the primal side. It is proven that, for a ``nonsingular" double well function, there exists at most one local, but non-global, minimizer and at most one local maximizer. Moreover, when it exists, the local maximizer is ``surrounded" by local minimizers in the sense that the norm of the local maximizer is strictly less than that of any local minimizer. We also establish some necessary and sufficient optimality conditions for the global minimizer, local non-global minimizer and local maximizer by studying a convex secular function over specific intervals. These conditions lead to three algorithms for identifying different types of critical points of a given double well function.

math.OC↗

arXiv2013

Combinations of Some Shop Scheduling Problems and the Shortest Path Problem: Complexity and Approximation Algorithms

We consider several combinatorial optimization problems which combine the classic shop scheduling problems, namely open shop scheduling or job shop scheduling, and the shortest path problem. The objective of the obtained problem is to select a subset of jobs that forms a feasible solution of the shortest path problem, and to execute the selected jobs on the open (or job) shop machines to minimize the makespan. We show that these problems are NP-hard even if the number of machines is two, and cannot be approximated within a factor less than 2 if the number of machines is an input unless P=NP. We present several approximation algorithms for these combination problems.

cs.DS↗