arXiv subjects

Nachuan Xiao

Publications and source records attributed to Nachuan Xiao.

At least 19 recordsLinked to original sources

arXiv2026

A Hybrid Subgradient Method for Nonsmooth Nonconvex Bilevel Optimization

In this paper, we focus on the nonconvex-nonconvex bilevel optimization problem (BLO), where both upper-level and lower-level objectives are nonconvex, with the upper-level problem potentially being nonsmooth. We develop a two-timescale momentum-accelerated subgradient method (TMG) that employs two-timescale stepsizes, and establish its local convergence when initialized within a sufficiently small neighborhood of the feasible region. To develop a globally convergent algorithm for (BLO), we introduce a feasibility restoration scheme (FRG) that drives iterates toward the feasible region. Both (TMG) and (FRG) only require the first-order derivatives of the upper-level and lower-level objective functions, ensuring efficient computations in practice. We then develop a novel hybrid method that alternates between (TMG) and (FRG) and adaptively estimates its hyperparameters. Under mild conditions, we establish the global convergence properties of our proposed algorithm. Preliminary numerical experiments demonstrate the high efficiency and promising potential of our proposed algorithm.

math.OC↗

arXiv2026

Exact Quadratic Penalty Function for Symplectic Eigenvalue Problem

The symplectic eigenvalue problem for symmetric positive-definite (spd) matrices plays a crucial role in various scientific fields, including quantum mechanics and control theory. This paper introduces a trace-penalty minimization method, which transforms the symplectic eigenvalue problem into the unconstrained minimization of the trace-penalty function. We prove the equivalence between the penalty problem and the original constrained optimization problem under mild conditions, in the sense that the second-order stationary points of the trace-penalty function correspond to the solutions of the symplectic eigenvalue problem. Moreover, we develop an algorithm to minimize the trace-penalty function efficiently, which follows the scheme of gradient methods, together with the Barzilai-Borwein (BB) adaptive step-size rule and non-monotone line-search technique. Numerical experiments demonstrate that the proposed algorithm outperforms a wide range of existing methods, such as Riemannian gradient-based methods, in terms of computational efficiency and convergence rate for dense, sparse, and sparse-add-low-rank matrices. These numerical results further demonstrate the great potential of our proposed algorithm, especially in solving large-scale symplectic eigenvalue problems.

math.OC↗

arXiv2026

Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization

In this paper, we focus on providing convergence guarantees for stochastic subgradient methods in minimizing nonsmooth nonconvex functions. We first investigate the global stability of a general framework for stochastic subgradient methods, where the corresponding differential inclusion admits a coercive Lyapunov function. We prove that, for any sequence of sufficiently small stepsizes and approximation parameters, coupled with sufficiently controlled noises, the iterates are uniformly bounded and asymptotically stabilize around the stable set of its corresponding differential inclusion. Moreover, we develop an improved analysis to apply our proposed framework to establish the global stability of a wide range of stochastic subgradient methods, where the corresponding Lyapunov functions are possibly non-coercive. These theoretical results illustrate the promising potential of our proposed framework for establishing the global stability of various stochastic subgradient methods.

math.OC↗

arXiv2026

Decentralized Stochastic Subgradient-type Methods with Communication Compression for Nonsmooth Nonconvex Optimization

In this paper, we consider the nonsmooth nonconvex decentralized optimization problem, where inter-agent communication is compressed. We propose a general framework that unifies various decentralized stochastic subgradient-type methods with unbiased compression and contractive compression with error compensation. By relating the consensus-error iterates and the averaged iterates to the trajectories of continuous-time differential inclusions, we establish global convergence for all methods encompassed by our framework when the objective functions are nonsmooth and lack Clarke regularity. Based on our framework, we further develop several compression-based methods, including decentralized stochastic subgradient methods utilizing sign-based regularization and gradient-tracking momentum. Preliminary numerical experiments empirically support our theoretical results and highlight the communication-accuracy trade-off of the newly developed methods.

math.OC↗

arXiv2026

A Single-Loop Regularized Newton Method for Nonconvex-Strongly-Concave Minimax Optimization

For smooth nonconvex-strongly-concave minimax problems, existing second-order methods share a common double-loop structure where the inner maximization is solved to sufficiently high accuracy before each second-order step. First-order methods are often adopted in the inner loops for scalability, but they also undermine the condition-insensitivity of second-order methods, limiting these methods to instances with mild conditioning. To resolve this issue, we propose a novel single-loop framework based on an equivalent regularized minimization reformulation of the original problem. By deriving a new adaptive cubic-quadratic majorization to dynamically absorb the non-Lipschitz components of the reformulated Hessian, we establish a regularized Newton method with robust theoretical guarantees across multiple settings. For deterministic problems, our single-loop method matches the $\mathcal{O}(\varepsilon^{-1.5})$ global iteration complexity of the existing double-loop second-order methods, while automatically achieving a local superlinear rate that is unavailable in existing works due to the inner-loop bottleneck. For the stochastic setting, we achieve $\mathcal{O}(\varepsilon^{-3})$ gradient and $\mathcal{O}(\varepsilon^{-2})$ Hessian complexities by integrating a recursive variance reduction, strictly improving those of the double-loop methods by $\mathcal{O}(\varepsilon^{-0.5})$ factors. In both deterministic and stochastic experiments, our methods significantly outperform the benchmarks, offering substantial speedups over the double-loop methods even under mildly conditioned instances. As a byproduct of our analysis, we close a gap in stochastic second-order methods for nonconvex minimization, where the best known result contains a nontrivial technical issue.

math.OC↗

arXiv2026

Improved Penalty Function Approaches for Optimization Problems with General Orthogonality

In this paper, we consider a class of generalized orthogonal optimization constraint problems (GOOCP) over $\mathbb{R}^{n \times p}$, where the variable $X$ is restricted within the intersection of a certain subspace $\mathcal{F}$ and satisfies the quadratic constraint $\{X \in \mathbb{R}^{n \times p}: X^{\top} ϕ(X) = I_p\}$. Such constraints generalize a wide range of structured matrix manifolds, such as the Stiefel manifold, the symplectic Stiefel manifold, the indefinite Stiefel manifold, the third-order tensor Stiefel manifold, etc. We show that the feasible region of GOOCP is a closed embedded submanifold of $\mathbb{R}^{n \times p}$ and characterize the necessary geometric materials for the existing Riemannian optimization frameworks. Based on the constraint dissolving approach for Riemannian optimization problems, we propose the constraint dissolving penalty function (GOCDF) for the constrained optimization problem GOOCP with easy-to-compute formulations. We further establish the equivalence between GOCDF and GOOCP in the aspects of first-order and second-order stationary points. We also analyze the computational complexity of applying first-order methods to minimize GOOCP, which could be significantly lower than those of first-order Riemannian optimization methods. Numerical experiments demonstrate that solving GOOCP through applying unconstrained optimization methods to minimize constraint dissolving function demonstrates superior efficiency to existing Riemannian optimization methods.

math.OC↗

arXiv2026

A Smooth Locally Exact Penalty Method for Optimization Problems over Generalized Stiefel Manifolds

In this paper, we consider a class of optimization problems constrained to the generalized Stiefel manifold. Such problems are fundamental to a wide range of real-world applications, including generalized canonical correlation analysis, linear discriminant analysis, and electronic structure calculations. Existing works mainly focuses on cases where the generalized orthogonality constraint is induced by a symmetric positive definite matrix M, a setting where the geometry essentially reduces to that of the standard Stiefel manifold. However, many practical scenarios involve a singular M, which introduces significant analytical and computational challenges. Therefore, we propose a Smooth Locally Exact Penalty model (SLEP) and establish its equivalence to the original problem in the aspect of stationary points under a finitly large penalty parameter. This penalty model admits the direct application of various unconstrained optimization techniques, with convergence guarantees inherited from established results. Compared to Riemannian optimization approaches, our proposed penalty mode eliminates the need for retractions and vector transports, hence significantly reducing per-iteration computational costs. Extensive numerical experiments validate our theoretical results and demonstrate the effectiveness and practical potential of the proposed penalty model SLEP.

math.OC↗

arXiv2026

Line-search and Adaptive Step Sizes for Nonconvex-strongly-concave Minimax Optimization

In this paper, we propose a novel reformulation of the smooth nonconvex-strongly-concave (NC-SC) minimax problems that casts the problem as a joint minimization. We show that our reformulation preserves not only first-order stationarity, but also global and local optimality, second-order stationarity, and the Kurdyka-Łojasiewicz (KL) property, of the original NC-SC problem, which is substantially stronger than its nonsmooth counterpart in the literature. With these enhanced structures, we design a versatile parameter-free and nonmonotone line-search framework that does not require evaluating the inner maximization. Under mild conditions, global convergence rates can be obtained, and, with KL property, full sequence convergence with asymptotic rates is also established. In particular, we show our framework is compatible with the gradient descent-ascent (GDA) algorithm. By equipping GDA with Barzilai-Borwein (BB) step sizes and nonmonotone line-search, our method exhibits superior numerical performance against the compared benchmarks.

math.OC↗

arXiv2026

Convergence of Decentralized Stochastic Subgradient-based Methods for Nonsmooth Nonconvex functions

In this paper, we focus on the decentralized stochastic subgradient-based methods in minimizing nonsmooth nonconvex functions without Clarke regularity, especially in the decentralized training of nonsmooth neural networks. We propose a general framework that unifies various decentralized subgradient-based methods, such as decentralized stochastic subgradient descent (DSGD), DSGD with gradient-tracking technique (DSGD-T), and DSGD with momentum (DSGD-M). To establish the convergence properties of our proposed framework, we relate the discrete iterates to the trajectories of a continuous-time differential inclusion, which is assumed to have a coercive Lyapunov function with a stable set $\mathcal{A}$. We prove the asymptotic convergence of the iterates to the stable set $\mathcal{A}$ with sufficiently small and diminishing step-sizes. These results provide first convergence guarantees for some well-recognized of decentralized stochastic subgradient-based methods without Clarke regularity of the objective function. Preliminary numerical experiments demonstrate that our proposed framework yields highly efficient decentralized stochastic subgradient-based methods with convergence guarantees in the training of nonsmooth neural networks.

math.OC↗

arXiv2025

A Quadratically Convergent Alternating Projection Method for Nonconvex Sets

In this paper, we consider the feasibility problem, which aims to find a feasible point for the constraint set $\{x \in \mathbb{R}^n: c(x) = 0\}$ over a possibly non-regular subset $\mathcal{X} \subset \mathbb{R}^n$. Under the constraint nondegeneracy condition, we propose a modified alternating projection method. In our proposed method, based on the concept of projective mapping for $\mathcal{X}$, we alternate a Newton step for finding an inexact solution within the limiting tangent cone of $\mathcal{X}$ and a projection to $\mathcal{X}$. Under mild conditions, we prove the local quadratic convergence of our proposed method. Preliminary numerical experiments demonstrate the high efficiency of our proposed alternating projection method.

math.OC↗

arXiv2025

Partial Envelope for Optimization Problem with Nonconvex Constraints

In this paper, we consider the nonlinear constrained optimization problem (NCP) with constraint set $\{x \in \mathcal{X}: c(x) = 0\}$, where $\mathcal{X}$ is a closed convex subset of $\mathbb{R}^n$. Building upon the forward-backward envelope framework for optimization over $\mathcal{X}$, we propose a forward-backward semi-envelope (FBSE) approach for solving (NCP). In the proposed semi-envelope approach, we eliminate the constraint $x \in \mathcal{X}$ through a specifically designed envelope scheme while preserving the constraint $x \in \mathcal{M} := \{x \in \mathbb{R}^n: c(x) = 0\}$. We establish that the forward-backward semi-envelope for (NCP) is well-defined and locally Lipschitz smooth over a neighborhood of $\mathcal{M}$. Furthermore, we prove that (NCP) and its corresponding forward-backward semi-envelope have the same first-order stationary points within a neighborhood of $\mathcal{X} \cap \mathcal{M}$. Consequently, our proposed forward-backward semi-envelope approach enables direct application of optimization methods over $\mathcal{M}$ while inheriting their convergence properties for (NCP). Additionally, we develop an inexact projected gradient descent method for minimizing the forward-backward semi-envelope over $\mathcal{M}$ and establish its global convergence. Preliminary numerical experiments demonstrate the practical efficiency and potential of our proposed approach.

math.OC↗

arXiv2025

An Exact Penalty Approach for Equality Constrained Optimization over a Convex Set

In this paper, we consider the nonlinear constrained optimization problem (NCP) with constraint set $\{x \in \mathcal{X}: c(x) = 0\}$, where $\mathcal{X}$ is a closed convex subset of $\mathbb{R}^n$. We propose an exact penalty approach, named constraint dissolving approach, that transforms (NCP) into its corresponding constraint dissolving problem (CDP). The transformed problem (CDP) admits $\mathcal{X}$ as its feasible region with a locally Lipschitz smooth objective function. We prove that (NCP) and (CDP) share the same first-order stationary points, second-order stationary points, second-order sufficient condition (SOSC) points, and strong SOSC points, in a neighborhood of the feasible region. Moreover, we prove that these equivalences extend globally under a particular error bound condition. Therefore, our proposed constraint dissolving approach enables direct implementations of optimization approaches over $\mathcal{X}$ and inherits their convergence properties to solve problems that take the form of (NCP). Preliminary numerical experiments illustrate the high efficiency of directly applying existing solvers for optimization over $\mathcal{X}$ to solve (NCP) through (CDP). These numerical results further demonstrate the practical potential of our proposed constraint dissolving approach.

math.OC↗

arXiv2024

Stochastic optimization over expectation-formulated generalized Stiefel manifold

In this paper, we consider a class of stochastic optimization problems over the expectation-formulated generalized Stiefel manifold (SOEGS), where the objective function $f$ is continuously differentiable. We propose a novel constraint dissolving penalty function with a customized penalty term (CDFDP), which maintains the same order of differentiability as $f$. Our theoretical analysis establishes the global equivalence between CDFCP and SOEGS in the sense that they share the same first-order and second-order stationary points under mild conditions. These results on equivalence enable the direct implementation of various stochastic optimization approaches to solve SOEGS. In particular, we develop a stochastic gradient algorithm and its accelerated variant by incorporating an adaptive step size strategy. Furthermore, we prove their $\mathcal{O}(\varepsilon^{-4})$ sample complexity for finding an $\varepsilon$-stationary point of CDFCP. Comprehensive numerical experiments show the efficiency and robustness of our proposed algorithms.

math.OC↗

arXiv2024

CDOpt: A Python Package for a Class of Riemannian Optimization

Optimization over the embedded submanifold defined by constraints $c(x) = 0$ has attracted much interest over the past few decades due to its wide applications in various areas. Plenty of related optimization packages have been developed based on Riemannian optimization approaches, which rely on some basic geometrical materials of Riemannian manifolds, including retractions, vector transports, etc. These geometrical materials can be challenging to determine in general. Existing packages only accommodate a few well-known manifolds whose geometrical materials are easily accessible. For other manifolds which are not contained in these packages, the users have to develop the geometric materials by themselves. In addition, it is not always tractable to adopt advanced features from various state-of-the-art unconstrained optimization solvers to Riemannian optimization approaches. We introduce CDOpt (available at https://cdopt.github.io/), a user-friendly Python package for a class Riemannian optimization. Based on constraint dissolving approaches, Riemannian optimization problems are transformed into their equivalent unconstrained counterparts in CDOpt. Therefore, solving Riemannian optimization problems through CDOpt directly benefits from various existing solvers and the rich expertise gained over decades for unconstrained optimization. Moreover, all the computations in CDOpt related to any manifold in question are conducted on its constraints expression, hence users can easily define new manifolds in CDOpt without any background on differential geometry. Furthermore, CDOpt extends the neural layers from PyTorch and Flax, thus allows users to train manifold constrained neural networks directly by the solvers for unconstrained optimization. Extensive numerical experiments demonstrate that CDOpt is highly efficient and robust in solving various classes of Riemannian optimization problems.

math.OC↗

arXiv2024

A Double Tracking Method for Optimization with Decentralized Generalized Orthogonality Constraints

In this paper, we consider the decentralized optimization problems with generalized orthogonality constraints, where both the objective function and the constraint exhibit a distributed structure. Such optimization problems, albeit ubiquitous in practical applications, remain unsolvable by existing algorithms in the presence of distributed constraints. To address this issue, we convert the original problem into an unconstrained penalty model by resorting to the recently proposed constraint-dissolving operator. However, this transformation compromises the essential property of separability in the resulting penalty function, rendering it impossible to employ existing algorithms to solve. We overcome this difficulty by introducing a novel algorithm that tracks the gradient of the objective function and the Jacobian of the constraint mapping simultaneously. The global convergence guarantee is rigorously established with an iteration complexity. To substantiate the effectiveness and efficiency of our proposed algorithm, we present numerical results on both synthetic and real-world datasets.

math.OC↗

arXiv2024

A Minimization Approach for Minimax Optimization with Coupled Constraints

In this paper, we focus on the nonconvex-strongly-concave minimax optimization problem (MCC), where the inner maximization subproblem contains constraints that couple the primal variable of the outer minimization problem. We prove that by introducing the dual variable of the inner maximization subproblem, (MCC) has the same first-order minimax points as a nonconvex-strongly-concave minimax optimization problem without coupled constraints (MOL). We then extend our focus to a class of nonconvex-strongly-concave minimax optimization problems (MM) that generalize (MOL). By performing the partial forward-backward envelope to the primal variable of the inner maximization subproblem, we propose a minimization problem (MMPen), where its objective function is explicitly formulated. We prove that the first-order stationary points of (MMPen) coincide with the first-order minimax points of (MM). Therefore, various efficient minimization methods and their convergence guarantees can be directly employed to solve (MM), hence solving (MCC) through (MOL). Preliminary numerical experiments demonstrate the great potential of our proposed approach.

math.OC↗

arXiv2024

Learning-rate-free Momentum SGD with Reshuffling Converges in Nonsmooth Nonconvex Optimization

In this paper, we propose a generalized framework for developing learning-rate-free momentum stochastic gradient descent (SGD) methods in the minimization of nonsmooth nonconvex functions, especially in training nonsmooth neural networks. Our framework adaptively generates learning rates based on the historical data of stochastic subgradients and iterates. Under mild conditions, we prove that our proposed framework enjoys global convergence to the stationary points of the objective function in the sense of the conservative field, hence providing convergence guarantees for training nonsmooth neural networks. Based on our proposed framework, we propose a novel learning-rate-free momentum SGD method (LFM). Preliminary numerical experiments reveal that LFM performs comparably to the state-of-the-art learning-rate-free methods (which have not been shown theoretically to be convergence) across well-known neural network training benchmarks.

math.OC↗

arXiv2024

Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization

In this paper, we consider the minimization of a nonsmooth nonconvex objective function $f(x)$ over a closed convex subset $\mathcal{X}$ of $\mathbb{R}^n$, with additional nonsmooth nonconvex constraints $c(x) = 0$. We develop a unified framework for developing Lagrangian-based methods, which takes a single-step update to the primal variables by some subgradient methods in each iteration. These subgradient methods are ``embedded'' into our framework, in the sense that they are incorporated as black-box updates to the primal variables. We prove that our proposed framework inherits the global convergence guarantees from these embedded subgradient methods under mild conditions. In addition, we show that our framework can be extended to solve constrained optimization problems with expectation constraints. Based on the proposed framework, we show that a wide range of existing stochastic subgradient methods, including the proximal SGD, proximal momentum SGD, and proximal ADAM, can be embedded into Lagrangian-based methods. Preliminary numerical experiments on deep learning tasks illustrate that our proposed framework yields efficient variants of Lagrangian-based methods with convergence guarantees for nonconvex nonsmooth constrained optimization problems.

math.OC↗