arXiv subjects

Kim-Chuan Toh

Publications and source records attributed to Kim-Chuan Toh.

At least 37 records · Page 2Linked to original sources

arXiv2026

Accelerating nuclear-norm regularized low-rank matrix optimization through Burer-Monteiro decomposition

This work proposes a rapid algorithm, BM-Global, for nuclear-norm-regularized convex and low-rank matrix optimization problems. BM-Global efficiently decreases the objective value via low-cost steps leveraging the nonconvex but smooth Burer-Monteiro (BM) decomposition, while effectively escapes saddle points and spurious local minima ubiquitous in the BM form to obtain guarantees of fast convergence rates to the global optima of the original nuclear-norm-regularized problem through aperiodic inexact proximal gradient steps on it. The proposed approach adaptively adjusts the rank for the BM decomposition and can provably identify an optimal rank for the BM decomposition problem automatically in the course of optimization through tools of manifold identification. BM-Global hence also spends significantly less time on parameter tuning than existing matrix-factorization methods, which require an exhaustive search for finding this optimal rank. Extensive experiments on real-world large-scale problems of recommendation systems, regularized kernel estimation, and molecular conformation confirm that BM-Global can indeed effectively escapes spurious local minima at which existing BM approaches are stuck, and is a magnitude faster than state-of-the-art algorithms for low-rank matrix optimization problems involving a nuclear-norm regularizer. Based on this research, we have released an open-source package of the proposed BM-Global at https://www.github.com/leepei/BM-Global/.

cs.LG↗

arXiv2025

Time-integrated Optimal Transport: A Robust Minimax Framework

Comparing time series in a principled manner requires capturing both temporal alignment and distributional similarity of features. Optimal transport (OT) has recently emerged as a powerful tool for this task, but existing OT-based approaches often depend on manually selected balancing parameters and can be computationally intensive. In this work, we introduce the Time-integrated Optimal Transport (TiOT) framework, which integrates temporal and feature components into a unified objective and yields a well-defined metric on the space of probability measures. This metric preserves fundamental properties of the Wasserstein distance, while avoiding the need for parameter tuning. To address the corresponding computational challenges, we introduce an entropic regularized approximation of TiOT, which can be efficiently solved using a block coordinate descent algorithm. Extensive experiments on both synthetic and real-world time series datasets demonstrate that our approach achieves improved accuracy and stability while maintaining comparable efficiency.

math.OC↗

arXiv2025

A Low-rank Augmented Lagrangian Method for Polyhedral-SDP and Moment-SOS Relaxations of Polynomial Optimization

Polynomial optimization problems (POPs) can be reformulated as geometric convex conic programs, as shown by Kim, Kojima, and Toh (SIOPT 30:1251-1273, 2020), though such formulations remain NP-hard. In this work, we prove that several well-known relaxations can be unified under a common polyhedral-SDP framework, which arises by approximating the intractable cone by tractable intersections of polyhedral cones with the positive semidefinite matrix cone. Although effective in providing tight lower bounds, these relaxations become computationally expensive as the number of variables and constraints grows at the rate of $Ω(n^{2τ})$ with the relaxation order $τ$. To address this challenge, we propose RiNNAL-POP, a low-rank augmented Lagrangian method (ALM) tailored to solve large-scale polyhedral-SDP relaxations of POPs. To efficiently handle the $Ω(n^{2τ})$ nonnegativity and consistency constraints, we design a tailored projection scheme whose computational cost scales linearly with the number of variables. In addition, we identify a hidden facial structure in the polyhedral-SDP relaxation, which enables us to eliminate a large number of linear constraints by restricting the matrix variable to affine subspaces corresponding to exposed faces of the semidefinite cone. The latter enables us to efficiently solve the factorized ALM subproblems over the affine subspaces. At each ALM iteration, we additionally carry out a single projected gradient step with respect to the original matrix variable to automatically adjust the rank and escape from spurious local minima when necessary. We also extend our RiNNAL-POP algorithmic framework to solve moment-SOS relaxations of POPs. Extensive numerical experiments on various benchmark problems demonstrate the robustness and efficiency of RiNNAL-POP in solving large-scale polyhedral-SDP relaxations.

math.OC↗

arXiv2025

A Quadratically Convergent Alternating Projection Method for Nonconvex Sets

In this paper, we consider the feasibility problem, which aims to find a feasible point for the constraint set $\{x \in \mathbb{R}^n: c(x) = 0\}$ over a possibly non-regular subset $\mathcal{X} \subset \mathbb{R}^n$. Under the constraint nondegeneracy condition, we propose a modified alternating projection method. In our proposed method, based on the concept of projective mapping for $\mathcal{X}$, we alternate a Newton step for finding an inexact solution within the limiting tangent cone of $\mathcal{X}$ and a projection to $\mathcal{X}$. Under mild conditions, we prove the local quadratic convergence of our proposed method. Preliminary numerical experiments demonstrate the high efficiency of our proposed alternating projection method.

math.OC↗

arXiv2025

A Bregman ADMM for Bethe variational problem

In this work, we propose a novel Bregman ADMM with nonlinear dual update to solve the Bethe variational problem (BVP), a key optimization formulation in graphical models and statistical physics. Our algorithm provides rigorous convergence guarantees, even if the objective function of BVP is non-convex and non-Lipschitz continuous on the boundary. A central result of our analysis is proving that the entries in local minima of BVP are strictly positive, effectively resolving non-smoothness issues caused by zero entries. Beyond theoretical guarantees, the algorithm possesses high level of separability and parallelizability to achieve highly efficient subproblem computation. Our Bregman ADMM can be easily extended to solve the quantum Bethe variational problem. Numerical experiments are conducted to validate the effectiveness and robustness of the proposed method. Based on this research, we have released an open-source package of the proposed method at https://github.com/TTYmath/BADMM-BVP.

math.OC↗

arXiv2025

Partial Envelope for Optimization Problem with Nonconvex Constraints

In this paper, we consider the nonlinear constrained optimization problem (NCP) with constraint set $\{x \in \mathcal{X}: c(x) = 0\}$, where $\mathcal{X}$ is a closed convex subset of $\mathbb{R}^n$. Building upon the forward-backward envelope framework for optimization over $\mathcal{X}$, we propose a forward-backward semi-envelope (FBSE) approach for solving (NCP). In the proposed semi-envelope approach, we eliminate the constraint $x \in \mathcal{X}$ through a specifically designed envelope scheme while preserving the constraint $x \in \mathcal{M} := \{x \in \mathbb{R}^n: c(x) = 0\}$. We establish that the forward-backward semi-envelope for (NCP) is well-defined and locally Lipschitz smooth over a neighborhood of $\mathcal{M}$. Furthermore, we prove that (NCP) and its corresponding forward-backward semi-envelope have the same first-order stationary points within a neighborhood of $\mathcal{X} \cap \mathcal{M}$. Consequently, our proposed forward-backward semi-envelope approach enables direct application of optimization methods over $\mathcal{M}$ while inheriting their convergence properties for (NCP). Additionally, we develop an inexact projected gradient descent method for minimizing the forward-backward semi-envelope over $\mathcal{M}$ and establish its global convergence. Preliminary numerical experiments demonstrate the practical efficiency and potential of our proposed approach.

math.OC↗

arXiv2025

GRIFFIN: Effective Token Alignment for Faster Speculative Decoding

Speculative decoding accelerates inference in large language models (LLMs) by generating multiple draft tokens simultaneously. However, existing methods often struggle with token misalignment between the training and decoding phases, limiting their performance. To address this, we propose GRIFFIN, a novel framework that incorporates a token-alignable training strategy and a token-alignable draft model to mitigate misalignment. The training strategy employs a loss masking mechanism to exclude highly misaligned tokens during training, preventing them from negatively impacting the draft model's optimization. The token-alignable draft model introduces input tokens to correct inconsistencies in generated features. Experiments on LLaMA, Vicuna, Qwen and Mixtral models demonstrate that GRIFFIN achieves an average acceptance length improvement of over 8% and a speedup ratio exceeding 7%, outperforming current speculative decoding state-of-the-art methods. Our code and GRIFFIN's draft models are released publicly in https://github.com/hsj576/GRIFFIN.

cs.CL↗

arXiv2025

Convergence rates of S.O.S hierarchies for polynomial semidefinite programs

We introduce an S.o.S hierarchy of lower bounds for a polynomial optimization problem whose constraint is expressed as a matrix polynomial semidefinite inequality. Our approach involves utilizing a penalty function framework to directly address the matrix-based constraint, making it applicable to both discrete and continuous polynomial optimization problems. We investigate the convergence rates of these bounds in both types of problems. The proposed method yields a variant of Putinar's theorem, tailored for positive polynomials within a compact semidefinite set $\mathcal{X}$ defined by a matrix polynomial semidefinite constraint. More specifically, we derive novel insights into the convergence rates and bounds on the degree of the S.o.S polynomials required to certify positivity on $\mathcal{X}$, based on Jackson's theorem and a variant of the Łojasiewicz inequality.

math.OC↗

arXiv2025

Nesterov's Accelerated Jacobi-Type Methods for Large-scale Symmetric Positive Semidefinite Linear Systems

Solving symmetric positive semidefinite linear systems is an essential task in many scientific computing problems. While Jacobi-type methods, including the classical Jacobi method and the weighted Jacobi method, exhibit simplicity in their forms and friendliness to parallelization, they are not attractive either because of the potential convergence failure or their slow convergence rate. This paper aims to showcase the possibility of improving classical Jacobi-type methods by employing Nesterov's acceleration technique that results in an accelerated Jacobi-type method with improved convergence properties. Simultaneously, it preserves the appealing features for parallel implementation. In particular, we show that the proposed method has an $O\left(t^{-2}\right)$ convergence rate in terms of objective function values of the associated convex quadratic optimization problem, where $t\geq 1$ denotes the iteration counter. To further improve the practical performance of the proposed method, we also develop and analyze a restarted variant of the method, which is shown to have an $O\left((\log_2(t))^2t^{-2}\right)$ convergence rate when the coefficient matrix is positive definite. Furthermore, we conduct appropriate numerical experiments to evaluate the efficiency of the proposed method. Our numerical results demonstrate that the proposed method outperforms the classical Jacobi-type methods and the conjugate gradient method, and shows a comparable performance as the preconditioned conjugate gradient method with a diagonal preconditioner. Finally, we develop a parallel implementation and conduct speed-up tests on some large-scale systems. Our results indicate that the proposed framework is highly scalable.

math.OC↗

arXiv2025

NewVEM: A Newton Vertex Exchange Method for a Class of Constrained Self-Concordant Minimization Problems

We propose \textbf{NewVEM}, a Newton vertex exchange method for efficiently solving self-concordant minimization problems under generalized simplex constraints. The algorithm features a two-level structure: the outer loop employs a projected Newton method, and the inner loop uses a vertex exchange approach to solve strongly convex quadratic subproblems. Both loops converge locally at linear rates under technical conditions, resulting in a ``fast $\times$ fast'' framework that demonstrates high efficiency and scalability in practice. To get a feasible initial point to execute the algorithm, we also present and analyze a highly efficient semismooth Newton method for computing the projection onto the generalized simplex. The excellent practical performance of the proposed algorithms is demonstrated by a set of numerical experiments. Our results further motivate the potential real-world applications of the considered model and the proposed algorithms.

math.OC↗

arXiv2025

On the B-subdifferential of proximal operators of affine-constrained $\ell_1$ regularizer

In this work, we study the affine-constrained $\ell_1$ regularizers, which frequently arise in statistical and machine learning problems across a variety of applications, including microbiome compositional data analysis and sparse subspace clustering. With the aim of developing scalable second-order methods for solving optimization problems involving such regularizers, we analyze the associated proximal mapping and characterize its generalized differentiability, with a focus on its B-subdifferential. The revealed structured sparsity in the B-subdifferential enables us to design efficient algorithms within the proximal point framework. Extensive numerical experiments on real applications, including comparisons with state-of-the-art solvers, further demonstrate the superior performance of our approach. Our findings provide new insights into the sensitivity and stability properties of affine-constrained nonsmooth regularizers, and contribute to the development of fast second-order methods for a class of structured, constrained sparse learning problems.

math.OC↗

arXiv2025

Improved Rates for Stochastic Variance-Reduced Difference-of-Convex Algorithms

In this work, we propose and analyze DCA-PAGE, a novel algorithm that integrates the difference-of-convex algorithm (DCA) with the ProbAbilistic Gradient Estimator (PAGE) to solve structured nonsmooth difference-of-convex programs. In the finite-sum setting, our method achieves a gradient computation complexity of $O(N + N^{1/2}\varepsilon^{-2})$ with sample size $N$, surpassing the previous best-known complexity of $O(N + N^{2/3}\varepsilon^{-2})$ for stochastic variance-reduced (SVR) DCA methods. Furthermore, DCA-PAGE readily extends to online settings with a similar optimal gradient computation complexity $O(b + b^{1/2}\varepsilon^{-2})$ with batch size $b$, a significant advantage over existing SVR DCA approaches that only work for the finite-sum setting. We further refine our analysis with a gap function, which enables us to obtain comparable convergence guarantees under milder assumptions.

math.OC↗

arXiv2025

On exploration of an interior mirror descent flow for stochastic nonconvex constrained problem

We study a nonsmooth nonconvex optimization problem defined over nonconvex constraints, where the feasible set is given by the intersection of the closure of an open set and a smooth manifold. By endowing the open set with a Riemannian metric induced by a barrier function, we obtain a Riemannian subgradient flow formulated as a differential inclusion, which remains strictly within the interior of the feasible set. This continuous dynamical system unifies two classes of iterative optimization methods, namely the Hessian barrier method and mirror descent scheme, by revealing that these methods can be interpreted as discrete approximations of the continuous flow. We explore the long-term behavior of the trajectories generated by this dynamical system and show that the existing deficient convergence properties of the Hessian barrier and mirror descent scheme can be unifily and more insightfully interpreted through these of the continuous trajectory. For instance, the notorious spurious stationary points \cite{chen2024spurious} observed in Hessian barrier method and mirror descent scheme are interpreted as stable equilibria of the dynamical system that do not correspond to real stationary points of the original optimization problem. We provide two sufficient condition such that these spurious stationary points can be avoided if the strict complementarity conditions holds. In the absence of these regularity condition, we propose a random perturbation strategy that ensures the trajectory converges (subsequentially) to an approximate stationary point. Building on these insights, we introduce two iterative Riemannian subgradient methods, form of interior point methods, that generalizes the existing Hessian barrier method and mirror descent scheme for solving nonsmooth nonconvex optimization problems.

math.OC↗

arXiv2025

RiNNAL+: a Riemannian ALM Solver for SDP-RLT Relaxations of Mixed-Binary Quadratic Programs

Doubly nonnegative (DNN) relaxation usually provides a tight lower bound for a mixed-binary quadratic program (MBQP). However, solving DNN problems is challenging because: (1) the problem size is $Ω((n+l)^2)$ for an MBQP with $n$ variables and $l$ inequality constraints, and (2) the rank of optimal solutions cannot be estimated a priori due to the absence of theoretical bounds. In this work, we propose RiNNAL+, a Riemannian augmented Lagrangian method (ALM) for solving DNN problems. We prove that the DNN relaxation of an MBQP, with matrix dimension $(n+l+1)$, is equivalent to the SDP-RLT relaxation (based on the reformulation-linearization technique) with a smaller matrix dimension $(n+1)$. In addition, we develop a hybrid method that alternates between two phases to solve the ALM subproblems. In phase one, we apply low-rank matrix factorization and random perturbation to transform the feasible region into a lower-dimensional manifold so that we can use the Riemannian gradient descent method. In phase two, we apply a single projected gradient step to update the rank of the underlying variable and escape from spurious local minima arising in the first phase if necessary. To reduce the computation cost of the projected gradient step, we develop pre-processing and warm-start techniques for acceleration. Unlike traditional rank-adaptive methods that require extensive parameter tuning, our hybrid method requires minimal tuning. Extensive experiments confirm the efficiency and robustness of RiNNAL+ in solving various classes of large-scale DNN problems.

math.OC↗

arXiv2025

Tractable hierarchies of convex relaxations for polynomial optimization on the nonnegative orthant

We consider polynomial optimization problems (POP) on a semialgebraic set contained in the nonnegative orthant (every POP on a compact set can be put in this format by a simple translation of the origin). Such a POP can be converted to an equivalent POP by squaring each variable. Using even symmetry and the concept of factor width, we propose a hierarchy of semidefinite relaxations based on the extension of Pólya's Positivstellensatz by Dickinson-Povh. As its distinguishing and crucial feature, the maximal matrix size of each resulting semidefinite relaxation can be chosen arbitrarily and in addition, we prove that the sequence of values returned by the new hierarchy converges to the optimal value of the original POP at the rate $O(\varepsilon^{-c})$ if the semialgebraic set has nonempty interior. When applied to (i) robustness certification of multi-layer neural networks and (ii) computation of positive maximal singular values, our method based on Pólya's Positivstellensatz provides better bounds and runs several hundred times faster than the standard Moment-SOS hierarchy.

math.OC↗

arXiv2025

Stochastic Bregman Subgradient Methods for Nonsmooth Nonconvex Optimization Problems

This paper focuses on the problem of minimizing a locally Lipschitz continuous function. Motivated by the effectiveness of Bregman gradient methods in training nonsmooth deep neural networks and the recent progress in stochastic subgradient methods for nonsmooth nonconvex optimization problems \cite{bolte2021conservative,bolte2022subgradient,xiao2023adam}, we investigate the long-term behavior of stochastic Bregman subgradient methods in such context, especially when the objective function lacks Clarke regularity. We begin by exploring a general framework for Bregman-type methods, establishing their convergence by a differential inclusion approach. For practical applications, we develop a stochastic Bregman subgradient method that allows the subproblems to be solved inexactly. Furthermore, we demonstrate how a single timescale momentum can be integrated into the Bregman subgradient method with slight modifications to the momentum update. Additionally, we introduce a Bregman proximal subgradient method for solving composite optimization problems possibly with constraints, whose convergence can be guaranteed based on the general framework. Numerical experiments on training nonsmooth neural networks are conducted to validate the effectiveness of our proposed methods.

math.OC↗

arXiv2025

An Inexact Halpern Iteration with Application to Distributionally Robust Optimization

The Halpern iteration for solving monotone inclusion problems has gained increasing interests in recent years due to its simple form and appealing convergence properties. In this paper, we investigate the inexact variants of the scheme in both deterministic and stochastic settings. We conduct extensive convergence analysis and show that by choosing the inexactness tolerances appropriately, the inexact schemes admit an $O(k^{-1})$ convergence rate in terms of the (expected) residue norm. Our results relax the state-of-the-art inexactness conditions employed in the literature while sharing the same competitive convergence properties. We then demonstrate how the proposed methods can be applied for solving two classes of data-driven Wasserstein distributionally robust optimization problems that admit convex-concave min-max optimization reformulations. We highlight its capability of performing inexact computations for distributionally robust learning with stochastic first-order methods and for general nonlinear convex-concave loss functions, which are competitive in the literature.

math.OC↗

arXiv2025

An Exact Penalty Approach for Equality Constrained Optimization over a Convex Set

In this paper, we consider the nonlinear constrained optimization problem (NCP) with constraint set $\{x \in \mathcal{X}: c(x) = 0\}$, where $\mathcal{X}$ is a closed convex subset of $\mathbb{R}^n$. We propose an exact penalty approach, named constraint dissolving approach, that transforms (NCP) into its corresponding constraint dissolving problem (CDP). The transformed problem (CDP) admits $\mathcal{X}$ as its feasible region with a locally Lipschitz smooth objective function. We prove that (NCP) and (CDP) share the same first-order stationary points, second-order stationary points, second-order sufficient condition (SOSC) points, and strong SOSC points, in a neighborhood of the feasible region. Moreover, we prove that these equivalences extend globally under a particular error bound condition. Therefore, our proposed constraint dissolving approach enables direct implementations of optimization approaches over $\mathcal{X}$ and inherits their convergence properties to solve problems that take the form of (NCP). Preliminary numerical experiments illustrate the high efficiency of directly applying existing solvers for optimization over $\mathcal{X}$ to solve (NCP) through (CDP). These numerical results further demonstrate the practical potential of our proposed constraint dissolving approach.

math.OC↗