arXiv subjects

Kim-Chuan Toh

Publications and source records attributed to Kim-Chuan Toh.

At least 19 recordsLinked to original sources

arXiv2026

A Hybrid Subgradient Method for Nonsmooth Nonconvex Bilevel Optimization

In this paper, we focus on the nonconvex-nonconvex bilevel optimization problem (BLO), where both upper-level and lower-level objectives are nonconvex, with the upper-level problem potentially being nonsmooth. We develop a two-timescale momentum-accelerated subgradient method (TMG) that employs two-timescale stepsizes, and establish its local convergence when initialized within a sufficiently small neighborhood of the feasible region. To develop a globally convergent algorithm for (BLO), we introduce a feasibility restoration scheme (FRG) that drives iterates toward the feasible region. Both (TMG) and (FRG) only require the first-order derivatives of the upper-level and lower-level objective functions, ensuring efficient computations in practice. We then develop a novel hybrid method that alternates between (TMG) and (FRG) and adaptively estimates its hyperparameters. Under mild conditions, we establish the global convergence properties of our proposed algorithm. Preliminary numerical experiments demonstrate the high efficiency and promising potential of our proposed algorithm.

math.OC↗

arXiv2026

Establishing Boundary KKT Convergence of Mirror Descent through Reparameterization

Sequence convergence to a boundary Karush--Kuhn--Tucker (KKT) point has long remained unclear for nonconvex mirror descent with Legendre kernels. The difficulty arises from the blow-up of the gradient of the Legendre kernel at the boundary. Recent work~\cite{dingtoh2026nonkkt} shows that mirror descent can accumulate at non-KKT boundary points despite decreasing objective values, precluding a convergence guarantee to KKT points in general. Despite this negative result, mirror descent remains effective in many real applications. Motivated by this contrast, we address the boundary difficulty directly and establish KKT convergence of mirror descent for a broad class of structured nonconvex problems. We analyze mirror descent in reparameterized variables, where the Hessian metric is flattened and remains nondegenerate as the boundary is approached. Under extension and definability conditions jointly coupling the objective, the Legendre kernel, and the feasible region, the reparameterized sequence has finite length and converges, thereby recovering convergence to a KKT point of the original sequence. Our general framework applies to some concrete instances: Shannon entropy, Fermi--Dirac entropy, and power kernels on polyhedron.

math.OC↗

arXiv2026

Non-KKT Accumulation in Entropic Mirror Descent

For mirror descent generated by a Legendre kernel, perhaps one of the most basic question in optimization is this: must every accumulation point of a bounded mirror descent sequence be Karush--Kuhn--Tucker (KKT) stationary under proper stepsizes? We show that the answer is no. A longstanding obstacle to resolving this question is the boundary blow-up of the Legendre gradient: it keeps every mirror step in the interior, while at a boundary limit, the inverse entropy metric vanishes on active coordinates and can erase the dual-feasibility in the KKT system. We construct $C^\infty$ objectives and bounded sequences generated by the Shannon-entropic mirror descent on the nonnegative orthant $\R_+^n$, for every $n\geq 3$, and on the probability simplex $Δ_n$, for every $n\geq 4$, such that, in each case, the set of accumulation points is a smooth boundary circle containing a nonempty relatively open arc of non-KKT points. The steps satisfy $α_k\asymp k^{-β}$ with $β\in(1/2,1)$, the objective values are nonincreasing, and the objectives are entropy-relatively smooth. Hence the pathology stems from the degeneracy of the Bregman geometry at the boundary, rather than from failure of descent, or improper stepsizes. To the best of our knowledge, these provide the first counterexamples to KKT accumulation for bounded mirror descent sequences with nonincreasing objective values.

math.OC↗

arXiv2026

Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization

In this paper, we focus on providing convergence guarantees for stochastic subgradient methods in minimizing nonsmooth nonconvex functions. We first investigate the global stability of a general framework for stochastic subgradient methods, where the corresponding differential inclusion admits a coercive Lyapunov function. We prove that, for any sequence of sufficiently small stepsizes and approximation parameters, coupled with sufficiently controlled noises, the iterates are uniformly bounded and asymptotically stabilize around the stable set of its corresponding differential inclusion. Moreover, we develop an improved analysis to apply our proposed framework to establish the global stability of a wide range of stochastic subgradient methods, where the corresponding Lyapunov functions are possibly non-coercive. These theoretical results illustrate the promising potential of our proposed framework for establishing the global stability of various stochastic subgradient methods.

math.OC↗

arXiv2026

Sparsity-Cone SDP Relaxations and Applications to Variable Fixing for Sparse Quadratic Programs

Quadratic programs (QPs) with sparsity constraint are generally NP-hard, and their efficient global solution depends crucially on tractable tight convex relaxations. In this paper, we propose a sparsity-cone semidefinite programming (SC-SDP) relaxation for sparse (indefinite) QPs. Unlike standard SDP liftings, such as the SDP--RLT relaxation, which involve a $(2n+1)$-dimensional semidefinite matrix, the proposed SC-SDP formulation uses only a $(n+1)$-dimensional matrix together with a single sparsity-cone constraint $\mathcal{K}$ to handle the relaxation of the $\ell_0$-norm constraint. We prove that SC-SDP is equivalent in strength to the SDP--RLT relaxation. We further study the sparsity cone $\mathcal{K}$, deriving structural characterizations and showing that projection onto $\mathcal{K}$ can be computed efficiently via a one-dimensional subproblem. Building on the dual of SC-SDP, we derive explicit presolving mechanisms, including a dual-fixing rule for individual variables, a screening-cut rule for excluding larger support patterns, and a dual-refinement step for improving presolving certificates. To solve the resulting relaxation SC-SDP efficiently, we develop a two-phase Riemannian-based augmented Lagrangian method and exploits the structured projection subproblems. Numerical experiments on several classes of sparse QPs show that SC-SDP preserves the bound quality of SDP--RLT while offering substantial computational advantages and practically effective presolving capabilities.

math.OC↗

arXiv2026

Optimization Hyper-parameter Laws for Large Language Models

Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they fall short in choosing dynamic hyper-parameters, such as learning-rate (LR) schedules, that evolve during training. To bridge this gap, we present Optimization Hyper-parameter Laws (Opt-Laws), a framework that predicts final training loss as a function of LR schedule, model size, and data size. Grounded in SDE-based convergence and escape analyses, Opt-Laws yield interpretable convergence and escape features that predict final training loss across model scales, enabling schedule pre-selection from small-scale experiments. Empirically, Opt-Laws achieve a 94% Top-2 hit rate for identifying near-optimal schedule candidates on held-out configurations, correctly identify the best-performing schedule family in all five evaluated out-of-family settings, and detect training divergence with F1 = 0.92.

cs.LG↗

arXiv2026

A preconditioned augmented Lagrangian method for solving semidefinite programming problems

In this work, we propose a preconditioned augmented Lagrangian method (ALM) for solving semidefinite programming (SDP) problems. The preconditioner is implemented via a weighted penalty function in the ALM subproblem, with the weight matrix derived from the projection operator onto the tangent space of the feasible region. This simple yet effective modification significantly accelerates ALM, particularly for ill-conditioned SDPs. By combining the preconditioned ALM with our previously developed feasible method SDPF, we develop SDPF+, an SDP solver capable of handling convex problems with possibly nonlinear objective functions. Extensive numerical experiments demonstrate the efficiency and robustness of SDPF+, showing that it can generally outperform other solvers on large-scale SDPs whose optimal solutions exhibit low-rank structure.

math.OC↗

arXiv2026

ripALM: A Relative-Type Inexact Proximal Augmented Lagrangian Method for Linearly Constrained Convex Optimization

Inexact proximal augmented Lagrangian methods (ipALMs) have been widely used for solving linearly constrained convex optimization problems, owing to their strong theoretical guarantees and excellent numerical performance. In practice, however, existing ipALMs typically employ Rockafellar-type absolute error criteria for solving the subproblems, which require delicate problem-dependent tuning of error-tolerance sequences. In this paper, we propose ripALM, a relative-type ipALM whose subproblem error criterion has only a \textit{single} tolerance parameter in $[0,1)$. This makes the method simpler to implement and less sensitive to parameter tuning in practice. On the other hand, the use of such a relative-type error criterion renders the convergence of our ripALM beyond the scope of the convergence theory of existing ipALMs. To address this gap, we develop a new analysis framework under which ripALM is shown to admit desirable global convergence properties and it achieves an asymptotic (super)linear convergence rate under a standard error bound condition. While there exist other relative-type inexact pALMs, to ensure convergence, they require additional correct steps that generally impede the convergence speed. To the best of our knowledge, ripALM is the first relative-type inexact version of the vanilla pALM that avoids both summable tolerance parameter sequences and correction steps, while retaining rigorous convergence guarantees. Numerical experiments on quadratically regularized optimal transport and basis pursuit denoising problems demonstrate the effectiveness and robustness of our proposed method.

math.OC↗

arXiv2026

Gradient flow for finding E-optimal designs

The $E$-optimality criterion for a regression model maximizes the smallest eigenvalue of the information matrix and becomes non-differentiable when this eigenvalue has multiplicity greater than one. Working in the $2$-Wasserstein space, we show that the Wasserstein gradient at an empirical measure coincides, up to a constant factor, with the Euclidean particle gradient for smooth criteria such as $D$- and $L$-optimality, and that the approximation gap for equal-weight $N$-particle designs vanishes at an explicit rate. The main challenge is the nonsmooth $E$-criterion, for which the Wasserstein gradient does not exist. We replace it with a constrained Wasserstein steepest-ascent field obtained by maximizing feasible directional derivatives over the tangent cone of the design space, and prove that the resulting flow satisfies an exact energy identity and that every limit point is first-order stationary. The particle ascent computation reduces to a convex semidefinite programme whose dimension equals the multiplicity of the smallest eigenvalue. In numerical comparisons on second-order response surface models and a seven-dimensional logistic regression model, the constrained Wasserstein steepest-ascent method attains near-optimal $E$-criterion values and is markedly more reliable than particle swarm optimization in higher-dimensional settings. The framework applies more broadly to other nonsmooth minimax criteria in optimal design, and a numerical experiment on the minimax-single-parameter criterion confirms that the method attains the theoretical optimum.

math.OC↗

arXiv2026

Convergence Analysis of a Relative-type Inexact Preconditioned Proximal ALM for Convex Nonlinear Programming

This article investigates the convergence properties of a relative-type inexact preconditioned proximal augmented Lagrangian method (rip$^2$ALM) for convex nonlinear programming, a fundamental class of optimization problems with broad applications in science and engineering. Inexact proximal augmented Lagrangian methods have proven to be highly effective for solving such problems, owing to their attractive theoretical properties and strong practical performance. However, the convergence behavior of the relative-type inexact preconditioned variant remains insufficiently understood. This work aims to reduce this gap by rigorously establishing the global convergence of the sequence generated by rip$^2$ALM and proving its asymptotic (super)linear convergence rate under standard assumptions. In addition, we derive the global ergodic convergence rate with respect to both the primal feasibility violation and the primal objective residual, thereby offering a more comprehensive understanding of the overall performance of rip$^2$ALM. These results deepen our theoretical understanding of the family of proximal augmented Lagrangian methods and motivate their development for practical, large-scale structured application problems.

math.OC↗

arXiv2026

Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability

Long-context autoregressive decoding remains expensive because each decoding step must repeatedly process a growing history. We observe a consistent pattern during decoding: within a sentence, and more generally within a short semantically coherent span, the dominant attention support often remains largely stable. Motivated by this observation, we propose Slow-Fast Inference (SFI), a training-free decoding framework that decouples generation into frequent low-cost fast steps and occasional dense-attention slow steps. Fast steps reuse a compact sparse memory for efficient decoding. Slow steps are triggered near semantic boundaries. At slow steps, the model revisits the broader context and uses the Selector to refresh the selected memory for subsequent fast steps. Across the evaluated context lengths, SFI delivers approximately $1.6\times$--$14.4\times$ higher decoding throughput while generally maintaining quality on par with the full-KV baseline across long-context and long-CoT settings. Because SFI is training-free and applies directly to existing checkpoints, it offers a practical path to reducing inference cost for contemporary autoregressive reasoning models in long-context, long-horizon, and agentic workloads.

cs.LG↗

arXiv2026

Robust principal component analysis with rank and cardinality regularization under matrix factorization

Robust principal component analysis is an important representative method in data analysis. It is usually viewed as an optimization problem involving the rank and $\ell_0$-norm of matrices. In this paper, we study the rank and $\ell_0$ regularized optimization problem and its matrix factorization problem. We establish their equivalences on global minimizers and stationary points, respectively. Furthermore, we construct a broadly applicable equivalent nonconvex relaxation framework for the constrained factorization model in the sense of global minimizers and stationary points with strong optimality conditions (called strong stationary points). For the general factorization problem with lower semicontinuous regularizers and a loss function whose gradient is locally Lipschitz, we propose a novel proximal gradient-based algorithm based on joint and alternating calculation with convergence to its limiting-critical points. The algorithm can attain the stationary points of the original problem and its adaptive counterpart can attain the strong stationary points of the factorization problem.

math.OC↗

arXiv2026

On the efficient computation of proximal operators of affine-constrained nonconvex functions

Proximal operators with affine constraints arise in numerous models in nonconvex projection, composite optimization, and structured regularization. However, their efficient computation remains challenging due to the simultaneous presence of affine constraints and nonsmooth, possibly nonconvex objectives. In this work, we develop a unified dual-representability framework for analyzing and computing affine-constrained proximal mappings. Specifically, we introduce a multiplier inclusion formulation that connects the primal affine-constrained proximal problem to an unconstrained convex dual problem. Based on this formulation, we prove that, whenever the associated dual inclusion problem admits a solution, strong duality holds. For convex functions and a broad class of prox-regular nonconvex functions, we establish that dual representability holds under a simple subdifferential sum rule, and further develop a hierarchy of verifiable regularity conditions that guarantee this sum rule. In addition, we analyze the smoothness and strong convexity properties of the dual objective, providing a rigorous foundation that guarantees fast local convergence rates for efficient first- and second-order methods. Numerical experiments demonstrate that the proposed dual reformulation enables the reliable computation of globally optimal solutions for a range of large-scale nonconvex proximal and projection problems using existing convex optimization solvers.

math.OC↗

arXiv2026

Accelerating nuclear-norm regularized low-rank matrix optimization through Burer-Monteiro decomposition

This work proposes a rapid algorithm, BM-Global, for nuclear-norm-regularized convex and low-rank matrix optimization problems. BM-Global efficiently decreases the objective value via low-cost steps leveraging the nonconvex but smooth Burer-Monteiro (BM) decomposition, while effectively escapes saddle points and spurious local minima ubiquitous in the BM form to obtain guarantees of fast convergence rates to the global optima of the original nuclear-norm-regularized problem through aperiodic inexact proximal gradient steps on it. The proposed approach adaptively adjusts the rank for the BM decomposition and can provably identify an optimal rank for the BM decomposition problem automatically in the course of optimization through tools of manifold identification. BM-Global hence also spends significantly less time on parameter tuning than existing matrix-factorization methods, which require an exhaustive search for finding this optimal rank. Extensive experiments on real-world large-scale problems of recommendation systems, regularized kernel estimation, and molecular conformation confirm that BM-Global can indeed effectively escapes spurious local minima at which existing BM approaches are stuck, and is a magnitude faster than state-of-the-art algorithms for low-rank matrix optimization problems involving a nuclear-norm regularizer. Based on this research, we have released an open-source package of the proposed BM-Global at https://www.github.com/leepei/BM-Global/.

cs.LG↗

arXiv2025

Time-integrated Optimal Transport: A Robust Minimax Framework

Comparing time series in a principled manner requires capturing both temporal alignment and distributional similarity of features. Optimal transport (OT) has recently emerged as a powerful tool for this task, but existing OT-based approaches often depend on manually selected balancing parameters and can be computationally intensive. In this work, we introduce the Time-integrated Optimal Transport (TiOT) framework, which integrates temporal and feature components into a unified objective and yields a well-defined metric on the space of probability measures. This metric preserves fundamental properties of the Wasserstein distance, while avoiding the need for parameter tuning. To address the corresponding computational challenges, we introduce an entropic regularized approximation of TiOT, which can be efficiently solved using a block coordinate descent algorithm. Extensive experiments on both synthetic and real-world time series datasets demonstrate that our approach achieves improved accuracy and stability while maintaining comparable efficiency.

math.OC↗

arXiv2025

A Low-rank Augmented Lagrangian Method for Polyhedral-SDP and Moment-SOS Relaxations of Polynomial Optimization

Polynomial optimization problems (POPs) can be reformulated as geometric convex conic programs, as shown by Kim, Kojima, and Toh (SIOPT 30:1251-1273, 2020), though such formulations remain NP-hard. In this work, we prove that several well-known relaxations can be unified under a common polyhedral-SDP framework, which arises by approximating the intractable cone by tractable intersections of polyhedral cones with the positive semidefinite matrix cone. Although effective in providing tight lower bounds, these relaxations become computationally expensive as the number of variables and constraints grows at the rate of $Ω(n^{2τ})$ with the relaxation order $τ$. To address this challenge, we propose RiNNAL-POP, a low-rank augmented Lagrangian method (ALM) tailored to solve large-scale polyhedral-SDP relaxations of POPs. To efficiently handle the $Ω(n^{2τ})$ nonnegativity and consistency constraints, we design a tailored projection scheme whose computational cost scales linearly with the number of variables. In addition, we identify a hidden facial structure in the polyhedral-SDP relaxation, which enables us to eliminate a large number of linear constraints by restricting the matrix variable to affine subspaces corresponding to exposed faces of the semidefinite cone. The latter enables us to efficiently solve the factorized ALM subproblems over the affine subspaces. At each ALM iteration, we additionally carry out a single projected gradient step with respect to the original matrix variable to automatically adjust the rank and escape from spurious local minima when necessary. We also extend our RiNNAL-POP algorithmic framework to solve moment-SOS relaxations of POPs. Extensive numerical experiments on various benchmark problems demonstrate the robustness and efficiency of RiNNAL-POP in solving large-scale polyhedral-SDP relaxations.

math.OC↗

arXiv2025

A Quadratically Convergent Alternating Projection Method for Nonconvex Sets

In this paper, we consider the feasibility problem, which aims to find a feasible point for the constraint set $\{x \in \mathbb{R}^n: c(x) = 0\}$ over a possibly non-regular subset $\mathcal{X} \subset \mathbb{R}^n$. Under the constraint nondegeneracy condition, we propose a modified alternating projection method. In our proposed method, based on the concept of projective mapping for $\mathcal{X}$, we alternate a Newton step for finding an inexact solution within the limiting tangent cone of $\mathcal{X}$ and a projection to $\mathcal{X}$. Under mild conditions, we prove the local quadratic convergence of our proposed method. Preliminary numerical experiments demonstrate the high efficiency of our proposed alternating projection method.

math.OC↗

arXiv2025

A Bregman ADMM for Bethe variational problem

In this work, we propose a novel Bregman ADMM with nonlinear dual update to solve the Bethe variational problem (BVP), a key optimization formulation in graphical models and statistical physics. Our algorithm provides rigorous convergence guarantees, even if the objective function of BVP is non-convex and non-Lipschitz continuous on the boundary. A central result of our analysis is proving that the entries in local minima of BVP are strictly positive, effectively resolving non-smoothness issues caused by zero entries. Beyond theoretical guarantees, the algorithm possesses high level of separability and parallelizability to achieve highly efficient subproblem computation. Our Bregman ADMM can be easily extended to solve the quantum Bethe variational problem. Numerical experiments are conducted to validate the effectiveness and robustness of the proposed method. Based on this research, we have released an open-source package of the proposed method at https://github.com/TTYmath/BADMM-BVP.

math.OC↗