Searcharxiv⌕ Search

arXiv · 2611.19887

First-order smoothed exact-penalty methods for equality-constrained regularized nonconvex optimization

Abstract

We consider regularized nonconvex optimization over a closed convex set $\mathcal{X} \subseteq \mathbb{R}^d$, which involves a convex nonsmooth regularizer $r$ in the objective and $n$ smooth equality constraints. To find a near-KKT point of the problem, we propose a prox-linear first-order method, named as SEPi-Prox, based on a smoothed exact-penalty formulation. At each outer iteration of SEPi-Prox, a strongly convex prox-linear subproblem is formed and solved inexactly to a sufficient accuracy. Under a local non-singularity condition that is weaker than some other commonly used regularity conditions or constraint qualifications, SEPi-Prox finds an $ε$-KKT point within $O(ε^{-2})$ first-order oracle calls, matching the lower bound for unconstrained smooth nonconvex optimization. We employ Nesterov's accelerated proximal-gradient method (APG) and a dual cutting-plane method (DCP) for the prox-linear subproblems. Let $\mathcal{C}_{\mathrm{FO}}$ and $\mathcal{C}_r$ denote the numbers of arithmetic operations of evaluating one first-order oracle and one proximal mapping of $r$ over $\mathcal{X}$, respectively. Then SEPi-Prox requires $\widetilde{O}(\mathcal{C}_{\mathrm{FO}}ε^{-2} + (dn + \mathcal{C}_r)ε^{-5/2})$ and $\widetilde{O}((\mathcal{C}_{\mathrm{FO}} + dn^2 + n\mathcal{C}_r + n^{3.373})ε^{-2})$ total arithmetic operations to produce an $ε$-KKT point, respectively by using APG and DCP as the subsolver. The latter result is preferable when $n$ is small and nearly matches the optimal dependence on $ε$. Numerical experiments on instances of the multi-class Neyman-Pearson classification demonstrate the efficiency of SEPi-Prox with both subsolvers.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Shiji Zuo, Qihang Lin, Wei Liu, Yangyang Xu. 2026-11-15. First-order smoothed exact-penalty methods for equality-constrained regularized nonconvex optimization. https://searcharxiv.com/abs/2611.19887

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

First-order methods for stochastic and finite-sum convex optimization with deterministic constraints

In this paper, we study a class of stochastic and finite-sum convex optimization problems with deterministic constraints. Existing methods typically aim to find an $ε$-$expected\ feasible\ stochastic\ optimal$ solution, in which both the expected constraint violation and the expected optimality gap are within a prescribed tolerance $ε$. However, in many practical applications, constraints must be nearly satisfied with certainty, rendering such solutions potentially unsuitable due to the risk of substantial violations. To address this issue, we propose stochastic first-order methods for finding an $ε$-$surely\ feasible\ stochastic\ optimal$ ($ε$-SFSO) solution, where the constraint violation is deterministically bounded by $ε$ and the expected optimality gap is at most $ε$. Our methods apply an accelerated stochastic gradient (ASG) scheme or a modified variance-reduced ASG scheme $only\ once$ to a sequence of quadratic penalty subproblems with appropriately chosen penalty parameters. We establish first-order oracle complexity bounds for the proposed methods in computing an $ε$-SFSO solution. Preliminary numerical results are presented to illustrate the performance of the proposed methods.

math.OC↗

Attention's forward pass and Frank-Wolfe

We study the hardmax limit of self-attention dynamics for token embeddings obtained in the zero-temperature ($β\to+\infty$) regime, and relate it to the finite-$β$ setting. In this limit, the update rule can be viewed as a Frank-Wolfe step for a quadratic objective over the convex hull of the current token embeddings. When the key-query matrix is negative semidefinite, the method linearly contracts all tokens to a single cluster at the origin. When it is positive semidefinite, extending the hardmax rule to the entire convex hull induces a Voronoi diagram: vertices are stationary, interior points remain in their initial cells, and each token moves along a straight line toward its cell's vertex, yielding (super-)exponential convergence. As a byproduct, we also establish well-posedness of the associated ODE limit in this regime. Returning to the finite-$β$ regime, we model self-attention dynamics as a Markov chain and prove dynamic metastability: with high probability, interior tokens reach near-vertex configurations in a constant number of steps and remain within a small neighborhood for times that grow exponentially in the inverse temperature $β$, before ultimately collapsing to the origin. Thus, the hardmax dynamics accurately approximate the finite-$β$ process over exponentially long time horizons.

math.OC↗

Finite de Finetti for convex bodies and Polynomial Optimization

We study polynomial optimization over convex bodies with separately affine objectives and local affine equality and inequality constraints. We construct a convergent hierarchy of conic relaxations with explicit finite-level error bounds of order $n^{-1/2}$ at level $n$, complemented by a constructive rounding scheme that produces feasible points. The relaxation bounds and rounded objective values bracket the optimum at the same rate. The analytic ingredient is a finite de Finetti representation theorem for arbitrary proper cones: bipartite marginals of symmetric extensions in maximal tensor products are $O(n^{-1/2})$-close, in base norm, to points of the minimal tensor product. Its proof combines a uniform, monogamy-type bound on a variational mutual information based on Frenkel's relative entropy with classical conditioning and informationally complete measurements. We apply the framework to the assemblage relaxation of two-player non-local games in general probabilistic theories. For polytopic state spaces, the hierarchy consists of linear programs, and experiments on the Clauser--Horne--Shimony--Holt (CHSH) game provide a numerical proof of concept for the outer approximations and feasible rounding procedure.

math.OC↗