arXiv subjects

Benjamin Grimmer

Publications and source records attributed to Benjamin Grimmer.

At least 19 recordsLinked to original sources

arXiv2026

On Minimax Optimality and Uniqueness of Fixed-Step First-Order Methods for Smooth Convex Optimization

This paper considers the design of optimal fixed-step first-order methods for high-dimensional minimization of $L$-smooth convex functions. For optimizing worst-case performance measured via suboptimality of the final function value (relative to the initial squared distance to a minimizer), we provide an algebraic proof of the optimality of the optimized gradient method (OGM) and establish its uniqueness among all fixed-step first-order methods. For the alternative measure of final squared gradient norm (relative to initial suboptimality), we prove the OGM-G method is optimal and uniquely so among fixed-step first-order methods. Finally, for the setting measuring the final squared gradient norm (relative to the initial squared distance to a minimizer), we show the recently proposed Lemniscate method is optimal and uniquely so. Our proofs rely on algebraic reductions for lower bound arguments rather than traditional information-theoretic bounds, which were previously only able to establish OGM's optimality but not uniqueness.

math.OC↗

arXiv2026

Implicit Primal-Dual Guarantees in Unconstrained First-Order Minimization

This work considers the design of first-order convex optimization algorithms and convergence proofs. In particular, we consider nonsmooth Lipschitz and smooth problems accessed through a subgradient or gradient oracle, respectively. For the general class of fixed-step first-order methods, prior work on Performance Estimation Problems (PEPs) has shown that structured, tight convergence proofs typically exist. Under mild conditions, we further show that any first-order method guaranteeing a bound on the primal objective gap $f(x_N)-f(x_\star)$ assuming only a bound on $\|x_0-x_\star\|$ actually has a stronger guarantee on an explicit, computable primal-dual gap at the same rate. These implicit optimal dual certificates, which take the form of affine lower bounds, also provide insight into the role of auxiliary sequences in momentum methods.

math.OC↗

arXiv2026

A Complete Characterization of Optimal Subgradient Methods for Lipschitz Convex Minimization

We consider the design of optimal fixed-step first-order methods for $M$-Lipschitz convex optimization given $\|x_0-x_\star\|\leq D$. Prior works have identified several distinct fixed-step methods, parameterized by a matrix of stepsizes $W$, with the (information-theoretic) minimax optimal rate $MD/\sqrt{N+1}$ of objective gap convergence. We provide a complete characterization of every optimal fixed-step method. Moreover, we show every optimal fixed-step method can be derived from the constructive approach of~\cite{constructive_approach} and provide a polyhedral representation of the set of optimal methods through proof multipliers. From this characterization, we show that no anytime optimal fixed-step subgradient methods exist.

math.OC↗

arXiv2026

Inexactly Smooth Performance Estimation and New Optimized Gradient Methods

We consider a general class of ``inexactly smooth'' convex functions, providing a universal model capturing as special cases $L$-smooth, $M$-Lipschitz, and H\"older smooth functions, and any combination thereof. Such functions possess a calculus closely following that of smooth functions. Our main results provide inexactly smooth functions with interpolation theorems that are necessary and sufficient up to modest universal constants. These enable analysis of first-order methods for any inexactly smooth convex problem class via solving convex Performance Estimation Problems (PEPs). Further, these enable the extension of Drori and Taylor's constructive approach to algorithm design. From this, we derive an exactly minimax optimal method for $(\beta,0)$-H\"older smooth problems, methods with the best-known convergence guarantees up to constants for any $(\beta,p)$-H\"older smooth convex minimization, and a new universal fast backtracking method for any inexactly smooth convex problem.

math.OC↗

arXiv2026

A Parameter-Free Restart Scheme with Only a Parallelizable $\log\log(1/\epsilon)$ Overhead

It is well-known that first-order methods can offer accelerated convergence rates in the presence of growth structures. Restarting schemes provide a general tool for such speed-ups. These schemes typically either require unrealistic problem knowledge, incur logarithmic overhead factors in oracle complexity, and/or have a nontrivial initial burn-in phase. We present a parameter-free approach for restarting any first-order method, avoiding these three drawbacks. Our approach dynamically deploys parallel instances of a given first-order method communicating progress in the style of Renegar and Grimmer. Our optimized scheme avoids expensive burn-ins and only requires $O(\log\log(1/\epsilon))$ parallel processes when the accelerated rate is sublinear.

math.OC↗

arXiv2026

A Theory of Composition and Duality of Extremal Optimal Fixed-Point Algorithms

In this work, we reveal a rich combinatorial structure underlying exact minimax optimal algorithms for classical nonexpansive fixed-point problems. This viewpoint unifies all extremal optimal methods and provides a systematic and practical framework for designing new algorithms via diagrams. Specifically, we study fixed-step algorithms represented by a lower triangular matrix H, and show that the set of optimal (N-1)-step algorithms has exactly (N-1)! vertices (extremal algorithms), each of which naturally corresponds to an arc diagram, a graph that encodes its convergence proof. Using these arc diagrams, we can compose, decompose, and analyze the properties of distinct optimal vertex algorithms. Furthermore, we determine when the H-dual operation, given by taking the anti-diagonal transpose of H, preserves the optimality of a vertex algorithm, and in such cases we characterize the convergence proof of the dual algorithm. Based on this machinery, we develop new optimal algorithms with quasi-anytime guarantees; that is, they admit an increasing integer sequence such that the corresponding iterates have the optimal residual guarantees, and are additionally robust to fixed-point operators that violate nonexpansiveness.

math.OC↗

arXiv2026

Nonsmooth Riemannian optimization with inexact manifold primitives via bundle methods

Optimization on Hadamard manifolds -- the natural Riemannian setting for globally geodesically convex problems -- relies on exponential maps to retract tangent vectors and parallel transport to connect tangent spaces across the manifold. These primitives are often computationally expensive, leading software packages to rely on approximations: first-order retractions and vector transports. However, existing results for optimization on Hadamard manifolds either require exact primitives or lack non-asymptotic rates. We bridge this gap by introducing a proximal bundle method for nonsmooth geodesically convex optimization and establishing the first oracle-complexity bounds that rely only on subgradients and inexact primitives. We obtain sublinear rates for general objectives and optimal linear convergence under sharp function growth.

math.OC↗

arXiv2026

Lower Bounds for Linear Minimization Oracle Methods Optimizing over Strongly Convex Sets

We consider the oracle complexity of constrained convex optimization given access to a Linear Minimization Oracle (LMO) for the constraint set and a gradient oracle for the $L$-smooth, $L$-strongly convex objective. This model includes Frank-Wolfe methods and their many variants. Over the problem class of $\alpha$-strongly convex constraint sets $S$, we demonstrate that one can construct hard ``zero-chain'' instances in the classical style of Nemirovski and Yudin. From our new approach to adversarial oracle construction, we prove that no such deterministic method can guarantee a final objective gap less than $\varepsilon$ in fewer than $\Omega(\sqrt{L\, \mathrm{diam}(S)^2/\varepsilon})$ iterations. Our lower bound partly matches the accelerated Frank-Wolfe theory of Garber and Hazan (2015) of $O(\sqrt{L(\mathrm{diam}(S)^2+1/\alpha^2)/\varepsilon})$. Second, we consider optimization over $\beta$-smooth sets, finding that in the modestly smooth regime of $\beta=\Omega(1/\sqrt{\varepsilon})$, no complexity improvement for span-based LMO methods is possible against either compact convex sets or strongly convex sets.

math.OC↗

arXiv2025

An Elementary Proof of the Near Optimality of LogSumExp Smoothing

We consider the design of smoothings of the (coordinate-wise) max function in $\mathbb{R}^d$ in the infinity norm. The LogSumExp function $f(x)=\ln(\sum^d_i\exp(x_i))$ provides a classical smoothing, differing from the max function in value by at most $\ln(d)$. We provide an elementary construction of a lower bound, establishing that every overestimating smoothing of the max function must differ by at least $\sim 0.8145\ln(d)$. Hence, LogSumExp is optimal up to small constant factors. However, we provide strictly stronger smoothings showing the entropy-based LogSumExp approach is not exactly optimal. In small dimensions, we propose exactly optimal smoothings, attaining our lower bound.

math.ST↗

arXiv2025

H-invariance theory: A complete characterization of minimax optimal fixed-point algorithms

For nonexpansive fixed-point problems, Halpern's method with optimal parameters, its so-called H-dual algorithm, and in fact, an infinite family of algorithms containing them, all exhibit the exactly minimax optimal convergence rates. In this work, we provide a characterization of the complete, exhaustive family of distinct algorithms using predetermined step-sizes, represented as lower triangular H-matrices, which attain the same optimal convergence rate. The characterization is based on polynomials in the entries of the H-matrix that we call H-invariants, whose values stay constant over all optimal H-matrices, together with H-certificates, of which nonnegativity precisely specifies the region of optimality within the common level set of H-invariants. The H-invariance theory we present offers a novel view of optimal acceleration in first-order optimization as a mathematical study of carefully selected invariants, certificates, and structures induced by them.

math.OC↗

arXiv2025

Subgame Perfect Methods in Nonsmooth Convex Optimization

This paper considers nonsmooth convex optimization with either a subgradient or proximal operator oracle. In both settings, we identify algorithms that achieve the recently introduced game-theoretic optimality notion for algorithms known as subgame perfection. Subgame perfect algorithms meet a more stringent requirement than just minimax optimality. Not only must they provide optimal uniform guarantees on the entire problem class, but also on any subclass determined by information revealed during the execution of the algorithm. In the setting of nonsmooth convex optimization with a subgradient oracle, we show that the Kelley cutting plane-Like Method due to Drori and Teboulle [1] is subgame perfect. For nonsmooth convex optimization with a proximal operator oracle, we develop a new algorithm, the Subgame Perfect Proximal Point Algorithm, and establish that it is subgame perfect. Both of these methods solve a history-aware second-order cone program within each iteration, independent of the ambient problem dimension, to plan their next steps. This yields performance guarantees that are never worse than the minimax optimal guarantees and often substantially better.

math.OC↗

arXiv2025

A Practical Adaptive Subgame Perfect Gradient Method

We present a performant gradient method for smooth convex optimization, drawing inspiration from several recent advances in the field. Our algorithm, the Adaptive Subgame Perfect Gradient Method (ASPGM) is based on the notion of subgame perfection, attaining a dynamic strengthening of minimax optimality. At each iteration, ASPGM makes a momentum-type update, optimized dynamically based on a (limited) memory/bundle of past first-order information. ASPGM is linesearch-free, parameter-free, and adaptive due to its use of recently developed auto-conditioning, restarting, and preconditioning ideas. We show that ASPGM is competitive with state-of-the-art L-BFGS methods on a wide range of smooth convex problems. Unlike quasi-Newton methods, however, our core algorithm underlying ASPGM has strong, subgame perfect, non-asymptotic guarantees, providing certificates of solution quality, resulting in simple stopping criteria and restarting conditions.

math.OC↗

arXiv2025

Some Unified Theory for Variance Reduced Prox-Linear Methods

This work considers the nonconvex, nonsmooth problem of minimizing a composite objective of the form $f(g(x))+h(x)$ where the inner mapping $g$ is a smooth finite summation or expectation amenable to variance reduction. In such settings, prox-linear methods can enjoy variance-reduced speed-ups despite the existence of nonsmoothness. We provide a unified convergence theory applicable to a wide range of common variance-reduced vector and Jacobian constructions. All the technical conditions we required for variance-reduced methods can be summarized in a single unified assumption. Our theory (i) only requires operator norm bounds on Jacobians (whereas prior works used potentially much larger Frobenius norms), (ii) provides state-of-the-art high probability guarantees, and (iii) allows inexactness in proximal computations.

math.OC↗

arXiv2025

Gauges and Accelerated Optimization over Smooth and/or Strongly Convex Sets

We consider feasibility and constrained optimization problems defined over smooth and/or strongly convex sets. These notions mirror their popular function counterparts but are much less explored in the first-order optimization literature. We propose new scalable, projection-free, accelerated first-order methods in these settings. Our methods avoid linear optimization or projection oracles, only using cheap one-dimensional linesearches and normal vector computations. Despite this, we derive optimal accelerated convergence guarantees of $O(1/T)$ for strongly convex problems, $O(1/T^2)$ for smooth problems, and accelerated linear convergence given both. Our algorithms and analysis are based on novel characterizations of the Minkowski gauge of smooth and/or strongly convex sets, which may be of independent interest: although the gauge is neither smooth nor strongly convex, we show the gauge squared inherits any structure present in the set.

math.OC↗

arXiv2025

Composing Optimized Stepsize Schedules for Gradient Descent

Recent works by Altschuler and Parrilo and the authors have shown that it is possible to accelerate the convergence of gradient descent on smooth convex functions, even without momentum, just by picking special stepsizes. In this paper, we provide a general theory for composing stepsize schedules capturing all recent advances in this area and more. We propose three notions of ``composable'' stepsize schedules with elementary associated composition operations for combining them. From these operations, in addition to recovering recent works, we construct three highly optimized sequences of stepsize schedules. We first construct optimized stepsize schedules of every length generalizing the exponentially spaced silver stepsizes. We then construct highly optimized stepsizes schedules for minimizing final objective gap or gradient norm, improving on prior rates by constants and, more importantly, matching or beating the numerically computed minimax optimal schedules. We conjecture these schedules are in fact minimax (information theoretic) optimal. Several novel tertiary results follow from our theory including recovery of the recent dynamic gradient norm minimizing short stepsizes and extending them to objective gap minimization.

math.OC↗

arXiv2025

Some New Insights from Highly Optimized Polyhedral Passages

A shape possesses Rupert's property if a hole can be cut through it such that a second identical copy of the shape can cleanly pass straight through the interior of the first. Such a passage proving cubes are Rupert was first shown more than 300 years ago. It remains open whether every polyhedron in three dimensions is Rupert. We propose a customized subgradient method providing high-accuracy local numerical optimization of the quality of a passage for a given polyhedron. From extensive numerical searches, we improve these best-known passages for more than half of the Platonic, Archimedean, and Catalan solids and for numerous Johnson solids. Our high accuracy solves support a new conjecture of a simple form for the Tetrahedron's optimal passage. Despite our computational search, three Archimedean and two Catalan solids remain open, providing further negative evidence against the conjecture that all polyhedrons are Rupert.

math.OC↗

arXiv2025

The Optimal Smoothings of Sublinear Functions and Convex Cones

This paper considers the problem of smoothing convex functions and sets, seeking the nearest smooth convex function or set to a given one. For convex cones and sublinear functions, a full characterization of the set of all optimal smoothings is given. These provide if and only if characterizations of the set of optimal smoothings for any target level of smoothness. Optimal smoothings restricting to either inner or outer approximations also follow from our theory. Finally, we apply our theory to provide insights into smoothing amenable functions given by compositions with sublinear functions and generic convex sets by expressing them as conic sections.

math.OC↗

arXiv2025

On Averaging and Extrapolation for Gradient Descent

This work considers the effect of averaging, and more generally extrapolation, of the iterates of gradient descent in smooth convex optimization. After running the method, rather than reporting the final iterate, one can report either a convex combination of the iterates (averaging) or a generic combination of the iterates (extrapolation). For several common stepsize sequences, including recently developed accelerated periodically long stepsize schemes, we show averaging cannot improve gradient descent's worst-case performance and is, in fact, strictly worse than simply returning the last iterate. In contrast, we prove a conceptually simple and computationally cheap extrapolation scheme strictly improves the worst-case convergence rate: when initialized at the origin, reporting $(1+1/\sqrt{16N\log(N)})x_N$ rather than $x_N$ improves the best possible worst-case performance by the same amount as conducting $O(\sqrt{N/\log(N)})$ more gradient steps. Our analysis and characterizations of the best-possible convergence guarantees are computer-aided, using performance estimation problems. Numerically, we find similar (small) benefits from such simple extrapolation for a range of gradient methods.

math.OC↗