arXiv subjects

Kevin Shu

Publications and source records attributed to Kevin Shu.

At least 19 recordsLinked to original sources

arXiv2026

A Unified Theory of H-Duality in First-Order Methods

We provide two complementary explanations of H-duality in smooth strongly convex optimization and contractive fixed-point problems. H-duality refers to the phenomenon where the worst-case performance of many fixed-step first-order methods (FSFOMs) for a given performance setup are exactly equal to the worst-case performance of their "time-reversed" counterparts and a paired performance setup. This concept gained interest as a mechanism for automatically converting an algorithm with guarantees on the final suboptimality to algorithms with guarantees on the final gradient norm. Since its initial discovery, additional H-dual performance setup pairings have been discovered empirically, yet a general explanation for these phenomena remained elusive. Our first explanation of H-duality shows that, on a class of instances with extremal curvature properties, FSFOMs produce a final iterate that is a scalar multiple of the initial iterate, and that this scalar is invariant under the time reversal of the underlying FSFOM. This, together with the empirical observation that many FSFOMs have such an instance with extremal curvature as their worst case, gives one explanation for why H-duality often occurs. The second explanation is complementary and uses the notion of a performance estimation certificate of the convergence rate of a FSFOM. We exhibit an explicit transformation of such certificates that preserves many of the properties that define such a certificate, and we show that in many numerical and analytical examples, this transformation in fact transforms certificates of the convergence rate of an algorithm into certificates for its time-reversal. As an application, we prove a convergence guarantee for the H-dual of ITEM matching the conjectured optimal gradient-to-gradient guarantee.

math.OC↗

arXiv2025

Lagrangian Dual Sections: A Topological Perspective on Hidden Convexity

Hidden convexity is a powerful idea in optimization: under the right transformations, nonconvex problems that are seemingly intractable can be solved efficiently using convex optimization. We introduce the notion of a Lagrangian dual section of a nonlinear program defined over a topological space, and we use it to give a sufficient condition for a nonconvex optimization problem to have a natural convex reformulation. We emphasize the topological nature of our framework, using only continuity and connectedness properties of a certain Lagrangian formulation of the problem to prove our results. We demonstrate the practical consequences of our framework in a range of applications and by developing new algorithmic methodology. First, we present families of nonconvex problem instances that can be transformed to convex programs in the context of spectral inverse problems -- which include quadratically constrained quadratic optimization and Stiefel manifold optimization as special cases -- as well as unbalanced Procrustes problems. In each of these applications, we both generalize prior results on hidden convexity and provide unifying proofs. For the case of the spectral inverse problems, we also present a Lie-theoretic approach that illustrates connections with the Kostant convexity theorem. Second, we introduce new algorithmic ideas that can be used to find globally optimal solutions to both Lagrangian forms of an optimization problem as well as constrained optimization problems when the underlying topological space is a Riemannian manifold.

math.OC↗

arXiv2025

Debiasing Polynomial and Fourier Regression

We study the problem of approximating an unknown function $f:\mathbb{R}\to\mathbb{R}$ by a degree-$d$ polynomial using as few function evaluations as possible, where error is measured with respect to a probability distribution $\mu$. Existing randomized algorithms achieve near-optimal sample complexities to recover a $ (1+\varepsilon) $-optimal polynomial but produce biased estimates of the best polynomial approximation, which is undesirable. We propose a simple debiasing method based on a connection between polynomial regression and random matrix theory. Our method involves evaluating $f(\lambda_1),\ldots,f(\lambda_{d+1})$ where $\lambda_1,\ldots,\lambda_{d+1}$ are the eigenvalues of a suitably designed random complex matrix tailored to the distribution $\mu$. Our estimator is unbiased, has near-optimal sample complexity, and experimentally outperforms iid leverage score sampling. Additionally, our techniques enable us to debias existing methods for approximating a periodic function with a truncated Fourier series with near-optimal sample complexity.

cs.DS↗

arXiv2024

Beyond Minimax Optimality: A Subgame Perfect Gradient Method

The study of convex optimization has historically been concerned with worst-case convergence rates. The development of the Optimized Gradient Method (OGM), due to \citet{drori2012PerformanceOF,Kim2016optimal}, marked a major milestone in this study, as OGM achieves the optimal worst-case convergence rate among all first-order methods for unconstrained smooth convex optimization. In order to examine the possibility of obtaining stronger convergence guarantees, we will consider algorithms with \emph{dynamic} convergence rates, which may improve as additional first-order information is revealed. Our main contribution is the development of an algorithm, the Subgame Perfect Gradient Method (SPGM), which refines OGM to make use of the full history of first-order information. We show that SPGM is \emph{dynamically optimal}, in the sense that in each iteration, no other algorithm can offer a strictly better convergence rate on all functions which agree with the observed first-order information up to that iteration. We formalize this notion of dynamic optimality using the game-theoretic notion of a subgame perfect equilibrium. We conclude our study with preliminary numerical experiments showing that SPGM strongly outperforms OGM.

math.OC↗

arXiv2024

Composing Optimized Stepsize Schedules for Gradient Descent

Recent works by Altschuler and Parrilo and the authors have shown that it is possible to accelerate the convergence of gradient descent on smooth convex functions, even without momentum, just by picking special stepsizes. In this paper, we provide a general theory for composing stepsize schedules capturing all recent advances in this area and more. We propose three notions of ``composable'' stepsize schedules with elementary associated composition operations for combining them. From these operations, in addition to recovering recent works, we construct three highly optimized sequences of stepsize schedules. We first construct optimized stepsize schedules of every length generalizing the exponentially spaced silver stepsizes. We then construct highly optimized stepsizes schedules for minimizing final objective gap or gradient norm, improving on prior rates by constants and, more importantly, matching or beating the numerically computed minimax optimal schedules. We conjecture these schedules are in fact minimax (information theoretic) optimal. Several novel tertiary results follow from our theory including recovery of the recent dynamic gradient norm minimizing short stepsizes and extending them to objective gap minimization.

math.OC↗

arXiv2024

A Strengthened Conjecture on the Minimax Optimal Constant Stepsize for Gradient Descent

Drori and Teboulle [4] conjectured that the minimax optimal constant stepsize for N steps of gradient descent is given by the stepsize that balances performance on Huber and quadratic objective functions. This was numerically supported by semidefinite program (SDP) solves of the associated performance estimation problems up to $N\approx 100$. This note presents a strengthened version of the initial conjecture. Specifically, we conjecture the existence of a certificate for the convergence rate with a very specific low-rank structure. This structure allows us to bypass SDPs and to numerically verify both conjectures up to $N = 20160$.

math.OC↗

arXiv2024

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps

This work considers gradient descent for L-smooth convex optimization with stepsizes larger than the classic regime where descent can be ensured. The stepsize schedules considered are similar to but differ slightly from the recent silver stepsizes of Altschuler and Parrilo. For one of our stepsize sequences, we prove a $O\left(N^{- 1.2716\dots}\right)$ convergence rate in terms of objective gap decrease and for the other, we show the same rate of decrease for squared-gradient-norm decrease. This first result improves on the recent result of Altschuler and Parrilo by a constant factor, while the second results improve on the exponent of the prior best squared-gradient-norm convergence guarantee of $O\left(N^{-1}\right)$.

math.OC↗

arXiv2023

Accelerated Gradient Descent via Long Steps

Recently Grimmer [1] showed for smooth convex optimization by utilizing longer steps periodically, gradient descent's textbook $LD^2/2T$ convergence guarantees can be improved by constant factors, conjecturing an accelerated rate strictly faster than $O(1/T)$ could be possible. Here we prove such a big-O gain, establishing gradient descent's first accelerated convergence rate in this setting. Namely, we prove a $O(1/T^{1.0564})$ rate for smooth convex minimization by utilizing a nonconstant nonperiodic sequence of increasingly large stepsizes. It remains open if one can achieve the $O(1/T^{1.178})$ rate conjectured by Das Gupta et. al. [2] or the optimal gradient method rate of $O(1/T^2)$. Big-O convergence rate accelerations from long steps follow from our theory for strongly convex optimization, similar to but somewhat weaker than those concurrently developed by Altschuler and Parrilo [3].

math.OC↗

arXiv2023

Symmetric Hyperbolic Polynomials

Hyperbolic polynomials have been of recent interest due to applications in a wide variety of fields. We seek to better understand these polynomials in the case when they are symmetric, i.e. invariant under all permutations of variables. We give a complete characterization of the set of symmetric hyperbolic polynomials of degree 3, and a large class of symmetric hyperbolic polynomials of degree 4. For a class of polynomials, which we call hook-shaped, we relate symmetric hyperbolic polynomials to a class of linear maps of univariate polynomials preserving hyperbolicity, and give evidence toward a beautiful characterization of all such hook-shaped symmetric hyperbolic polynomials. We show that hyperbolicity cones of a class of symmetric hyperbolic polynomials, including all symmetric hyperbolic cubics, are spectrahedral. Finally, we connect testing hyperbolicity of a symmetric polynomial to the degree principle for symmetric nonnegative polynomials.

math.AG↗

arXiv2023

Hidden convexity, optimization, and algorithms on rotation matrices

This paper studies hidden convexity properties associated with constrained optimization problems over the set of rotation matrices $\text{SO}(n)$. Such problems are nonconvex due to the constraint $X \in \text{SO}(n)$. Nonetheless, we show that certain linear images of $\text{SO}(n)$ are convex, opening up the possibility for convex optimization algorithms with provable guarantees for these problems. Our main technical contributions show that any two-dimensional image of $\text{SO}(n)$ is convex and that the projection of $\text{SO}(n)$ onto its strict upper triangular entries is convex. These results allow us to construct exact convex reformulations for constrained optimization problems over $\text{SO}(n)$ with a single constraint or with constraints defined by low-rank matrices. Both of these results are optimal in a formal sense.

math.OC↗

arXiv2022

Quadratic Programming with Sparsity Constraints via Polynomial Roots

Quadratically constrained quadratic programs (QCQPs) are an expressive family of optimization problems that occur naturally in many applications. It is often of interest to seek out sparse solutions, where many of the entries of the solution are zero. This paper will consider QCQPs with a single linear constraint, together with a sparsity constraint that requires that the set of nonzero entries of a solution be small. This problem class includes many fundamental problems of interest, such as sparse versions of linear regression and principal component analysis, which are both known to be very hard to approximate. We introduce a family of tractable approximations of such sparse QCQPs using the roots of polynomials which can be expressed as linear combinations of principal minors of a matrix. These polynomials arose naturally from the study of hyperbolic polynomials. Our main contributions are formulations of these approximations and computational methods for finding good solutions to a sparse QCQP. We will also give numerical evidence that these methods can be effective on practical problems.

math.OC↗

arXiv2021

Linear Principal Minor Polynomials: Hyperbolic Determinantal Inequalities and Spectral Containment

A linear principal minor polynomial or lpm polynomial is a linear combination of principal minors of a symmetric matrix. By restricting to the diagonal, lpm polynomials are in bijection to multiaffine polynomials. We show that this establishes a one-to-one correspondence between homogeneous multiaffine stable polynomials and PSD-stable lpm polynomials. This yields new construction techniques for hyperbolic polynomials and allows us to generalize the well-known Fisher--Hadamard and Koteljanskii inequalities from determinants to PSD-stable lpm polynomials. We investigate the relationship between the associated hyperbolicity cones and conjecture a relationship between the eigenvalues of a symmetric matrix and the values of certain lpm polynomials evaluated at that matrix. We refer to this relationship as spectral containment.

math.AG↗

arXiv2021

On the computational equivalence of co-NP refutations of a matrix being a P-matrix

A P-matrix is a square matrix $X$ such that all principal submatrices of $X$ have positive determinant. Such matrices appear naturally in instances of the linear complementarity problem, where these are precisely the matrices for which the corresponding linear complementarity problem has a unique solution for any input vector. Testing whether or not a square matrix is a P-matrix is co-NP complete, so while it is possible to exhibit polynomially-sized witnesses for the fact that a matrix is not a P-matrix, it is believed that there is no efficient way to prove that a given matrix is a P-matrix. We will show that several well known witnesses for the fact that a matrix is not a P-matrix are computationally equivalent, so that we are able to convert between them in polynomial time, answering a question raised in arXiv:1811.03841 .

cs.DM↗

arXiv2021

Approximate PSD-Completion for Generalized Chordal Graphs

Recently, there has been interest in the question of whether a partial matrix in which many of the fully defined principal submatrices are PSD is approximately PSD completable. These questions are related to graph theory because we can think of the entries of a symmetric matrix as corresponding to the edges of a graph. We first introduce a family of graphs, which we call thickened graphs; these contain both triangle-free and chordal graphs, and can be viewed as the result of replacing the edges of a graph by an arbitrary chordal graph. We believe these graphs might be of independent interest. We then show that for a class of graphs including thickened graphs, it is possible to get quantitative bounds on how well the property of having these principal submatrices being PSD approximates the PSD-completability property. These bounds frequently only depend on the size of the smallest cycle of size at least 4 in the graph. We introduce some tools that allow us to better control the quality of these approximations and indicate how these approximations can be used to improve the performance of semidefinite programs. The tools we use in this paper are an interesting mix of algebraic topology, structural graph theory, and spectral analysis.

math.OC↗

arXiv2021

Extreme Nonnegative Quadratics over Stanley Reisner Varieties

We consider the convex geometry of the cone of nonnegative quadratics over Stanley-Reisner varieties. Stanley-Reisner varieties (which are unions of coordinate planes) are amongst the simplest real projective varieties, so this is potentially a starting point that can generalize to more complicated real projective varieties. This subject has some suprising connections to algebraic topology and category theory, which we exploit heavily in our work. These questions are also valuable in applied math, because they directly translate to questions about positive semidefinite (PSD) matrices. In particular, this relates to a long line of work concerning the extent to which it is possible to approximately check that a matrix is PSD by checking that some principle submatrices are PSD, or to check if a partial matrix can be approximately completed to full PSD matrix. We systematize both these practical and theoretical questions using a framework based on algebraic topology, category theory, and convex geometry. As applications of this framework we are able to classify the extreme nonnegative quadratics over many Stanley-Reisner varieties. We plan to follow these structural results with a paper that is more focused on quantitative questions about PSD matrix completion, which have applications in sparse semidefinite programming.

math.AG↗

arXiv2021

Causal Channels

We consider causal models with two observed variables and one latent variables, each variable being discrete, with the goal of characterizing the possible distributions on outcomes that can result from controlling one of the observed variables. We optimize linear functions over the space of all possible interventional distributions, which allows us find properties of the interventional distribution even when we cannot uniquely identify what it is. We show that, under certain mild assumptions about the correlation between controlled variable and the latent variable, the resulting interventional distribution must be close to the observed conditional distribution in a quantitative sense. Specifically, we show that if the observed variables are sufficiently highly correlated, and the latent variable can only take on a small number of distinct values, then the variables will remain causally related after passing to the interventional distribution. Another result, possibly of more general interest, is a bound on the distance between the interventional distribution and the observed conditional distribution in terms of the mutual information between the controlled variable and the latent variable, which shows that the controlled variable and the latent variable must be tightly correlated for the interventional distribution to differ significantly from the observed distribution. We believe that this type of result may make it possible to rigorously consider 'weak' experiments, where the causal variable is not entirely independent from the environment, but only approximately so. More generally, we suggest a connection between the theory of causality to polynomial optimization, which give useful bounds on the space of interventional distributions.

cs.IT↗

arXiv2020

Hyperbolic Relaxation of $k$-Locally Positive Semidefinite Matrices

A successful computational approach for solving large-scale positive semidefinite (PSD) programs is to enforce PSD-ness on only a collection of submatrices. For our study, we let $\mathcal{S}^{n,k}$ be the convex cone of $n\times n$ symmetric matrices where all $k\times k$ principal submatrices are PSD. We call a matrix in this $k$-\emph{locally PSD}. In order to compare $S^{n,k}$ to the of PSD matrices, we study eigenvalues of $k$-{locally PSD} matrices. The key insight in this paper is that there is a convex cone $H(e_k^n)$ so that if $X \in \mathcal{S}^{n,k}$, then the vector of eigenvalues of $X$ is contained in $H(e_k^n)$. The cone $H(e_k^n)$ is the hyperbolicity cone of the elementary symmetric polynomial $e^k_n$ (where $e_k^n(x) = \sum_{S \subseteq [n] : |S| = k} \prod_{i \in S} x_i$) with respect to the all ones vector. Using this insight, we are able to improve previously known upper bounds on the Frobenius distance between matrices in $\mathcal{S}^{n,k}$ and PSD matrices. We also study the quality of the convex relaxation $H(e^n_k)$. We first show that this relaxation is tight for the case of $k = n -1$, that is, for every vector in $H(e^n_{n -1})$ there exists a matrix in $\mathcal{S}^{n, n -1}$ whose eigenvalues are equal to the components of the vector. We then prove a structure theorem on nonsingular matrices in $\mathcal{S}^{n,k}$ all of whose $k\times k$ principal minors are zero, which we believe is of independent interest. %We then prove a structure theorem that precisely characterizes the non-singular matrices in $\mathcal{S}^{n,k}$ whose vector of eigenvalues belongs to the boundary of $H(e^n_k)$. This result shows shows that for $1< k < n -1$ "large parts" of the boundary of $H(e_k^n)$ do not intersect with the eigenvalues of matrices in $\mathcal{S}^{n,k}$.

math.OC↗

arXiv2020

Sums of Squares and Sparse Semidefinite Programming

We consider two seemingly unrelated questions: the relationship between nonnegative polynomials and sums of squares on real varieties, and sparse semidefinite programming. This connection is natural when a real variety $X$ is defined by a quadratic square-free monomial ideal. In this case nonnegative polynomials and sums of squares on $X$ are also natural objects in positive semidefinite matrix completion. Nonnegative quadratic forms over $X$ naturally correspond to partially specified matrices where all of the fully specified square blocks are PSD, and sums of squares quadratic forms naturally correspond to partially specified matrices which can be completed to a PSD matrix. We show quantitative results on approximation of nonnegative polynomials by sums of squares, which leads to applications in sparse semidefinite programming.

math.AG↗