arXiv subjects

Damek Davis

Publications and source records attributed to Damek Davis.

At least 19 recordsLinked to original sources

arXiv2026

Average Gradient Outer Product in kernel regression provably recovers the central subspace for multi-index models

We study a prototypical situation when a learned predictor can discover useful low-dimensional structure in data, while using fewer samples than are needed for accurate prediction. Specifically, we consider the problem of recovering a multi-index polynomial $f^*(x)=h(Ux)$, with $U\in\mathbb{R}^{r\times d}$ and $r\ll d$, from finitely many data/label pairs. Importantly, the target function depends on input $x$ only through the projection onto an unknown $r$-dimensional central subspace. The algorithm we analyze is appealingly simple: fit kernel ridge regression (KRR) to the data and compute the Average Gradient Outer Product (AGOP) from the fitted predictor. Our main results show that under reasonable assumptions the top $r$-dimensional eigenspace of AGOP provably recovers the central subspace, even in regimes when the prediction error remains large. Specifically, if the target function $f^*$ has degree $p^*$, it is known that $n\asymp d^{p^*}$ samples are necessary for KRR to achieve accurate prediction. In contrast, we show that if a low degree $p$ component of $f^*$ already carries all relevant directions for prediction, subspace recovery occurs in the much lower sample regime $n\asymp d^{p+\delta}$ for any $\delta\in(0,1)$. Our results thus demonstrate a separation between prediction and representation, and provide an explanation for why iterative kernel methods such as Recursive Feature Machines (RFM) can be sample-efficient in practice.

stat.ML↗

arXiv2026

Forbidden subgraphs in divisor graphs and an Erd\H{o}s divisibility problem

Erd\H{o}s asked for the largest size $f(n)$ of a subset of $\{1,\dots,n\}$ with no element dividing two others. We show that $f(n)=c_2\,n+o(n)$ for an effectively computable constant $c_2$, and moreover that the number $q(n)$ of such subsets satisfies $q(n)=\beta_2^{n+o(n)}$ for a computable constant $\beta_2$. To prove this, we recast the divisibility constraint as forbidding a certain directed subgraph in the divisor graph on $\{1,\dots,n\}$ and prove a more general result: for any finite family of connected forbidden subgraphs of the divisor graph, both the extremal density and counting rate are effectively computable. The proof uses a theorem of McNew on local statistics of divisor graphs.

math.CO↗

arXiv2026

A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity

Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically away from their minimizers. The argument is intricate, relying on monitoring the performance of the algorithm relative to a certain manifold of slow growth -- called the ravine. In this work, we provide a direct Lyapunov-based argument that bypasses these difficulties when the objective is in addition convex and a has a unique minimizer. As a byproduct of the argument, we obtain a more adaptive variant than the original algorithm with encouraging numerical performance.

math.OC↗

arXiv2026

The sharp one-dimensional convex sub-Gaussian comparison constant

Let $X$ be an integrable real random variable with mean zero and two-sided sub-Gaussian tail $\mathbb{P}(|X|>t)\le 2e^{-t^{2}/2}$ for all $t\ge 0$. We determine the smallest constant $c_\star$ such that $X$ is dominated in convex order by $c_\star G$, where $G$ is standard normal. Equivalently, $c_\star^2$ is the sharp one-dimensional convex sub-Gaussian comparison constant appearing in the \emph{Optimization Constants in Mathematics} repository~\cite{optimization-constants-repo}. We show that $c_\star$ is given by an explicit system of one-dimensional equations and is attained by an extremal distribution that saturates the tail constraint. Numerically, $c_\star \approx 2.30952$ (so $c_\star^2 \approx 5.33386$). We also determine the analogous sharp constant under a two-sided sub-exponential tail bound, with convex domination by a scaled Laplace law. Finally, we record two higher-dimensional consequences: a sequential tensorization principle for multivariate convex domination, and a dimension-free Gaussian comparator for the cone generated by convex ridge functions (the linear convex order).

math.PR↗

arXiv2026

Counting partial Hadamard matrices in the cubic regime

We give a precise asymptotic formula for the number of $n\times 4t$ partial Hadamard matrices in the regimes $t/n^3\to\infty$ and $t/n^3\to\Theta$ for sufficiently large fixed $\Theta$. This strengthens earlier results of de~Launey and Levin, who obtained the asymptotic for $t/n^{12}\to\infty$, and of Canfield, who extended this to $t/n^4\to\infty$.

math.PR↗

arXiv2025

When do spectral gradient updates help in deep learning?

Spectral gradient methods, such as the recently popularized Muon optimizer, are a promising alternative to standard Euclidean gradient descent for training deep neural networks and transformers, but it is still unclear in which regimes they are expected to perform better. We propose a simple layerwise condition that predicts when a spectral update yields a larger decrease in the loss than a Euclidean gradient step. This condition compares, for each parameter block, the squared nuclear-to-Frobenius ratio of the gradient to the stable rank of the incoming activations. To understand when this condition may be satisfied, we first prove that post-activation matrices have low stable rank at Gaussian initialization in random feature regression, feedforward networks, and transformer blocks. In spiked random feature models we then show that, after a short burn-in, the Euclidean gradient's nuclear-to-Frobenius ratio grows with the data dimension while the stable rank of the activations remains bounded, so the predicted advantage of spectral updates scales with dimension. We validate these predictions in synthetic regression experiments and in NanoGPT-scale language model training, where we find that intermediate activations have low-stable-rank throughout training and the corresponding gradients maintain large nuclear-to-Frobenius ratios. Together, these results identify conditions for spectral gradient methods, such as Muon, to be effective in training deep networks and transformers.

cs.LG↗

arXiv2025

What is the objective of reasoning with reinforcement learning?

We show that several popular algorithms for reinforcement learning in large language models with binary rewards can be viewed as stochastic gradient ascent on a monotone transform of the probability of a correct answer given a prompt. In particular, the transformation associated with rejection sampling algorithms is the logarithm and that associated with the GRPO algorithm is the arcsine of the square root.

cs.LG↗

arXiv2025

Iteratively reweighted kernel machines efficiently learn sparse functions

The impressive practical performance of neural networks is often attributed to their ability to learn low-dimensional data representations and hierarchical structure directly from data. In this work, we argue that these two phenomena are not unique to neural networks, and can be elicited from classical kernel methods. Namely, we show that the derivative of the kernel predictor can detect the influential coordinates with low sample complexity. Moreover, by iteratively using the derivatives to reweight the data and retrain kernel machines, one is able to efficiently learn hierarchical polynomials with finite leap complexity. Numerical experiments illustrate the developed theory.

stat.ML↗

arXiv2025

Spectral norm bound for the product of random Fourier-Walsh matrices

We consider matrix products of the form $A_1(A_2A_2)^\top\ldots(A_{m}A_{m}^\top)A_{m+1}$, where $A_i$ are normalized random Fourier-Walsh matrices. We identify an interesting polynomial scaling regime when the operator norm of the expected matrix product tends to zero as the dimension tends to infinity.

math.PR↗

arXiv2025

Online Covariance Estimation in Nonsmooth Stochastic Approximation

We consider applying stochastic approximation (SA) methods to solve nonsmooth variational inclusion problems. Existing studies have shown that the averaged iterates of SA methods exhibit asymptotic normality, with an optimal limiting covariance matrix in the local minimax sense of H\'ajek and Le Cam. However, no methods have been proposed to estimate this covariance matrix in a nonsmooth and potentially non-monotone (nonconvex) setting. In this paper, we study an online batch-means covariance matrix estimator introduced in Zhu et al.(2023). The estimator groups the SA iterates appropriately and computes the sample covariance among batches as an estimate of the limiting covariance. Its construction does not require prior knowledge of the total sample size, and updates can be performed recursively as new data arrives. We establish that, as long as the batch size sequence is properly specified (depending on the stepsize sequence), the estimator achieves a convergence rate of order $O(\sqrt{d}n^{-1/8+\varepsilon})$ for any $\varepsilon>0$, where $d$ and $n$ denote the problem dimensionality and the number of iterations (or samples) used. Although the problem is nonsmooth and potentially non-monotone (nonconvex), our convergence rate matches the best-known rate for covariance estimation methods using only first-order information in smooth and strongly-convex settings. The consistency of this covariance estimator enables asymptotically valid statistical inference, including constructing confidence intervals and performing hypothesis testing.

stat.ML↗

arXiv2024

Gradient descent with adaptive stepsize converges (nearly) linearly under fourth-order growth

A prevalent belief among optimization specialists is that linear convergence of gradient descent is contingent on the function growing quadratically away from its minimizers. In this work, we argue that this belief is inaccurate. We show that gradient descent with an adaptive stepsize converges at a local (nearly) linear rate on any smooth function that merely exhibits fourth-order growth away from its minimizer. The adaptive stepsize we propose arises from an intriguing decomposition theorem: any such function admits a smooth manifold around the optimal solution -- which we call the ravine -- so that the function grows at least quadratically away from the ravine and has constant order growth along it. The ravine allows one to interlace many short gradient steps with a single long Polyak gradient step, which together ensure rapid convergence to the minimizer. We illustrate the theory and algorithm on the problems of matrix sensing and factorization and learning a single neuron in the overparameterized regime.

math.OC↗

arXiv2023

A nearly linearly convergent first-order method for nonsmooth functions with quadratic growth

Classical results show that gradient descent converges linearly to minimizers of smooth strongly convex functions. A natural question is whether there exists a locally nearly linearly convergent method for nonsmooth functions with quadratic growth. This work designs such a method for a wide class of nonsmooth and nonconvex locally Lipschitz functions, including max-of-smooth, Shapiro's decomposable class, and generic semialgebraic functions. The algorithm is parameter-free and derives from Goldstein's conceptual subgradient method.

math.OC↗

arXiv2023

Computational Microscopy beyond Perfect Lenses

We demonstrate that in situ coherent diffractive imaging (CDI), which harnesses the coherent interference between a strong and a weak beam illuminating a static and dynamic structure, can be a very dose-efficient imaging method. At low doses, in situ CDI can achieve higher resolution than perfect lenses with the point spread function as a delta function. Both our numerical simulation and experimental results show that the combination of in situ CDI and ptychography can reduce the dose by two orders of magnitude over ptychography. We expect that computational microscopy based on in situ CDI can be implemented in different imaging modalities with photons and electrons for low-dose imaging of radiation-sensitive materials and biological samples.

physics.optics↗

arXiv2023

Aiming towards the minimizers: fast convergence of SGD for overparametrized problems

Modern machine learning paradigms, such as deep learning, occur in or close to the interpolation regime, wherein the number of model parameters is much larger than the number of data samples. In this work, we propose a regularity condition within the interpolation regime which endows the stochastic gradient method with the same worst-case iteration complexity as the deterministic gradient method, while using only a single sampled gradient (or a minibatch) in each iteration. In contrast, all existing guarantees require the stochastic gradient method to take small steps, thereby resulting in a much slower linear rate of convergence. Finally, we demonstrate that our condition holds when training sufficiently wide feedforward neural networks with a linear output layer.

cs.LG↗

arXiv2023

Asymptotic normality and optimality in nonsmooth stochastic approximation

In their seminal work, Polyak and Juditsky showed that stochastic approximation algorithms for solving smooth equations enjoy a central limit theorem. Moreover, it has since been argued that the asymptotic covariance of the method is best possible among any estimation procedure in a local minimax sense of Hájek and Le Cam. A long-standing open question in this line of work is whether similar guarantees hold for important non-smooth problems, such as stochastic nonlinear programming or stochastic variational inequalities. In this work, we show that this is indeed the case.

math.OC↗

arXiv2023

Active manifolds, stratifications, and convergence to local minima in nonsmooth optimization

We show that the subgradient method converges only to local minimizers when applied to generic Lipschitz continuous and subdifferentially regular functions that are definable in an o-minimal structure. At a high level, the argument we present is appealingly transparent: we interpret the nonsmooth dynamics as an approximate Riemannian gradient method on a certain distinguished submanifold that captures the nonsmooth activity of the function. In the process, we develop new regularity conditions in nonsmooth analysis that parallel the stratification conditions of Whitney, Kuo, and Verdier and extend stochastic processes techniques of Pemantle.

math.OC↗

arXiv2022

A linearly convergent Gauss-Newton subgradient method for ill-conditioned problems

We analyze a preconditioned subgradient method for optimizing composite functions $h \circ c$, where $h$ is a locally Lipschitz function and $c$ is a smooth nonlinear mapping. We prove that when $c$ satisfies a constant rank property and $h$ is semismooth and sharp on the image of $c$, the method converges linearly. In contrast to standard subgradient methods, its oracle complexity is invariant under reparameterizations of $c$.

math.OC↗

arXiv2022

A gradient sampling method with complexity guarantees for Lipschitz functions in high and low dimensions

Zhang et al. introduced a novel modification of Goldstein's classical subgradient method, with an efficiency guarantee of $O(\varepsilon^{-4})$ for minimizing Lipschitz functions. Their work, however, makes use of a nonstandard subgradient oracle model and requires the function to be directionally differentiable. In this paper, we show that both of these assumptions can be dropped by simply adding a small random perturbation in each step of their algorithm. The resulting method works on any Lipschitz function whose value and gradient can be evaluated at points of differentiability. We additionally present a new cutting plane algorithm that achieves better efficiency in low dimensions: $O(d\varepsilon^{-3})$ for Lipschitz functions and $O(d\varepsilon^{-2})$ for those that are weakly convex.

math.OC↗