arXiv subjects
Publications and source records attributed to Elliot Paquette.
We consider the extremes of the logarithm of the characteristic polynomial of matrices from the C$β$E ensemble. We prove convergence in distribution of the centered maxima (of the real and imaginary parts) towards the sum of a Gumbel variable and another independent variable, which we characterize as the total mass of a "derivative martingale". We also provide a description of the landscape near extrema points.
The success of modern machine learning is due in part to the adaptive optimization methods that have been developed to deal with the difficulties of training large models over complex datasets. One such method is gradient clipping: a practical procedure with limited theoretical underpinnings. In this work, we study clipping in a least squares problem under streaming SGD. We develop a theoretical analysis of the learning dynamics in the limit of large intrinsic dimension-a model and dataset dependent notion of dimensionality. In this limit we find a deterministic equation that describes the evolution of the loss and demonstrate that this equation predicts the path of clipped SGD on synthetic, CIFAR10, and Wikitext2 data. We show that with Gaussian noise clipping cannot improve SGD performance. Yet, in other noisy settings, clipping can provide benefits with tuning of the clipping threshold. We propose a simple heuristic for near optimal scheduling of the clipping threshold which requires the tuning of only one hyperparameter. We conclude with a discussion about the links between high-dimensional clipping and neural network training.
We consider the problem $(\mathrm{P})$ of fitting $n$ standard Gaussian random vectors in $\mathbb{R}^d$ to the boundary of a centered ellipsoid, as $n, d \to \infty$. This problem is conjectured to have a sharp feasibility transition: for any $\varepsilon > 0$, if $n \leq (1 - \varepsilon) d^2 / 4$ then $(\mathrm{P})$ has a solution with high probability, while $(\mathrm{P})$ has no solutions with high probability if $n \geq (1 + \varepsilon) d^2 /4$. So far, only a trivial bound $n \geq d^2 / 2$ is known on the negative side, while the best results on the positive side assume $n \leq d^2 / \mathrm{polylog}(d)$. In this work, we improve over previous approaches using a key result of Bartl & Mendelson (2022) on the concentration of Gram matrices of random vectors under mild assumptions on their tail behavior. This allows us to give a simple proof that $(\mathrm{P})$ is feasible with high probability when $n \leq d^2 / C$, for a (possibly large) constant $C > 0$.
We consider the range of random analytic functions with finite radius of convergence. We show that any unbounded random Taylor series with rotationally invariant coefficients has dense image in the plane. We moreover show that if in addition the coefficients are complex Gaussian with sufficiently regular variances, then the image is the whole complex plane. We do this by exploiting an approximate connection between the coverage problem and spatial branching processes. This answers a long-standing open question of J.-P. Kahane, with sufficient regularity.
There has been increasing demand for establishing privacy-preserving methodologies for modern statistics and machine learning. Differential privacy, a mathematical notion from computer science, is a rising tool offering robust privacy guarantees. Recent work focuses primarily on developing differentially private versions of individual statistical and machine learning tasks, with nontrivial upstream pre-processing typically not incorporated. An important example is when record linkage is done prior to downstream modeling. Record linkage refers to the statistical task of linking two or more data sets of the same group of entities without a unique identifier. This probabilistic procedure brings additional uncertainty to the subsequent task. In this paper, we present two differentially private algorithms for linear regression with linked data. In particular, we propose a noisy gradient method and a sufficient statistics perturbation approach for the estimation of regression coefficients. We investigate the privacy-accuracy tradeoff by providing finite-sample error bounds for the estimators, which allows us to understand the relative contributions of linkage error, estimation error, and the cost of privacy. The variances of the estimators are also discussed. We demonstrate the performance of the proposed algorithms through simulations and an application to synthetic data.
We analyze the dynamics of streaming stochastic gradient descent (SGD) in the high-dimensional limit when applied to generalized linear models and multi-index models (e.g. logistic regression, phase retrieval) with general data-covariance. In particular, we demonstrate a deterministic equivalent of SGD in the form of a system of ordinary differential equations that describes a wide class of statistics, such as the risk and other measures of sub-optimality. This equivalence holds with overwhelming probability when the model parameter count grows proportionally to the number of data. This framework allows us to obtain learning rate thresholds for stability of SGD as well as convergence guarantees. In addition to the deterministic equivalent, we introduce an SDE with a simplified diffusion coefficient (homogenized SGD) which allows us to analyze the dynamics of general statistics of SGD iterates. Finally, we illustrate this theory on some standard examples and show numerical simulations which give an excellent match to the theory.
We give a description of the high-dimensional limit of one-pass single-batch stochastic gradient descent (SGD) on a least squares problem. This limit is taken with non-vanishing step-size, and with proportionally related number of samples to problem-dimensionality. The limit is described in terms of a stochastic differential equation in high dimensions, which is shown to approximate the state evolution of SGD. As a corollary, the statistical risk is shown to be approximated by the solution of a convolution-type Volterra equation with vanishing errors as dimensionality tends to infinity. The sense of convergence is the weakest that shows that statistical risks of the two processes coincide. This is distinguished from existing analyses by the type of high-dimensional limit given as well as generality of the covariance structure of the samples.
Stochastic gradient descent (SGD) is a pillar of modern machine learning, serving as the go-to optimization algorithm for a diverse array of problems. While the empirical success of SGD is often attributed to its computational efficiency and favorable generalization behavior, neither effect is well understood and disentangling them remains an open problem. Even in the simple setting of convex quadratic problems, worst-case analyses give an asymptotic convergence rate for SGD that is no better than full-batch gradient descent (GD), and the purported implicit regularization effects of SGD lack a precise explanation. In this work, we study the dynamics of multi-pass SGD on high-dimensional convex quadratics and establish an asymptotic equivalence to a stochastic differential equation, which we call homogenized stochastic gradient descent (HSGD), whose solutions we characterize explicitly in terms of a Volterra integral equation. These results yield precise formulas for the learning and risk trajectories, which reveal a mechanism of implicit conditioning that explains the efficiency of SGD relative to GD. We also prove that the noise from SGD negatively impacts generalization performance, ruling out the possibility of any type of implicit regularization in this context. Finally, we show how to adapt the HSGD formalism to include streaming SGD, which allows us to produce an exact prediction for the excess risk of multi-pass SGD relative to that of streaming SGD (bootstrap risk).
We analyze the dynamics of large batch stochastic gradient descent with momentum (SGD+M) on the least squares problem when both the number of samples and dimensions are large. In this setting, we show that the dynamics of SGD+M converge to a deterministic discrete Volterra equation as dimension increases, which we analyze. We identify a stability measurement, the implicit conditioning ratio (ICR), which regulates the ability of SGD+M to accelerate the algorithm. When the batch size exceeds this ICR, SGD+M converges linearly at a rate of $\mathcal{O}(1/\sqrtκ)$, matching optimal full-batch momentum (in particular performing as well as a full-batch but with a fraction of the size). For batch sizes smaller than the ICR, in contrast, SGD+M has rates that scale like a multiple of the single batch SGD rate. We give explicit choices for the learning rate and momentum parameter in terms of the Hessian spectra that achieve this performance.
In this paper we establish necessary and sufficient conditions for the existence of line segments (or flats) in the sphere of the nuclear norm via the notion of simultaneous polarization and a refined expression for the subdifferential of the nuclear norm. This is then leveraged to provide (point-based) necessary and sufficient conditions for uniqueness of solutions for minimizing the nuclear norm over an affine manifold. We further establish an alternative set of sufficient conditions for uniqueness, based on the interplay of the subdifferential of the nuclear norm and the range of the problem-defining linear operator. Finally, using convex duality, we show how to transfer the uniqueness results for the original problem to a whole class of nuclear norm-regularized minimization problems with a strictly convex fidelity term.
We develop a stochastic differential equation, called homogenized SGD, for analyzing the dynamics of stochastic gradient descent (SGD) on a high-dimensional random least squares problem with $\ell^2$-regularization. We show that homogenized SGD is the high-dimensional equivalence of SGD -- for any quadratic statistic (e.g., population risk with quadratic loss), the statistic under the iterates of SGD converges to the statistic under homogenized SGD when the number of samples $n$ and number of features $d$ are polynomially related ($d^c < n < d^{1/c}$ for some $c > 0$). By analyzing homogenized SGD, we provide exact non-asymptotic high-dimensional expressions for the generalization performance of SGD in terms of a solution of a Volterra integral equation. Further we provide the exact value of the limiting excess risk in the case of quadratic losses when trained by SGD. The analysis is formulated for data matrices and target vectors that satisfy a family of resolvent conditions, which can roughly be viewed as a weak (non-quantitative) form of delocalization of sample-side singular vectors of the data. Several motivating applications are provided including sample covariance matrices with independent samples and random features with non-generative model targets.
We investigate the characteristic polynomials $φ_N$ of the Gaussian $β$-ensemble for general $β>0$ through its transfer matrix recurrence. Our motivation is to obtain a (probabilistic) approximation for $φ_N$ in terms of a Gaussian log--correlated field in order to ultimately deduce some of its fine asymptotic properties. We distinguish between different types of transfer matrices and analyze completely the hyperbolic regime of the recurrence. As a result, we obtain a new coupling between $φ_N(z)$ and a Gaussian analytic function with an error which is uniform for $z \in \mathbb{C}$ separated from the support of the semicircle law. We use this as input to give the almost sure scaling limit of the characteristic polynomial at the edge in arXiv:2009.05003. This is also required to obtain analogous strong approximations inside of the bulk of the semicircle law. Our analysis relies on moderate deviation estimates for the product of transfer matrices and this approach might also be useful in different contexts.
Consider a doubly-infinite array of iid centered variables with moment conditions, from which one can extract a finite number of rectangular, overlapping submatrices, and form the corresponding Wishart matrices. We show that under basic smoothness assumptions, centered linear eigenstatistics of such matrices converge jointly to a Gaussian vector with an interesting covariance structure. This structure, which is similar to those appearing in work of Borodin, Borodin and Gorin, and Johnson and Pal can be described in terms of the height function, and leads to a connection with the Gaussian Free Field on the upper half-plane. Finally, we generalize our results from univariate polynomials to a special class of planar functions.
We analyze a class of stochastic gradient algorithms with momentum on a high-dimensional random least squares problem. Our framework, inspired by random matrix theory, provides an exact (deterministic) characterization for the sequence of loss values produced by these algorithms which is expressed only in terms of the eigenvalues of the Hessian. This leads to simple expressions for nearly-optimal hyperparameters, a description of the limiting neighborhood, and average-case complexity. As a consequence, we show that (small-batch) stochastic heavy-ball momentum with a fixed momentum parameter provides no actual performance improvement over SGD when step sizes are adjusted correctly. For contrast, in the non-strongly convex setting, it is possible to get a large improvement over SGD using momentum. By introducing hyperparameters that depend on the number of samples, we propose a new algorithm sDANA (stochastic dimension adjusted Nesterov acceleration) which obtains an asymptotically optimal average-case complexity while remaining linearly convergent in the strongly convex setting without adjusting parameters.
We consider the random geometric graph on $n$ vertices drawn uniformly from a $d$--dimensional sphere. We focus on the sparse regime, when the expected degree is constant independent of $d$ and $n$. We show that, when $d$ is larger than $n$ by logarithmic factors, this graph is comparable to the Erdős--Rényi random graph of the same edge density in the \emph{inclusion divergence} between the graph laws. This divergence functions in certain ways like a relaxation of the total variation distance, but is strong enough to distinguish Erdős--Rényi graphs of different densities with a higher resolution than the total variation distance. To do the analysis, we derive some exact statistics of the \emph{spherical Wishart matrix}, the Gram matrix of $n$ independent uniformly random $d$--dimensional spherical vectors. In particular we give expressions for the characteristic function of the spherical Wishart matrix which are well--approximated using steepest descent.
Average-case analysis computes the complexity of an algorithm averaged over all possible inputs. Compared to worst-case analysis, it is more representative of the typical behavior of an algorithm, but remains largely unexplored in optimization. One difficulty is that the analysis can depend on the probability distribution of the inputs to the model. However, we show that this is not the case for a class of large-scale problems trained with first-order methods including random least squares and one-hidden layer neural networks with random weights. In fact, the halting time exhibits a universality property: it is independent of the probability distribution. With this barrier for average-case analysis removed, we provide the first explicit average-case convergence rates showing a tighter complexity not captured by traditional worst-case analysis. Finally, numerical simulations suggest this universality property holds for a more general class of algorithms and problems.
We give sufficient conditions for a discrete set of points in any dimensional real hyperbolic space to have positive anchored expansion. The first condition is a bounded mean density property, ensuring not too many points can accumulate in large regions. The second is a bounded mean vacancy condition, effectively ensuring there is not too much space left vacant by the points over large regions. These properties give as an easy corollary that stationary Poisson--Delaunay graphs have positive anchored expansion, as well as Delaunay graphs built from stationary determinantal point processes. We introduce a family of stationary determinantal point processes on any dimension of real hyperbolic space, the Berezin point processes, and we partially characterize them. We pose many questions related to this process and stationary determinantal point processes.
We compute the leading asymptotics as $N\to\infty$ of the maximum of the field $Q_N(q)= \log\det|q- A_N|$, $q\in \mathbb{C}$, for any unitarily invariant Hermitian random matrix $A_N$ associated to a non-critical real-analytic potential. Hence, we verify the leading order in a conjecture of Fyodorov and Simm formulated for the GUE. The method relies on a classical upper-bound and a more sophisticated lower-bound based on a variant of the second-moment method which exploits the hyperbolic branching structure of the field $Q_N(q)$, $q$ in the upper half plane. Specifically, we compare $Q_N$ to an idealized Gaussian field by means of exponential moments. In principle, this method could also be applied to random fields coming from other point processes provided that one can compute certain mixed exponential moments. For unitarily invariant ensembles, we show that these assumptions follow from the Fyodorov-Strahov formula and asymptotics of orthogonal polynomials derived by Deift, Kriecherbauer, McLaughlin, Venakides, and Zhou.