arXiv subjects

Jeffrey Pennington

Publications and source records attributed to Jeffrey Pennington.

46 records · Page 3Linked to original sources

arXiv2018

Estimating the Spectral Density of Large Implicit Matrices

Many important problems are characterized by the eigenvalues of a large matrix. For example, the difficulty of many optimization problems, such as those arising from the fitting of large models in statistics and machine learning, can be investigated via the spectrum of the Hessian of the empirical loss function. Network data can be understood via the eigenstructure of a graph Laplacian matrix using spectral graph theory. Quantum simulations and other many-body problems are often characterized via the eigenvalues of the solution space, as are various dynamic systems. However, naive eigenvalue estimation is computationally expensive even when the matrix can be represented; in many of these situations the matrix is so large as to only be available implicitly via products with vectors. Even worse, one may only have noisy estimates of such matrix vector products. In this work, we combine several different techniques for randomized estimation and show that it is possible to construct unbiased estimators to answer a broad class of questions about the spectra of such implicit matrices, even in the presence of noise. We validate these methods on large-scale problems in which graph theory and random matrix theory provide ground truth.

stat.ML↗

arXiv2017

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is $O(1)$ is essential for avoiding the exponential vanishing or explosion of gradients. The stronger condition that all singular values of the Jacobian concentrate near $1$ is a property known as dynamical isometry. For deep linear networks, dynamical isometry can be achieved through orthogonal weight initialization and has been shown to dramatically speed up learning; however, it has remained unclear how to extend these results to the nonlinear setting. We address this question by employing powerful tools from free probability theory to compute analytically the entire singular value distribution of a deep network's input-output Jacobian. We explore the dependence of the singular value distribution on the depth of the network, the weight initialization, and the choice of nonlinearity. Intriguingly, we find that ReLU networks are incapable of dynamical isometry. On the other hand, sigmoidal networks can achieve isometry, but only with orthogonal weight initialization. Moreover, we demonstrate empirically that deep nonlinear networks achieving dynamical isometry learn orders of magnitude faster than networks that do not. Indeed, we show that properly-initialized deep sigmoidal networks consistently outperform deep ReLU networks. Overall, our analysis reveals that controlling the entire distribution of Jacobian singular values is an important design consideration in deep learning.

cs.LG↗

arXiv2017

A Correspondence Between Random Neural Networks and Statistical Field Theory

A number of recent papers have provided evidence that practical design questions about neural networks may be tackled theoretically by studying the behavior of random networks. However, until now the tools available for analyzing random neural networks have been relatively ad-hoc. In this work, we show that the distribution of pre-activations in random neural networks can be exactly mapped onto lattice models in statistical physics. We argue that several previous investigations of stochastic networks actually studied a particular factorial approximation to the full lattice model. For random linear networks and random rectified linear networks we show that the corresponding lattice models in the wide network limit may be systematically approximated by a Gaussian distribution with covariance between the layers of the network. In each case, the approximate distribution can be diagonalized by Fourier transformation. We show that this approximation accurately describes the results of numerical simulations of wide random neural networks. Finally, we demonstrate that in each case the large scale behavior of the random networks can be approximated by an effective field theory.

stat.ML↗

arXiv2014

Bootstrapping six-gluon scattering in planar ${\cal N}=4$ super-Yang-Mills theory

We describe the hexagon function bootstrap for solving for six-gluon scattering amplitudes in the large $N_c$ limit of ${\cal N}=4$ super-Yang-Mills theory. In this method, an ansatz for the finite part of these amplitudes is constrained at the level of amplitudes, not integrands, using boundary information. In the near-collinear limit, the dual picture of the amplitudes as Wilson loops leads to an operator product expansion which has been solved using integrability by Basso, Sever and Vieira. Factorization of the amplitudes in the multi-Regge limit provides additional boundary data. This bootstrap has been applied successfully through four loops for the maximally helicity violating (MHV) configuration of gluon helicities, and through three loops for the non-MHV case.

hep-th↗

arXiv2014

The four-loop remainder function and multi-Regge behavior at NNLLA in planar N=4 super-Yang-Mills theory

We present the four-loop remainder function for six-gluon scattering with maximal helicity violation in planar N=4 super-Yang-Mills theory, as an analytic function of three dual-conformal cross ratios. The function is constructed entirely from its analytic properties, without ever inspecting any multi-loop integrand. We employ the same approach used at three loops, writing an ansatz in terms of hexagon functions, and fixing coefficients in the ansatz using the multi-Regge limit and the operator product expansion in the near-collinear limit. We express the result in terms of multiple polylogarithms, and in terms of the coproduct for the associated Hopf algebra. From the remainder function, we extract the BFKL eigenvalue at next-to-next-to-leading logarithmic accuracy (NNLLA), and the impact factor at NNNLLA. We plot the remainder function along various lines and on one surface, studying ratios of successive loop orders. As seen previously through three loops, these ratios are surprisingly constant over large regions in the space of cross ratios, and they are not far from the value expected at asymptotically large orders of perturbation theory.

hep-th↗

arXiv2014

The BFKL equation, Mueller-Navelet jets and single-valued harmonic polylogarithms

We introduce a generating function for the coefficients of the leading logarithmic BFKL Green's function in transverse-momentum space, order by order in alpha_s, in terms of single-valued harmonic polylogarithms. As an application, we exhibit fully analytic azimuthal-angle and transverse-momentum distributions for Mueller-Navelet jet cross sections at each order in alpha_s. We also provide a generating function for the total cross section valid to any number of loops.

hep-ph↗

arXiv2013

Hexagon functions and the three-loop remainder function

We present the three-loop remainder function, which describes the scattering of six gluons in the maximally-helicity-violating configuration in planar N=4 super-Yang-Mills theory, as a function of the three dual conformal cross ratios. The result can be expressed in terms of multiple Goncharov polylogarithms. We also employ a more restricted class of "hexagon functions" which have the correct branch cuts and certain other restrictions on their symbols. We classify all the hexagon functions through transcendental weight five, using the coproduct for their Hopf algebra iteratively, which amounts to a set of first-order differential equations. The three-loop remainder function is a particular weight-six hexagon function, whose symbol was determined previously. The differential equations can be integrated numerically for generic values of the cross ratios, or analytically in certain kinematics limits, including the near-collinear and multi-Regge limits. These limits allow us to impose constraints from the operator product expansion and multi-Regge factorization directly at the function level, and thereby to fix uniquely a set of Riemann-zeta-valued constants that could not be fixed at the level of the symbol. The near-collinear limits agree precisely with recent predictions by Basso, Sever and Vieira based on integrability. The multi-Regge limits agree with the factorization formula of Fadin and Lipatov, and determine three constants entering the impact factor at this order. We plot the three-loop remainder function for various slices of the Euclidean region of positive cross ratios, and compare it to the two-loop one. For large ranges of the cross ratios, the ratio of the three-loop to the two-loop remainder function is relatively constant, and close to -7.

hep-th↗

arXiv2013

Leading singularities and off-shell conformal integrals

The three-loop four-point function of stress-tensor multiplets in N=4 super Yang-Mills theory contains two so far unknown, off-shell, conformal integrals, in addition to the known, ladder-type integrals. In this paper we evaluate the unknown integrals, thus obtaining the three-loop correlation function analytically. The integrals have the generic structure of rational functions multiplied by (multiple) polylogarithms. We use the idea of leading singularities to obtain the rational coefficients, the symbol - with an appropriate ansatz for its structure - as a means of characterising multiple polylogarithms, and the technique of asymptotic expansion of Feynman integrals to obtain the integrals in certain limits. The limiting behaviour uniquely fixes the symbols of the integrals, which we then lift to find the corresponding polylogarithmic functions. The final formulae are numerically confirmed. The techniques we develop can be applied more generally, and we illustrate this by analytically evaluating one of the integrals contributing to the same four-point function at four loops. This example shows a connection between the leading singularities and the entries of the symbol.

hep-th↗

arXiv2012

Single-valued harmonic polylogarithms and the multi-Regge limit

We argue that the natural functions for describing the multi-Regge limit of six-gluon scattering in planar N=4 super Yang-Mills theory are the single-valued harmonic polylogarithmic functions introduced by Brown. These functions depend on a single complex variable and its conjugate, (w,w*). Using these functions, and formulas due to Fadin, Lipatov and Prygarin, we determine the six-gluon MHV remainder function in the leading-logarithmic approximation (LLA) in this limit through ten loops, and the next-to-LLA (NLLA) terms through nine loops. In separate work, we have determined the symbol of the four-loop remainder function for general kinematics, up to 113 constants. Taking its multi-Regge limit and matching to our four-loop LLA and NLLA results, we fix all but one of the constants that survive in this limit. The multi-Regge limit factorizes in the variables (ν,n) which are related to (w,w*) by a Fourier-Mellin transform. We can transform the single-valued harmonic polylogarithms to functions of (ν,n) that incorporate harmonic sums, systematically through transcendental weight six. Combining this information with the four-loop results, we determine the eigenvalues of the BFKL kernel in the adjoint representation to NNLLA accuracy, and the MHV product of impact factors to NNNLLA accuracy, up to constants representing beyond-the-symbol terms and the one symbol-level constant. Remarkably, only derivatives of the polygamma function enter these results. Finally, the LLA approximation to the six-gluon NMHV amplitude is evaluated through ten loops.

hep-th↗

arXiv2012

The six-point remainder function to all loop orders in the multi-Regge limit

We present an all-orders formula for the six-point amplitude of planar maximally supersymmetric N=4 Yang-Mills theory in the leading-logarithmic approximation of multi-Regge kinematics. In the MHV helicity configuration, our results agree with an integral formula of Lipatov and Prygarin through at least 14 loops. A differential equation linking the MHV and NMHV helicity configurations has a natural action in the space of functions relevant to this problem---the single-valued harmonic polylogarithms introduced by Brown. These functions depend on a single complex variable and its conjugate, w and w*, which are quadratically related to the original kinematic variables. We investigate the all-orders formula in the near-collinear limit, which is approached as |w|->0. Up to power-suppressed terms, the resulting expansion may be organized by powers of log|w|. The leading term of this expansion agrees with the all-orders double-leading-logarithmic approximation of Bartels, Lipatov, and Prygarin. The explicit form for the sub-leading powers of log|w| is given in terms of modified Bessel functions.

hep-th↗