Searcharxiv⌕ Search

arXiv subjects

Lechao Xiao

Publications and source records attributed to Lechao Xiao.

29 records · Page 2Linked to original sources
arXiv2020

Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks

The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes. In this work, we analyze the effect of initialization in deep linear networks, and provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights. We show that for deep networks, the width needed for efficient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence with Gaussian initializations scales linearly in the depth. Our results demonstrate how the benefits of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry.

cs.LG↗
arXiv2019

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks have made a theory of learning dynamics elusive. In this work, we show that for wide neural networks the learning dynamics simplify considerably and that, in the infinite width limit, they are governed by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters. Furthermore, mirroring the correspondence between wide Bayesian neural networks and Gaussian processes, gradient-based training of wide neural networks with a squared loss produces test set predictions drawn from a Gaussian process with a particular compositional kernel. While these theoretical results are only exact in the infinite width limit, we nevertheless find excellent empirical agreement between the predictions of the original network and those of the linearized version even for finite practically-sized networks. This agreement is robust across different architectures, optimization methods, and loss functions.

stat.ML↗
arXiv2019

Neural Tangents: Fast and Easy Infinite Neural Networks in Python

Neural Tangents is a library designed to enable research into infinite-width neural networks. It provides a high-level API for specifying complex and hierarchical neural network architectures. These networks can then be trained and evaluated either at finite-width as usual or in their infinite-width limit. Infinite-width networks can be trained analytically using exact Bayesian inference or using gradient descent via the Neural Tangent Kernel. Additionally, Neural Tangents provides tools to study gradient descent training dynamics of wide but finite networks in either function space or weight space. The entire library runs out-of-the-box on CPU, GPU, or TPU. All computations can be automatically distributed over multiple accelerators with near-linear scaling in the number of devices. Neural Tangents is available at www.github.com/google/neural-tangents. We also provide an accompanying interactive Colab notebook.

stat.ML↗
arXiv2019

Oscillatory Loomis-Whitney and Projections of Sublevel Sets

We consider an oscillatory integral operator with Loomis-Whitney multilinear form. The phase is real analytic in a neighborhood of the origin in $\mathbb{R}^d$ and satisfies a nondegeneracy condition related to its Newton polyhedron. Maximal decay is obtained for this operator in certain cases, depending on the Newton polyhedron of the phase and the given Lebesgue exponents. Our estimates imply volumes of sublevel sets of such real analytic functions are small relative to the product of areas of projections onto coordinate hyperplanes.

math.CA↗
arXiv2018

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks

In recent years, state-of-the-art methods in computer vision have utilized increasingly deep convolutional neural network architectures (CNNs), with some of the most successful models employing hundreds or even thousands of layers. A variety of pathologies such as vanishing/exploding gradients make training such deep networks challenging. While residual connections and batch normalization do enable training at these depths, it has remained unclear whether such specialized architecture designs are truly necessary to train deep CNNs. In this work, we demonstrate that it is possible to train vanilla CNNs with ten thousand layers or more simply by using an appropriate initialization scheme. We derive this initialization scheme theoretically by developing a mean field theory for signal propagation and by characterizing the conditions for dynamical isometry, the equilibration of singular values of the input-output Jacobian matrix. These conditions require that the convolution operator be an orthogonal transformation in the sense that it is norm-preserving. We present an algorithm for generating such random initial orthogonal convolution kernels and demonstrate empirically that they enable efficient training of extremely deep architectures.

stat.ML↗
arXiv2016

Higher decay inequalities for multilinear oscillatory integrals

In this paper we establish sharp estimates (up to logarithmic losses) for the multilinear oscillatory integral operator studied by Phong, Stein, and Sturm and Carbery and Wright on any product $\prod_{j=1}^d L^{p_j}(\mathbb R)$ with each $p_j \geq 2$, expanding the known results for this operator well outside the previous range $\sum_{j=1}^d p_j^{-1} = d-1$. Our theorem assumes second-order nondegeneracy condition of Varchenko type, and as a corollary reproduces Varchenko's theorem and implies Fourier decay estimates for measures of smooth density on degenerate hypersurfaces in $\mathbb R^d$.

math.CA↗
arXiv2016

Endpoint estimates for one-dimensional oscillatory integral operator

The one-dimensional oscillatory integral operator associated to a real analytic phase $S$ is given by $$ T_λf(x) =\int_{-\infty}^\infty e^{iλS(x,y)} χ(x,y) f(y) dy. $$ In this paper, we obtain a complete characterization for the mapping properties of $T_λ$ on $L^p(\mathbb R)$ spaces, namely we prove that $\|T_λ\|_p \lesssim |λ|^{-α}\|f\|_p$ for some $α>0$ if and only if the point $(\frac 1 {αp} , \frac 1 {αp'})$ lies in the reduced Newton polygon of $S$, and this estimate is sharp if and only if it lies on the reduced Newton diagram.

math.CA↗
arXiv2016

Sharp estimates for trilinear oscillatory integrals and an algorithm of two-dimensional resolution of singularities

We obtain sharp estimates for certain trilinear oscillatory integrals. In particular, we extend Phong and Stein's seminal result to a trilinear setting. This result partially answers a question raised by Christ, Li, Tao and Thiele concerning the sharp estimates for certain multilinear oscillatory integrals. The method in this paper relies on a self-contained algorithm of resolution of singularities in $\mathbb R^2$, which may be of independent interest.

math.CA↗
arXiv2015

Maximal Decay Inequalities for Trilinear oscillatory integrals of convolution type

In this paper we prove sharp $L^\infty$-$L^\infty$-$L^\infty$ decay for certain trilinear oscillatory integral forms of convolution type on $\mathbb R^2$. These estimates imply earlier $L^2$-$L^2$-$L^2$ results obtained by the second author as well as corresponding sharp, stable sublevel set estimates of the form studied by Christ and Christ, Li, Tao, and Thiele. New connections to the multilinear results of Phong, Stein, and Sturm are also considered.

math.CA↗