arXiv subjects

Jiaoyang Huang

Publications and source records attributed to Jiaoyang Huang.

At least 73 records · Page 4Linked to original sources

arXiv2019

Every Local Minimum Value is the Global Minimum Value of Induced Model in Non-convex Machine Learning

For nonconvex optimization in machine learning, this article proves that every local minimum achieves the globally optimal value of the perturbable gradient basis model at any differentiable point. As a result, nonconvex machine learning is theoretically as supported as convex machine learning with a handcrafted basis in terms of the loss at differentiable local minima, except in the case when a preference is given to the handcrafted basis over the perturbable gradient basis. The proofs of these results are derived under mild assumptions. Accordingly, the proven results are directly applicable to many machine learning models, including practical deep neural networks, without any modification of practical methods. Furthermore, as special cases of our general results, this article improves or complements several state-of-the-art theoretical results on deep neural networks, deep residual networks, and overparameterized deep neural networks with a unified proof technique and novel geometric insights. A special case of our results also contributes to the theoretical foundation of representation learning.

stat.ML↗

arXiv2019

Dynamics of Deep Neural Networks and Neural Tangent Hierarchy

The evolution of a deep neural network trained by the gradient descent can be described by its neural tangent kernel (NTK) as introduced in [20], where it was proven that in the infinite width limit the NTK converges to an explicit limiting kernel and it stays constant during training. The NTK was also implicit in some other recent papers [6,13,14]. In the overparametrization regime, a fully-trained deep neural network is indeed equivalent to the kernel regression predictor using the limiting NTK. And the gradient descent achieves zero training loss for a deep overparameterized neural network. However, it was observed in [5] that there is a performance gap between the kernel regression using the limiting NTK and the deep neural networks. This performance gap is likely to originate from the change of the NTK along training due to the finite width effect. The change of the NTK along the training is central to describe the generalization features of deep neural networks. In the current paper, we study the dynamic of the NTK for finite width deep fully-connected neural networks. We derive an infinite hierarchy of ordinary differential equations, the neural tangent hierarchy (NTH) which captures the gradient descent dynamic of the deep neural network. Moreover, under certain conditions on the neural network width and the data set dimension, we prove that the truncated hierarchy of NTH approximates the dynamic of the NTK up to arbitrary precision. This description makes it possible to directly study the change of the NTK for deep neural networks, and sheds light on the observation that deep neural networks outperform kernel regressions using the corresponding limiting NTK.

cs.LG↗

arXiv2019

Eigenvalues for the Minors of Wigner Matrices

The eigenvalues for the minors of real symmetric ($β=1$) and complex Hermitian ($β=2$) Wigner matrices form the Wigner corner process, which is a multilevel interlacing particle system. In this paper, we study the microscopic scaling limit of the Wigner corner process both near the spectral edge and in the bulk, and prove they are universal. We show: (i) Near the spectral edge, the corner process exhibit a decoupling phenomenon, as first observed in [24]. Individual extreme particles have Tracy-Widom$_β$ distribution; the spacings between the extremal particles on adjacent levels converge to independent Gamma distributions in a much smaller scale. (ii) In the bulk, the microscopic scaling limit of the Wigner corner process is given by the bead process for general Sine$_β$ process, as constructed recently in [34].

math.PR↗

arXiv2019

Effect of Depth and Width on Local Minima in Deep Learning

In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. Without any simplification assumption, for deep nonlinear neural networks with the squared loss, we theoretically show that the quality of local minima tends to improve towards the global minimum value as depth and width increase. Furthermore, with a locally-induced structure on deep nonlinear neural networks, the values of local minima of neural networks are theoretically proven to be no worse than the globally optimal values of corresponding classical machine learning models. We empirically support our theoretical observation with a synthetic dataset as well as MNIST, CIFAR-10 and SVHN datasets. When compared to previous studies with strong over-parameterization assumptions, the results in this paper do not require over-parameterization, and instead show the gradual effects of over-parameterization as consequences of general results.

cs.LG↗

arXiv2018

Invertibility of adjacency matrices for random d-regular directed graphs

Let $d\geq 3$ be a fixed integer, and a prime number $p$ such that $\gcd(p,d)=1$. Let $A$ be the adjacency matrix of a random $d$-regular directed graph on $n$ vertices. We show that as a random matrix in ${\mathbb F}_p$, \begin{equation} {\mathbb P}(\text{$A$ is singular in ${\mathbb F}_p$})\leq \frac{1+{\mathrm{o}}(1)}{p-1}, \end{equation} as $n$ goes to infinity. As a consequence, as a random matrix in $\mathbb R$, \begin{equation} {\mathbb P}(\text{$A$ is singular in $\mathbb R$})={\mathrm{o}}(1) \end{equation} as $n$ goes to infinity. This answers an open problem by Frieze [12] and Vu [29,30], for random $d$-regular bipartite graphs. The proof combines a local central limit theorem and a large deviation estimate.

math.PR↗

arXiv2018

Invertibility of adjacency matrices for random $d$-regular graphs

Let $d\geq 3$ be a fixed integer and $A$ be the adjacency matrix of a random $d$-regular directed or undirected graph on $n$ vertices. We show there exist constants $\mathfrak d>0$, \begin{align*} {\mathbb P}(\text{$A$ is singular in $\mathbb R$})\leq n^{-\mathfrak{d}}, \end{align*} for $n$ sufficiently large. This answers an open problem by Frieze [12] and Vu [28,19]. The key idea is to study the singularity probability of adjacency matrices over a finite field ${\mathbb F}_p$. The proof combines a local central limit theorem and a large deviation estimate.

math.PR↗

arXiv2018

Dyson Brownian Motion for General $β$ and Potential at the Edge

In this paper, we compare the solutions of Dyson Brownian motion with general $β$ and potential $V$ and the associated McKean-Vlasov equation near the edge. Under suitable conditions on the initial data and potential $V$, we obtain the optimal rigidity estimates of particle locations near the edge for short time $t=\text{o}(1)$. Our argument uses the method of characteristics along with a careful estimate involving an equation of the edge. With the rigidity estimates as an input, we prove a central limit theorem for mesoscopic statistics near the edge which, as far as we know, have been done for the first time in this paper. Additionally, combining with \cite{LandonEdge}, our rigidity estimates are used to give a proof of the local ergodicity of Dyson Brownian motion for general $β$ and potential at the edge, i.e. the distribution of extreme particles converges to Tracy-Widom $β$ distribution in short time.

math.PR↗

arXiv2018

Law of Large Numbers and Central Limit Theorems by Jack Generating Functions

In a series of papers [22-24] by Bufetov and Gorin, Schur generating functions as the Fourier transforms on the unitary group $U(N)$, are introduced to study the asymptotic behaviors of random $N$-particle systems. We introduce and study the Jack generating functions of random $N$-particle systems. In special cases, this can be viewed as the Fourier transforms on the Gelfand pairs $(GL_N(\mathbb R), O(N))$, $(GL_N(\mathbb C), U(N))$ and $(GL_N(\mathbb H), Sp(N))$. Our main results state that the law of large numbers and the central limit theorems for such particle systems, is equivalent to certain conditions on the germ at unity of their Jack generating functions. Our main tool is the Nazarov-Sklyanin operators [50], which have Jack symmetric functions as their eigenfunctions. As applications, we derive asymptotics of Jack characters, prove law of large numbers and central limit theorems for the Littlewood-Richardson coefficients of zonal polynomials, and show that the fluctuations of the height functions of a general family of nonintersecting random walks are asymptotically equal to those of the pullback of the Gaussian free field on the upper half plane.

math.PR↗

arXiv2017

Transition from Tracy-Widom to Gaussian fluctuations of extremal eigenvalues of sparse Erdős-Rényi graphs

We consider the statistics of the extreme eigenvalues of sparse random matrices, a class of random matrices that includes the normalized adjacency matrices of the Erdős-Rényi graph $G(N,p)$. Tracy-Widom fluctuations of the extreme eigenvalues for $p\gg N^{-2/3}$ was proved in [17,46]. We prove that there is a crossover in the behavior of the extreme eigenvalues at $p\sim N^{-2/3}$. In the case that $N^{-7/9}\ll p\ll N^{-2/3}$, we prove that the extreme eigenvalues have asymptotically Gaussian fluctuations. Under a mean zero condition and when $p=CN^{-2/3}$, we find that the fluctuations of the extreme eigenvalues are given by a combination of the Gaussian and the Tracy-Widom distribution. These results show that the eigenvalues at the edge of the spectrum of sparse Erdős-Rényi graphs are less rigid than those of random $d$-regular graphs [4] of the same average degree.

math.PR↗

arXiv2017

Rigidity and Edge Universality of Discrete $β$-Ensembles

We study discrete $β$-ensembles as introduced in [17]. We obtain rigidity estimates on the particle locations, i.e. with high probability, the particles are close to their classical locations with an optimal error estimate. We prove the edge universality of the discrete $β$-ensembles, i.e. for $β\geq 1$, the distribution of extreme particles converges to the Tracy-Widom $β$ distribution. As far as we know, this is the first proof of general Tracy-Widom $β$ distributions in the discrete setting. A special case of our main results implies that under the Jack deformation of the Plancherel measure, the distribution of the lengths of the first few rows in Young diagrams, converges to the Tracy-Widom $β$ distribution, which answers an open problem in [39]. Our proof relies on Nekrasov's (or loop) equations, a multiscale analysis and a comparison argument with continuous $β$-ensembles.

math.PR↗

arXiv2017

Eigenvector Statistics of Sparse Random Matrices

We prove that the bulk eigenvectors of sparse random matrices, i.e. the adjacency matrices of Erdős-Rényi graphs or random regular graphs, are asymptotically jointly normal, provided the averaged degree increases with the size of the graphs. Our methodology follows [6] by analyzing the eigenvector flow under Dyson Brownian motion, combining with an isotropic local law for Green's function. As an auxiliary result, we prove that for the eigenvector flow of Dyson Brownian motion with general initial data, the eigenvectors are asymptotically jointly normal in the direction $q$ after time $η_*\ll t\ll r$, if in a window of size $r$, the initial density of states is bounded below and above down to the scale $η_*$, and the initial eigenvectors are delocalized in the direction $q$ down to the scale $η_*$.

math.PR↗

arXiv2017

Local law and mesoscopic fluctuations of Dyson Brownian motion for general $β$ and potential

We study Dyson Brownian motion with general potential $V$ and for general $β\geq 1$. For short times $t = o (1)$ and under suitable conditions on $V$ we obtain a local law and corresponding rigidity estimates on the particle locations; that is, with overwhelming probability, the particles are close to their classical locations with an almost-optimal error estimate. Under the condition that the density of states of the initial data is bounded below and above down to the scale $η_* \ll t \ll 1$, we prove a mesoscopic central limit theorem for linear statistics at all scales $N^{-1}\llη\ll t$

math.PR↗

arXiv2016

Local Kesten--McKay law for random regular graphs

We study the adjacency matrices of random $d$-regular graphs with large but fixed degree $d$. In the bulk of the spectrum $[-2\sqrt{d-1}+\varepsilon, 2\sqrt{d-1}-\varepsilon]$ down to the optimal spectral scale, we prove that the Green's functions can be approximated by those of certain infinite tree-like (few cycles) graphs that depend only on the local structure of the original graphs. This result implies that the Kesten--McKay law holds for the spectral density down to the smallest scale and the complete delocalization of bulk eigenvectors. Our method is based on estimating the Green's function of the adjacency matrices and a resampling of the boundary edges of large balls in the graphs.

math.PR↗

arXiv2016

Bulk eigenvalue statistics for random regular graphs

We consider the uniform random $d$-regular graph on $N$ vertices, with $d \in [N^α, N^{2/3-α}]$ for arbitrary $α> 0$. We prove that in the bulk of the spectrum the local eigenvalue correlation functions and the distribution of the gaps between consecutive eigenvalues coincide with those of the Gaussian Orthogonal Ensemble.

math.PR↗

arXiv2015

Spectral statistics of sparse Erdős-Rényi graph Laplacians

We consider the bulk eigenvalue statistics of Laplacian matrices of large Erdős-Rényi random graphs in the regime $p \geq N^δ/N$ for any fixed $δ>0$. We prove a local law down to the optimal scale $η\gtrsim N^{-1}$ which implies that the eigenvectors are delocalized. We consider the local eigenvalue statistics and prove that both the gap statistics and averaged correlation functions coincide with the GOE in the bulk.

math.PR↗

arXiv2015

Bulk universality of sparse random matrices

We consider the adjacency matrix of the ensemble of Erdős-Rényi random graphs which consists of graphs on $N$ vertices in which each edge occurs independently with probability $p$. We prove that in the regime $pN \gg 1$ these matrices exhibit bulk universality in the sense that both the averaged $n$-point correlation functions and distribution of a single eigenvalue gap coincide with those of the GOE. Our methods extend to a class of random matrices which includes sparse ensembles whose entries have different variances.

math.PR↗

arXiv2015

Mesoscopic Perturbations of Large Random Matrices

We consider the eigenvalues and eigenvectors of small rank perturbations of random $N\times N$ matrices. We allow the rank of perturbation $M$ increases with $N$, and the only assumption is $M=o(N)$. In both additive and multiplicative perturbation models, we prove rigidity results for the outliers of the perturbed random matrices. Based on the rigidity results we derive the empirical distribution of outliers of the perturbed random matrices. We also compute the appropriate projection of eigenvectors corresponding to the outliers of the perturbed random matrices, which are approximate eigenvectors of the perturbing matrix. Our results can be regarded as the extension of the finite rank perturbation case to the full generality up to $M=o(N)$.

math.PR↗

arXiv2014

Asymptotic Expansion of Spherical Integral

We consider the spherical integral of real symmetric or Hermitian matrices when the rank of one matrix is one. We prove the existence of the full asymptotic expansions of these spherical integrals and derive the first and the second term in the asymptotic expansion. Using asymptotic expression of the spherical integral, we derive the asymptotic freeness of Wigner matrices with (deterministic) Hermitian matrices.

math.PR↗