arXiv subjects

Huy Tuan Pham

Publications and source records attributed to Huy Tuan Pham.

50 records · Page 3Linked to original sources

arXiv2021

Entropic Independence II: Optimal Sampling and Concentration via Restricted Modified Log-Sobolev Inequalities

We introduce a framework for obtaining tight mixing times for Markov chains based on what we call restricted modified log-Sobolev inequalities. Modified log-Sobolev inequalities (MLSI) quantify the rate of relative entropy contraction for the Markov operator, and are notoriously difficult to establish. However, infinitesimally close to stationarity, entropy contraction becomes equivalent to variance contraction, a.k.a. a Poincare inequality, which is significantly easier to establish through, e.g., spectral analysis. Motivated by this observation, we study restricted modified log-Sobolev inequalities that guarantee entropy contraction not for all starting distributions, but for those in a large neighborhood of the stationary distribution. We show how to sample from the hardcore and Ising models on $n$-node graphs that have a constant $δ$ relative gap to the tree-uniqueness threshold, in nearly-linear time $\widetilde O_δ(n)$. Notably, our bound does not depend on the maximum degree $Δ$, and is therefore optimal even for high-degree graphs. This improves on prior mixing time bounds of $\widetilde O_{δ, Δ}(n)$ and $\widetilde O_δ(n^2)$, established via (non-restricted) modified log-Sobolev and Poincare inequalities respectively. We further show that optimal concentration inequalities can still be achieved from the restricted form of modified log-Sobolev inequalities. To establish restricted entropy contraction, we extend the entropic independence framework of Anari, Jain, Koehler, Pham, and Vuong to the setting of distributions that are spectrally independent under a restricted set of external fields. We also develop an orthogonal trick that might be of independent interest: utilizing Bernoulli factories we show how to implement Glauber dynamics updates on high-degree graphs in $O(1)$ time, assuming standard adjacency array representation of the graph.

cs.DS↗

arXiv2021

Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field training

The mean field (MF) theory of multilayer neural networks centers around a particular infinite-width scaling, where the learning dynamics is closely tracked by the MF limit. A random fluctuation around this infinite-width limit is expected from a large-width expansion to the next order. This fluctuation has been studied only in shallow networks, where previous works employ heavily technical notions or additional formulation ideas amenable only to that case. Treatment of the multilayer case has been missing, with the chief difficulty in finding a formulation that captures the stochastic dependency across not only time but also depth. In this work, we initiate the study of the fluctuation in the case of multilayer networks, at any network depth. Leveraging on the neuronal embedding framework recently introduced by Nguyen and Pham, we systematically derive a system of dynamical equations, called the second-order MF limit, that captures the limiting fluctuation distribution. We demonstrate through the framework the complex interaction among neurons in this second-order MF limit, the stochasticity with cross-layer dependency and the nonlinear time evolution inherent in the limiting fluctuation. A limit theorem is proven to relate quantitatively this limit to the fluctuation of large-width networks. We apply the result to show a stability property of gradient descent MF training: in the large-width regime, along the training trajectory, it progressively biases towards a solution with "minimal fluctuation" (in fact, vanishing fluctuation) in the learned output function, even after the network has been initialized at or has converged (sufficiently fast) to a global optimum. This extends a similar phenomenon previously shown only for shallow networks with a squared loss in the ERM setting, to multilayer networks with a loss function that is not necessarily convex in a more general setting.

cs.LG↗

arXiv2021

Irreducibility of random polynomials of bounded degree

It is known that random monic integral polynomials of bounded degree $d$ and integral coefficients distributed uniformly and independently in $[-H,H]$ are irreducible over $\mathbb{Z}$ with probability tending to $1$ as $H\to \infty$. In this paper, we give a general criterion for guaranteeing the same conclusion under much more general coefficient distributions, allowing them to be nonuniformly and dependently distributed over arbitrary sets.

math.NT↗

arXiv2021

Global Convergence of Three-layer Neural Networks in the Mean Field Regime

In the mean field regime, neural networks are appropriately scaled so that as the width tends to infinity, the learning dynamics tends to a nonlinear and nontrivial dynamical limit, known as the mean field limit. This lends a way to study large-width neural networks via analyzing the mean field limit. Recent works have successfully applied such analysis to two-layer networks and provided global convergence guarantees. The extension to multilayer ones however has been a highly challenging puzzle, and little is known about the optimization efficiency in the mean field regime when there are more than two layers. In this work, we prove a global convergence result for unregularized feedforward three-layer networks in the mean field regime. We first develop a rigorous framework to establish the mean field limit of three-layer networks under stochastic gradient descent training. To that end, we propose the idea of a \textit{neuronal embedding}, which comprises of a fixed probability space that encapsulates neural networks of arbitrary sizes. The identified mean field limit is then used to prove a global convergence guarantee under suitable regularity and convergence mode assumptions, which -- unlike previous works on two-layer networks -- does not rely critically on convexity. Underlying the result is a universal approximation property, natural of neural networks, which importantly is shown to hold at \textit{any} finite training time (not necessarily at convergence) via an algebraic topology argument.

cs.LG↗

arXiv2021

Spectral independence, coupling with the stationary distribution, and the spectral gap of the Glauber dynamics

We present a new lower bound on the spectral gap of the Glauber dynamics for the Gibbs distribution of a spectrally independent $q$-spin system on a graph $G = (V,E)$ with maximum degree $Δ$. Notably, for several interesting examples, our bound covers the entire regime of $Δ$ excluded by arguments based on coupling with the stationary distribution. As concrete applications, by combining our new lower bound with known spectral independence computations and known coupling arguments: (1) We show that for a triangle-free graph $G = (V,E)$ with maximum degree $Δ\geq 3$, the Glauber dynamics for the uniform distribution on proper $k$-colorings with $k \geq (1.763\dots + δ)Δ$ colors has spectral gap $\tildeΩ_δ(|V|^{-1})$. Previously, such a result was known either if the girth of $G$ is at least $5$ [Dyer et.~al, FOCS 2004], or under restrictions on $Δ$ [Chen et.~al, STOC 2021; Hayes-Vigoda, FOCS 2003]. (2) We show that for a regular graph $G = (V,E)$ with degree $Δ\geq 3$ and girth at least $6$, and for any $\varepsilon, δ> 0$, the partition function of the hardcore model with fugacity $λ\leq (1-δ)λ_{c}(Δ)$ may be approximated within a $(1+\varepsilon)$-multiplicative factor in time $\tilde{O}_δ(n^{2}\varepsilon^{-2})$. Previously, such a result was known if the girth is at least $7$ [Efthymiou et.~al, SICOMP 2019]. (3) We show for the binomial random graph $G(n,d/n)$ with $d = O(1)$, with high probability, an approximately uniformly random matching may be sampled in time $O_{d}(n^{2+o(1)})$. This improves the corresponding running time of $\tilde{O}_{d}(n^{3})$ due to [Jerrum-Sinclair, SICOMP 1989; Jerrum, 2003].

cs.DS↗

arXiv2021

Subset sums, completeness and colorings

We develop novel techniques which allow us to prove a diverse range of results relating to subset sums and complete sequences of positive integers, including solutions to several longstanding open problems. These include: solutions to the three problems of Burr and Erdős on Ramsey complete sequences, for which Erdős later offered a combined total of \$350; analogous results for the new notion of density complete sequences; the solution to a conjecture of Alon and Erdős on the minimum number of colors needed to color the positive integers less than $n$ so that $n$ cannot be written as a monochromatic sum; the exact determination of an extremal function introduced by Erdős and Graham on sets of integers avoiding a given subset sum; and, answering a question reiterated by several authors, a homogeneous strengthening of a seminal result of Szemerédi and Vu on long arithmetic progressions in subset sums.

math.CO↗

arXiv2021

Mixing time of fractional random walk on finite fields

We study a random walk on $\mathbb{F}_p$ defined by $X_{n+1}=1/X_n+\varepsilon_{n+1}$ if $X_n\neq 0$, and $X_{n+1}=\varepsilon_{n+1}$ if $X_n=0$, where $\varepsilon_{n+1}$ are independent and identically distributed. This can be seen as a non-linear analogue of the Chung--Diaconis--Graham process. We show that the mixing time is of order $\log p$, answering a question of Chatterjee and Diaconis.

math.PR↗

arXiv2021

On the sampling Lovász Local Lemma for atomic constraint satisfaction problems

We study the problem of sampling an approximately uniformly random satisfying assignment for atomic constraint satisfaction problems i.e. where each constraint is violated by only one assignment to its variables. Let $p$ denote the maximum probability of violation of any constraint and let $Δ$ denote the maximum degree of the line graph of the constraints. Our main result is a nearly-linear (in the number of variables) time algorithm for this problem, which is valid in a Lovász local lemma type regime that is considerably less restrictive compared to previous works. In particular, we provide sampling algorithms for the uniform distribution on: (1) $q$-colorings of $k$-uniform hypergraphs with $Δ\lesssim q^{(k-4)/3 + o_{q}(1)}.$ The exponent $1/3$ improves the previously best-known $1/7$ in the case $q, Δ= O(1)$ [Jain, Pham, Vuong; arXiv, 2020] and $1/9$ in the general case [Feng, He, Yin; STOC 2021]. (2) Satisfying assignments of Boolean $k$-CNF formulas with $Δ\lesssim 2^{k/5.741}.$ The constant $5.741$ in the exponent improves the previously best-known $7$ in the case $k = O(1)$ [Jain, Pham, Vuong; arXiv, 2020] and $13$ in the general case [Feng, He, Yin; STOC 2021]. (3) Satisfying assignments of general atomic constraint satisfaction problems with $p\cdot Δ^{7.043} \lesssim 1.$ The constant $7.043$ improves upon the previously best-known constant of $350$ [Feng, He, Yin; STOC 2021]. At the heart of our analysis is a novel information-percolation type argument for showing the rapid mixing of the Glauber dynamics for a carefully constructed projection of the uniform distribution on satisfying assignments. Notably, there is no natural partial order on the space, and we believe that the techniques developed for the analysis may be of independent interest.

cs.DS↗

arXiv2020

Common and Sidorenko Linear Equations

A linear equation with coefficients in $\mathbb{F}_q$ is common if the number of monochromatic solutions in any two-coloring of $\mathbb{F}_q^n$ is asymptotically (as $n \to \infty$) at least the number expected in a random two-coloring. The linear equation is Sidorenko if the number of solutions in any dense subset of $\mathbb{F}_q^n$ is asymptotically at least the number expected in a random set of the same density. In this paper, we characterize those linear equations which are common, and those which are Sidorenko. The main novelty is a construction based on choosing random Fourier coefficients that shows that certain linear equations do not have these properties. This solves problems posed in a paper of Saad and Wolf.

math.CO↗

arXiv2020

Towards the sampling Lovász Local Lemma

Let $Φ= (V, \mathcal{C})$ be a constraint satisfaction problem on variables $v_1,\dots, v_n$ such that each constraint depends on at most $k$ variables and such that each variable assumes values in an alphabet of size at most $[q]$. Suppose that each constraint shares variables with at most $Δ$ constraints and that each constraint is violated with probability at most $p$ (under the product measure on its variables). We show that for $k, q = O(1)$, there is a deterministic, polynomial time algorithm to approximately count the number of satisfying assignments and a randomized, polynomial time algorithm to sample from approximately the uniform distribution on satisfying assignments, provided that \[C\cdot q^{3}\cdot k \cdot p \cdot Δ^{7} < 1, \quad \text{where }C \text{ is an absolute constant.}\] Previously, a result of this form was known essentially only in the special case when each constraint is violated by exactly one assignment to its variables. For the special case of $k$-CNF formulas, the term $Δ^{7}$ improves the previously best known $Δ^{60}$ for deterministic algorithms [Moitra, J.ACM, 2019] and $Δ^{13}$ for randomized algorithms [Feng et al., arXiv, 2020]. For the special case of properly $q$-coloring $k$-uniform hypergraphs, the term $Δ^{7}$ improves the previously best known $Δ^{14}$ for deterministic algorithms [Guo et al., SICOMP, 2019] and $Δ^{9}$ for randomized algorithms [Feng et al., arXiv, 2020].

cs.DS↗

arXiv2020

A Note on the Global Convergence of Multilayer Neural Networks in the Mean Field Regime

In a recent work, we introduced a rigorous framework to describe the mean field limit of the gradient-based learning dynamics of multilayer neural networks, based on the idea of a neuronal embedding. There we also proved a global convergence guarantee for three-layer (as well as two-layer) networks using this framework. In this companion note, we point out that the insights in our previous work can be readily extended to prove a global convergence guarantee for multilayer networks of any depths. Unlike our previous three-layer global convergence guarantee that assumes i.i.d. initializations, our present result applies to a type of correlated initialization. This initialization allows to, at any finite training time, propagate a certain universal approximation property through the depth of the neural network. To achieve this effect, we introduce a bidirectional diversity condition.

cs.LG↗

arXiv2020

Tower-type bounds for Roth's theorem with popular differences

Green developed an arithmetic regularity lemma to prove a strengthening of Roth's theorem on arithmetic progressions in dense sets. It states that for every $ε> 0$ there is some $N_0(ε)$ such that for every $N \ge N_0(ε)$ and $A \subset [N]$ with $|A| = αN$, there is some nonzero $d$ such that $A$ contains at least $(α^3 - ε) N$ three-term arithmetic progressions with common difference $d$. We prove that the minimum $N_0(ε)$ in Green's theorem is an exponential tower of 2s of height on the order of $\log(1/ε)$. Both the lower and upper bounds are new. It shows that the tower-type bounds that arise from the use of a regularity lemma in this application are quantitatively necessary.

math.CO↗

arXiv2019

Popular progression differences in vector spaces II

Green used an arithmetic analogue of Szemerédi's celebrated regularity lemma to prove the following strengthening of Roth's theorem in vector spaces. For every $α>0$, $β<α^3$, and prime number $p$, there is a least positive integer $n_p(α,β)$ such that if $n \geq n_p(α,β)$, then for every subset of $\mathbb{F}_p^n$ of density at least $α$ there is a nonzero $d$ for which the density of three-term arithmetic progressions with common difference $d$ is at least $β$. We determine for $p \geq 19$ the tower height of $n_p(α,β)$ up to an absolute constant factor and an additive term depending only on $p$. In particular, if we want half the random bound (so $β=α^3/2$), then the dimension $n$ required is a tower of twos of height $Θ\left((\log p) \log \log (1/α)\right)$. It turns out that the tower height in general takes on a different form in several different regions of $α$ and $β$, and different arguments are used both in the upper and lower bounds to handle these cases.

math.CO↗

arXiv2017

Popular progression differences in vector spaces

Green proved an arithmetic analogue of Szemerédi's celebrated regularity lemma and used it to verify a conjecture of Bergelson, Host, and Kra which sharpens Roth's theorem on three-term arithmetic progressions in dense sets. It shows that for every subset of $\mathbb{F}_p^n$ with $n$ sufficiently large, the density of three-term arithmetic progressions with some nonzero common difference is at least the random bound (the cube of the set density) up to an additive $ε$. For a fixed odd prime $p$, we prove that the required dimension grows as an exponential tower of $p$'s of height $Θ(\log(1/ε))$. This improves both the lower and upper bound, and is the first example of a result where a tower-type bound coming from applying a regularity lemma is shown to be necessary.

math.CO↗