Searcharxiv⌕ Search

arXiv subjects

Ronen Eldan

Publications and source records attributed to Ronen Eldan.

At least 37 records · Page 2Linked to original sources
arXiv2022

Localization Schemes: A Framework for Proving Mixing Bounds for Markov Chains

Two recent and seemingly-unrelated techniques for proving mixing bounds for Markov chains are: (i) the framework of Spectral Independence, introduced by Anari, Liu and Oveis Gharan, and its numerous extensions, which have given rise to several breakthroughs in the analysis of mixing times of discrete Markov chains and (ii) the Stochastic Localization technique which has proven useful in establishing mixing and expansion bounds for both log-concave measures and for measures on the discrete hypercube. In this paper, we introduce a framework which connects ideas from both techniques. Our framework unifies, simplifies and extends those two techniques. In its center is the concept of a localization scheme which, to every probability measure, assigns a martingale of probability measures which localize in space as time evolves. As it turns out, to every such scheme corresponds a Markov chain, and many chains of interest appear naturally in this framework. This viewpoint provides tools for deriving mixing bounds for the dynamics through the analysis of the corresponding localization process. Generalizations of concepts of Spectral Independence and Entropic Independence naturally arise from our definitions, and in particular we recover the main theorems in the spectral and entropic independence frameworks via simple martingale arguments (completely bypassing the need to use the theory of high-dimensional expanders). We demonstrate the strength of our proposed machinery by giving short and (arguably) simpler proofs to many mixing bounds in the recent literature, including giving the first $O(n \log n)$ bound for the mixing time of Glauber dynamics on the hardcore-model (of arbitrary degree) in the tree-uniqueness regime.

math.PR↗
arXiv2021

A Spectral Condition for Spectral Gap: Fast Mixing in High-Temperature Ising Models

We prove that Ising models on the hypercube with general quadratic interactions satisfy a Poincaré inequality with respect to the natural Dirichlet form corresponding to Glauber dynamics, as soon as the operator norm of the interaction matrix is smaller than $1$. The inequality implies a control on the mixing time of the Glauber dynamics. Our techniques rely on a localization procedure which establishes a structural result, stating that Ising measures may be decomposed into a mixture of measures with quadratic potentials of rank one, and provides a framework for proving concentration bounds for high temperature Ising models.

math.PR↗
arXiv2021

Reduction From Non-Unique Games To Boolean Unique Games

We reduce the problem of proving a "Boolean Unique Games Conjecture" (with gap 1-delta vs. 1-C*delta, for any C> 1, and sufficiently small delta>0) to the problem of proving a PCP Theorem for a certain non-unique game. In a previous work, Khot and Moshkovitz suggested an inefficient candidate reduction (i.e., without a proof of soundness). The current work is the first to provide an efficient reduction along with a proof of soundness. The non-unique game we reduce from is similar to non-unique games for which PCP theorems are known. Our proof relies on a new concentration theorem for functions in Gaussian space that are restricted to a random hyperplane. We bound the typical Euclidean distance between the low degree part of the restriction of the function to the hyperplane and the restriction to the hyperplane of the low degree part of the function.

cs.CC↗
arXiv2021

Depth Separations in Neural Networks: What is Actually Being Separated?

Existing depth separation results for constant-depth networks essentially show that certain radial functions in $\mathbb{R}^d$, which can be easily approximated with depth $3$ networks, cannot be approximated by depth $2$ networks, even up to constant accuracy, unless their size is exponential in $d$. However, the functions used to demonstrate this are rapidly oscillating, with a Lipschitz parameter scaling polynomially with the dimension $d$ (or equivalently, by scaling the function, the hardness result applies to $\mathcal{O}(1)$-Lipschitz functions only when the target accuracy $ε$ is at most $\text{poly}(1/d)$). In this paper, we study whether such depth separations might still hold in the natural setting of $\mathcal{O}(1)$-Lipschitz radial functions, when $ε$ does not scale with $d$. Perhaps surprisingly, we show that the answer is negative: In contrast to the intuition suggested by previous work, it \emph{is} possible to approximate $\mathcal{O}(1)$-Lipschitz radial functions with depth $2$, size $\text{poly}(d)$ networks, for every constant $ε$. We complement it by showing that approximating such functions is also possible with depth $2$, size $\text{poly}(1/ε)$ networks, for every constant $d$. Finally, we show that it is not possible to have polynomial dependence in both $d,1/ε$ simultaneously. Overall, our results indicate that in order to show depth separations for expressing $\mathcal{O}(1)$-Lipschitz functions with constant accuracy -- if at all possible -- one would need fundamentally different techniques than existing ones in the literature.

cs.LG↗
arXiv2021

Non-asymptotic approximations of neural networks by Gaussian processes

We study the extent to which wide neural networks may be approximated by Gaussian processes when initialized with random weights. It is a well-established fact that as the width of a network goes to infinity, its law converges to that of a Gaussian process. We make this quantitative by establishing explicit convergence rates for the central limit theorem in an infinite-dimensional functional space, metrized with a natural transportation distance. We identify two regimes of interest; when the activation function is polynomial, its degree determines the rate of convergence, while for non-polynomial activations, the rate is governed by the smoothness of the function.

math.PR↗
arXiv2020

Network size and weights size for memorization with two-layers neural networks

In 1988, Eric B. Baum showed that two-layers neural networks with threshold activation function can perfectly memorize the binary labels of $n$ points in general position in $\mathbb{R}^d$ using only $\ulcorner n/d \urcorner$ neurons. We observe that with ReLU networks, using four times as many neurons one can fit arbitrary real labels. Moreover, for approximate memorization up to error $ε$, the neural tangent kernel can also memorize with only $O\left(\frac{n}{d} \cdot \log(1/ε) \right)$ neurons (assuming that the data is well dispersed too). We show however that these constructions give rise to networks where the magnitude of the neurons' weights are far from optimal. In contrast we propose a new training procedure for ReLU networks, based on complex (as opposed to real) recombination of the neurons, for which we show approximate memorization with both $O\left(\frac{n}{d} \cdot \frac{\log(1/ε)}ε\right)$ neurons, as well as nearly-optimal size of the weights.

cs.LG↗
arXiv2020

The CLT in high dimensions: quantitative bounds via martingale embedding

We introduce a new method for obtaining quantitative convergence rates for the central limit theorem (CLT) in a high dimensional setting. Using our method, we obtain several new bounds for convergence in transportation distance and entropy, and in particular: (a) We improve the best known bound, obtained by the third named author, for convergence in quadratic Wasserstein transportation distance for bounded random vectors; (b) We derive the first non-asymptotic convergence rate for the entropic CLT in arbitrary dimension, for general log-concave random vectors; (c) We give an improved bound for convergence in transportation distance under a log-concavity assumption and improvements for both metrics under the assumption of strong log-concavity. Our method is based on martingale embeddings and specifically on the Skorokhod embedding constructed by the first named author.

math.PR↗
arXiv2020

Log concavity and concentration of Lipschitz functions on the Boolean hypercube

It is well-known that measures whose density is the form $e^{-V}$ where $V$ is a uniformly convex potential on $\RR^n$ attain strong concentration properties. In search of a notion of log-concavity on the discrete hypercube, we consider measures on $\{-1,1\}^n$ whose multi-linear extension $f$ satisfies $\log \nabla^2 f(x) \preceq β\Id$, for $β\geq 0$, which we refer to as $β$-semi-log-concave. We prove that these measures satisfy a nontrivial concentration bound, namely, any Hamming Lipchitz test function $φ$ satisfies $\Var_ν[φ] \leq n^{2-C_β}$ for $C_β>0$. As a corollary, we prove a concentration bound for measures which exhibit the so-called Rayleigh property. Namely, we show that for measures such that under any external field (or exponential tilt), the correlation between any two coordinates is non-positive, Hamming-Lipschitz functions admit nontrivial concentration.

math.PR↗
arXiv2020

Stability of the logarithmic Sobolev inequality via the Föllmer Process

We study the stability and instability of the Gaussian logarithmic Sobolev inequality, in terms of covariance, Wasserstein distance and Fisher information, addressing several open questions in the literature. We first establish an improved logarithmic Sobolev inequality which is at the same time scale invariant and dimension free. As a corollary, we show that if the covariance of the measure is bounded by the identity, one may obtain a sharp and dimension-free stability bound in terms of the Fisher information matrix. We then investigate under what conditions stability estimates control the covariance, and when such control is impossible. For the class of measures whose covariance matrix is dominated by the identity, we obtain optimal dimension-free stability bounds which show that the deficit in the logarithmic Sobolev inequality is minimized by Gaussian measures, under a fixed covariance constraint. On the other hand, we construct examples showing that without the boundedness of the covariance, the inequality is not stable. Finally, we study stability in terms of the Wasserstein distance, and show that even for the class of measures with a bounded covariance matrix, it is hopeless to obtain a dimension-free stability result. The counterexamples provided motivate us to put forth a new notion of stability, in terms of proximity to mixtures of the Gaussian distribution. We prove new estimates (some dimension-free) based on this notion. These estimates are strictly stronger than some of the existing stability results in terms of the Wasserstein metric. Our proof techniques rely heavily on stochastic methods.

math.PR↗
arXiv2020

A simple approach to chaos for $p$-spin models

We prove that, in mixed $p$-spin models of spin glasses, the location of the ground state is chaotic under small Gaussian perturbations. For the case of even $p$-spin models, this was shown by Chen, Handschy and Lerman [2018]. We rely on a different approach which only uses the Parisi formula as a black box.

math-ph↗
arXiv2020

Concentration on the Boolean hypercube via pathwise stochastic analysis

We develop a new technique for proving concentration inequalities which relate between the variance and influences of Boolean functions. Using this technique, we 1. Settle a conjecture of Talagrand [Tal97] proving that $$\int_{\left\{ -1,1\right\} ^{n}}\sqrt{h_{f}\left(x\right)}dμ\geq C\cdot\mathrm{var}\left(f\right)\cdot\left(\log\left(\frac{1}{\sum\mathrm{Inf}_{i}^{2}\left(f\right)}\right)\right)^{1/2},$$ where $h_{f}\left(x\right)$ is the number of edges at $x$ along which $f$ changes its value, and $\mathrm{Inf}_{i}\left(f\right)$ is the influence of the $i$-th coordinate. 2. Strengthen several classical inequalities concerning the influences of a Boolean function, showing that near-maximizers must have large vertex boundaries. An inequality due to Talagrand states that for a Boolean function $f$, $\mathrm{var}\left(f\right)\leq C\sum_{i=1}^{n}\frac{\mathrm{Inf}_{i}\left(f\right)}{1+\log\left(1/\mathrm{Inf}_{i}\left(f\right)\right)}$. We give a lower bound for the size of the vertex boundary of functions saturating this inequality. As a corollary, we show that for sets that satisfy the edge-isoperimetric inequality or the Kahn-Kalai-Linial inequality up to a constant, a constant proportion of the mass is in the inner vertex boundary. 3. Improve a quantitative relation between influences and noise stability given by Keller and Kindler. Our proofs rely on techniques based on stochastic calculus, and bypass the use of hypercontractivity common to previous proofs.

math.PR↗
arXiv2020

Information and dimensionality of anisotropic random geometric graphs

This paper deals with the problem of detecting non-isotropic high-dimensional geometric structure in random graphs. Namely, we study a model of a random geometric graph in which vertices correspond to points generated randomly and independently from a non-isotropic $d$-dimensional Gaussian distribution, and two vertices are connected if the distance between them is smaller than some pre-specified threshold. We derive new notions of dimensionality which depend upon the eigenvalues of the covariance of the Gaussian distribution. If $α$ denotes the vector of eigenvalues, and $n$ is the number of vertices, then the quantities $\left(\frac{||α||_2}{||α||_3}\right)^6/n^3$ and $\left(\frac{||α||_2}{||α||_4}\right)^4/n^3$ determine upper and lower bounds for the possibility of detection. This generalizes a recent result by Bubeck, Ding, Rácz and the first named author from [BDER14] which shows that the quantity $d/n^3$ determines the boundary of detection for isotropic geometry. Our methods involve Fourier analysis and the theory of characteristic functions to investigate the underlying probabilities of the model. The proof of the lower bound uses information theoretic tools, based on the method presented in [BG15].

math.ST↗
arXiv2019

Second-order bounds on correlations between increasing families

Harris's correlation inequality states that any two monotone functions on the Boolean hypercube are positively correlated. Talagrand \cite{Talcorr} started a line of works in search of quantitative versions of this fact by providing a lower bound on the correlation in terms of the influences of the functions. A famous conjecture of Chvátal \cite{Chvatal} was found by Friedgut, Kahn, Kalai and Keller \cite{FKKK} to be equivalent to a certain strengthening of Talagrand's bound, conjectured to hold true when one of the functions is antipodal (hence $g(x) = 1-g(-x)$). Motivated by this conjecture, we strengthen some of those bounds by giving estimates that also involve the second order Fourier coefficients of the functions. In particular we show that in the bounds due to Talagrand and due to Keller, Mossel and Sen \cite{KMS14}, a logarithmic factor can be replaced by its square root when one of the functions is antipodal. Our proofs follow a different route than the ones in the literature, and the analysis is carried out in the Gaussian setting.

math.CO↗
arXiv2019

Krivine diffusions attain the Goemans--Williamson approximation ratio

Answering a question of Abbasi-Zadeh, Bansal, Guruganesh, Nikolov, Schwartz and Singh (2018), we prove the existence of a slowed-down sticky Brownian motion whose induced rounding for MAXCUT attains the Goemans--Williamson approximation ratio. This is an especially simple particular case of the general rounding framework of Krivine diffusions that we investigate elsewhere.

cs.DS↗
arXiv2019

Taming correlations through entropy-efficient measure decompositions with applications to mean-field approximation

The analysis of various models in statistical physics relies on the existence of decompositions of measures into mixtures of product-like components, where the goal is to attain a decomposition into measures whose entropy is close to that of the original measure, yet with small correlations between coordinates. We prove a related general result: For every isotropic measure $μ$ on $\mathbb{R}^n$ and every $ε> 0$, there exists a decomposition $μ= \int μ_θd m(θ)$ such that $H(μ) - \mathbb{E}_{θ\sim m} H(μ_θ) \leq n ε$ and $\mathbb{E}_{θ\sim m} \mathrm{Cov}(μ_θ) \preceq \mathrm{Id}/ε$. As an application, we prove a general bound for the mean-field approximation of Ising and Potts models, which is in a sense dimension free, in both continuous and discrete settings. In particular, for an Ising model on $\{\pm 1 \}^n$ or on $[-1,1]^n$, we show that the deficit between the mean-field approximation and the free energy is at most $C \frac{1+p}{p} \left ( n\|J\|_{S_p} \right)^{\frac{p}{1+p}} $ for all $p>0$, where $\|J\|_{S_p}$ denotes the Schatten-$p$ norm of the interaction matrix. For the case $p=2$, this recovers the result of [Jain et al., 2018], but for an optimal choice of $p$ it often allows to get almost dimension-free bounds.

math.PR↗
arXiv2019

Stability of the Shannon-Stam inequality via the Föllmer process

We prove stability estimates for the Shannon-Stam inequality (also known as the entropy-power inequality) for log-concave random vectors in terms of entropy and transportation distance. In particular, we give the first stability estimate for general log-concave random vectors in the following form: for log-concave random vectors $X,Y \in \mathbb{R}^d$, the deficit in the Shannon-Stam inequality is bounded from below by the expression $$ C \left(\mathrm{D}\left(X||G\right) + \mathrm{D}\left(Y||G\right)\right), $$ where $\mathrm{D}\left( \cdot ~ ||G\right)$ denotes the relative entropy with respect to the standard Gaussian and the constant $C$ depends only on the covariance structures and the spectral gaps of $X$ and $Y$. In the case of uniformly log-concave vectors our analysis gives dimension-free bounds. Our proofs are based on a new approach which uses an entropy-minimizing process from stochastic control theory.

cs.IT↗
arXiv2018

Gaussian-width gradient complexity, reverse log-Sobolev inequalities and nonlinear large deviations

We prove structure theorems for measures on the discrete cube and on Gaussian space, which provide sufficient conditions for mean-field behavior. These conditions rely on a new notion of complexity for such measures, namely the Gaussian-width of the gradient of the log-density. On the cube $\{-1,1\}^n$, we show that a measure $ν$ which exhibits low complexity can be written as a mixture of measures $\{ν_θ\}_{θ\in \mathcal{I}}$ such that: i. for each $θ$, the measure $ν_θ$ is a small perturbation of $ν$ such that $\log \tfrac{d ν_θ}{d ν}$ is a linear function whose gradient is small and, ii. $ν_θ$ is close to some product measure, in Wasserstein distance, for most $θ$. Thus, our framework can be used to study the behavior of low-complexity measures beyond approximation of the partition function, showing that those measures are roughly mixtures of product measures whose entropy is close to that of the original measure. In particular, as a corollary of our theorems, we derive a bound for the naïve mean-field approximation of the log-partition function which improves the nonlinear large deviation framework of Chatterjee and Dembo in several ways: 1. It does not require any bounds on second derivatives. 2. The covering number is replaced by the weaker notion of Gaussian-width 3. We obtain stronger asymptotics with respect to the dimension. Two other corollaries are decomposition theorems for exponential random graphs and large-degree Ising models. In the Gaussian case, we show that measures of low-complexity exhibit an almost-tight reverse Log-Sobolev inequality.

math.PR↗