arXiv subjects

Giulio Biroli

Publications and source records attributed to Giulio Biroli.

At least 91 records · Page 5Linked to original sources

arXiv2021

Mean-field caging in a random Lorentz gas

The random Lorentz gas (RLG) is a minimal model of both percolation and glassiness, which leads to a paradox in the infinite-dimensional, $d\rightarrow\infty$ limit: the localization transition is then expected to be continuous for the former and discontinuous for the latter. As a putative resolution, we have recently suggested that as $d$ increases the behavior of the RLG converges to the glassy description, and that percolation physics is recovered thanks to finite-$d$ perturbative and non-perturbative (instantonic) corrections [Biroli et al. arXiv:2003.11179]. Here, we expand on the $d\rightarrow\infty$ physics by considering a simpler static solution as well as the dynamical solution of the RLG. Comparing the $1/d$ correction of this solution with numerical results reveals that even perturbative corrections fall out of reach of existing theoretical descriptions. Comparing the dynamical solution with the mode-coupling theory (MCT) results further reveals that although key quantitative features of MCT are far off the mark, it does properly capture the discontinuous nature of the $d\rightarrow\infty$ RLG. These insights help chart a path toward a complete description of finite-dimensional glasses.

cond-mat.dis-nn↗

arXiv2021

The Lévy-Rosenzweig-Porter random matrix ensemble

In this paper we consider an extension of the Rosenzweig-Porter (RP) model, the Lévy-RP (L-RP) model, in which the off-diagonal matrix elements are broadly distributed, providing a more realistic benchmark to develop an effective description of non-ergodic extended (NEE) states in interacting many-body disordered systems. We put forward a simple, general, and intuitive argument that allows one to unveil the multifractal structure of the mini-bands in the local spectrum when hybridization is due to anomalously large transition amplitudes in the tails of the distribution. The idea is that the energy spreading of the mini-bands can be determined self-consistently by requiring that the maximum of the matrix elements between a site $i$ and the other $N^{D_1}$ sites of the support set is of the same order of the Thouless energy itself $N^{D_1 - 1}$. This argument yields the fractal dimensions that characterize the statistics of the multifractal wave-functions in the NEE phase, as well as the whole phase diagram of the L-RP ensemble. Its predictions are confirmed both analytically, by a thorough investigation of the self-consistent equation for the local density of states obtained using the cavity approach, and numerically, via extensive exact diagonalizations.

cond-mat.dis-nn↗

arXiv2021

Amorphous Order & Non-linear Susceptibilities in Glassy Materials

We review 15 years of theoretical and experimental work on the non-linear response of glassy systems. We argue that an anomalous growth of the peak value of non-linear susceptibilities is a signature of growing "amorphous order" in the system, with spin-glasses as a case in point. Experimental results on supercooled liquids are fully compatible with the RFOT prediction of compact "glassites" of increasing volume as temperature is decreased, or as the system ages. We clarify why such a behaviour is hard to explain within purely kinetic theories of glass formation, despite recent claims to the contrary.

cond-mat.dis-nn↗

arXiv2020

Searching for the Gardner transition in glassy glycerol

We search for a Gardner transition in glassy glycerol, a standard molecular glass, measuring the third harmonics cubic susceptibility $χ_3^{(3)}$ from slightly below the usual glass transition temperature down to $10K$. According to the mean field picture, if local motion within the glass were becoming highly correlated due to the emergence of a Gardner phase then $χ_3^{(3)}$, which is analogous to the dynamical spin-glass susceptibility, should increase and diverge at the Gardner transition temperature $T_G$. We find instead that upon cooling $| χ_3^{(3)} |$ decreases by several orders of magnitude and becomes roughly constant in the regime $100K-10K$. We rationalize our findings by assuming that the low temperature physics is described by localized excitations weakly interacting via a spin-glass dipolar pairwise interaction in a random magnetic field. Our quantitative estimations show that the spin-glass interaction is twenty to fifty times smaller than the local random field contribution, thus rationalizing the absence of the spin-glass Gardner phase. This hints at the fact that a Gardner phase may be suppressed in standard molecular glasses, but it also suggests ways to favor its existence in other amorphous solids and by changing the preparation protocol.

cond-mat.dis-nn↗

arXiv2020

Triple descent and the two kinds of overfitting: Where & why do they appear?

A recent line of research has highlighted the existence of a "double descent" phenomenon in deep learning, whereby increasing the number of training examples $N$ causes the generalization error of neural networks to peak when $N$ is of the same order as the number of parameters $P$. In earlier works, a similar phenomenon was shown to exist in simpler models such as linear regression, where the peak instead occurs when $N$ is equal to the input dimension $D$. Since both peaks coincide with the interpolation threshold, they are often conflated in the litterature. In this paper, we show that despite their apparent similarity, these two scenarios are inherently different. In fact, both peaks can co-exist when neural networks are applied to noisy regression tasks. The relative size of the peaks is then governed by the degree of nonlinearity of the activation function. Building on recent developments in the analysis of random feature models, we provide a theoretical ground for this sample-wise triple descent. As shown previously, the nonlinear peak at $N\!=\!P$ is a true divergence caused by the extreme sensitivity of the output function to both the noise corrupting the labels and the initialization of the random features (or the weights in neural networks). This peak survives in the absence of noise, but can be suppressed by regularization. In contrast, the linear peak at $N\!=\!D$ is solely due to overfitting the noise in the labels, and forms earlier during training. We show that this peak is implicitly regularized by the nonlinearity, which is why it only becomes salient at high noise and is weakly affected by explicit regularization. Throughout the paper, we compare analytical results obtained in the random feature model with the outcomes of numerical experiments involving deep neural networks.

cs.LG↗

arXiv2020

Glasses and aging: A Statistical Mechanics Perspective

We review the field of the glass transition, glassy dynamics and aging from a statistical mechanics perspective. We give a brief introduction to the subject and explain the main phenomenology encountered in glassy systems, with a particular emphasis on spatially heterogeneous dynamics. We review the main theoretical approaches currently available to account for these glassy phenomena, including recent developments regarding mean-field theory of liquids and glasses, novel computational tools, and connections to the jamming transition. Finally, the physics of aging and off-equilibrium dynamics exhibited by glassy materials is discussed.

cond-mat.stat-mech↗

arXiv2020

Properties of equilibria and glassy phases of the random Lotka-Volterra model with demographic noise

In this letter we study a reference model in theoretical ecology, the disordered Lotka-Volterra model for ecological communities, in the presence of finite demographic noise. Our theoretical analysis, which takes advantage of a mapping to an equilibrium disordered system, proves that for sufficiently heterogeneous interactions and low demographic noise the system displays a multiple equilibria phase, which we fully characterize. In particular, we show that in this phase the number of stable equilibria is exponential in the number of species. Upon further decreasing the demographic noise, we unveil a "Gardner" transition to a marginally stable phase, similar to that observed in jamming of amorphous materials. We confirm and complement our analytical results by numerical simulations. Furthermore, we extend their relevance by showing that they hold for others interacting random dynamical systems, such as the Random Replicant Model. Finally, we discuss their extension to the case of asymmetric couplings.

cond-mat.dis-nn↗

arXiv2020

An analytic theory of shallow networks dynamics for hinge loss classification

Neural networks have been shown to perform incredibly well in classification tasks over structured high-dimensional datasets. However, the learning dynamics of such networks is still poorly understood. In this paper we study in detail the training dynamics of a simple type of neural network: a single hidden layer trained to perform a classification task. We show that in a suitable mean-field limit this case maps to a single-node learning problem with a time-dependent dataset determined self-consistently from the average nodes population. We specialize our theory to the prototypical case of a linearly separable dataset and a linear hinge loss, for which the dynamics can be explicitly solved. This allow us to address in a simple setting several phenomena appearing in modern networks such as slowing down of training dynamics, crossover between rich and lazy learning, and overfitting. Finally, we asses the limitations of mean-field theory by studying the case of large but finite number of nodes and of training samples.

stat.ML↗

arXiv2020

Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval

Despite the widespread use of gradient-based algorithms for optimizing high-dimensional non-convex functions, understanding their ability of finding good minima instead of being trapped in spurious ones remains to a large extent an open problem. Here we focus on gradient flow dynamics for phase retrieval from random measurements. When the ratio of the number of measurements over the input dimension is small the dynamics remains trapped in spurious minima with large basins of attraction. We find analytically that above a critical ratio those critical points become unstable developing a negative direction toward the signal. By numerical experiments we show that in this regime the gradient flow algorithm is not trapped; it drifts away from the spurious critical points along the unstable direction and succeeds in finding the global minimum. Using tools from statistical physics we characterize this phenomenon, which is related to a BBP-type transition in the Hessian of the spurious minima.

cs.LG↗

arXiv2020

Role of fluctuations in the yielding transition of two-dimensional glasses

We numerically study yielding in two-dimensional glasses which are generated with a very wide range of stabilities by swap Monte-Carlo simulations and then slowly deformed at zero temperature. We provide strong numerical evidence that stable glasses yield via a nonequilibrium discontinuous transition in the thermodynamic limit. A critical point separates this brittle yielding from the ductile one observed in less stable glasses. We find that two-dimensional glasses yield similarly to their three-dimensional counterparts but display larger sample-to-sample disorder-induced fluctuations, stronger finite-size effects, and rougher spatial wandering of the observed shear bands. These findings strongly constrain effective theories of yielding.

cond-mat.dis-nn↗

arXiv2020

Dynamical Mean-Field Theory and Aging Dynamics

Dynamical Mean-Field Theory (DMFT) replaces the many-body dynamical problem with one for a single degree of freedom in a thermal bath whose features are determined self-consistently. By focusing on models with soft disordered $p$-spin interactions, we show how to incorporate the mean-field theory of aging within dynamical mean-field theory. We study cases with only one slow time-scale, corresponding statically to the one-step replica symmetry breaking (1RSB) phase, and cases with an infinite number of slow time-scales, corresponding statically to the full replica symmetry breaking (FRSB) phase. For the former, we show that the effective temperature of the slow degrees of freedom is fixed by requiring critical dynamical behavior on short time-scales, i.e. marginality. For the latter, we find that aging on an infinite number of slow time-scales is governed by a stochastic equation where the clock for dynamical evolution is fixed by the change of effective temperature, hence obtaining a dynamical derivation of the stochastic equation at the basis of the FRSB phase. Our results extend the realm of the mean-field theory of aging to all situations where DMFT holds.

cond-mat.dis-nn↗

arXiv2020

Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime

Deep neural networks can achieve remarkable generalization performances while interpolating the training data perfectly. Rather than the U-curve emblematic of the bias-variance trade-off, their test error often follows a "double descent" - a mark of the beneficial role of overparametrization. In this work, we develop a quantitative theory for this phenomenon in the so-called lazy learning regime of neural networks, by considering the problem of learning a high-dimensional function with random features regression. We obtain a precise asymptotic expression for the bias-variance decomposition of the test error, and show that the bias displays a phase transition at the interpolation threshold, beyond which it remains constant. We disentangle the variances stemming from the sampling of the dataset, from the additive noise corrupting the labels, and from the initialization of the weights. Following up on Geiger et al. 2019, we first show that the latter two contributions are the crux of the double descent: they lead to the overfitting peak at the interpolation threshold and to the decay of the test error upon overparametrization. We then quantify how they are suppressed by ensemble averaging the outputs of K independently initialized estimators. When K is sent to infinity, the test error remains constant beyond the interpolation threshold. We further compare the effects of overparametrizing, ensembling and regularizing. Finally, we present numerical experiments on classic deep learning setups to show that our results hold qualitatively in realistic lazy learning scenarios.

cs.LG↗

arXiv2020

Anomalous dynamics in the ergodic side of the Many-Body Localization transition and the glassy phase of Directed Polymers in Random Media

Using the non-interacting Anderson tight-binding model on the Bethe lattice as a toy model for the many-body quantum dynamics, we propose a novel and transparent theoretical explanation of the anomalously slow dynamics that emerges in the bad metal phase preceding the Many-Body Localization transition. By mapping the time-decorrelation of many-body wave-functions onto Directed Polymers in Random Media, we show the existence of a glass transition within the extended regime separating a metallic-like phase at small disorder, where delocalization occurs on an exponential number of paths, from a bad metal-like phase at intermediate disorder, where resonances are formed on rare, specific, disorder dependent site orbitals on very distant generations. The physical interpretation of subdiffusion and non-exponential relaxation emerging from this picture is complementary to the Griffiths one, although both scenarios rely on the presence of heavy-tailed distribution of the escape times. We relate the dynamical evolution in the glassy phase to the depinning transition of Directed Polymers, which results in macroscopic and abrupt jumps of the preferred delocalizing paths when a parameter like the energy is varied, and produce a singular behavior of the overlap correlation function between eigenstates at different energies. By comparing the quantum dynamics on loop-less Cayley trees and Random Regular Graphs we discuss the effect of loops, showing that in the latter slow dynamics and apparent power-laws extend on a very large time-window but are eventually cut-off on a time-scale that diverges at the MBL transition.

cond-mat.dis-nn↗

arXiv2020

How to iron out rough landscapes and get optimal performances: Averaged Gradient Descent and its application to tensor PCA

In many high-dimensional estimation problems the main task consists in minimizing a cost function, which is often strongly non-convex when scanned in the space of parameters to be estimated. A standard solution to flatten the corresponding rough landscape consists in summing the losses associated to different data points and obtain a smoother empirical risk. Here we propose a complementary method that works for a single data point. The main idea is that a large amount of the roughness is uncorrelated in different parts of the landscape. One can then substantially reduce the noise by evaluating an empirical average of the gradient obtained as a sum over many random independent positions in the space of parameters to be optimized. We present an algorithm, called Averaged Gradient Descent, based on this idea and we apply it to tensor PCA, which is a very hard estimation problem. We show that Averaged Gradient Descent over-performs physical algorithms such as gradient descent and approximate message passing and matches the best algorithmic thresholds known so far, obtained by tensor unfolding and methods based on sum-of-squares.

stat.ML↗

arXiv2020

Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias

Despite the phenomenal success of deep neural networks in a broad range of learning tasks, there is a lack of theory to understand the way they work. In particular, Convolutional Neural Networks (CNNs) are known to perform much better than Fully-Connected Networks (FCNs) on spatially structured data: the architectural structure of CNNs benefits from prior knowledge on the features of the data, for instance their translation invariance. The aim of this work is to understand this fact through the lens of dynamics in the loss landscape. We introduce a method that maps a CNN to its equivalent FCN (denoted as eFCN). Such an embedding enables the comparison of CNN and FCN training dynamics directly in the FCN space. We use this method to test a new training protocol, which consists in training a CNN, embedding it to FCN space at a certain ``relax time'', then resuming the training in FCN space. We observe that for all relax times, the deviation from the CNN subspace is small, and the final performance reached by the eFCN is higher than that reachable by a standard FCN of same architecture. More surprisingly, for some intermediate relax times, the eFCN outperforms the CNN it stemmed, by combining the prior information of the CNN and the expressivity of the FCN in a complementary way. The practical interest of our protocol is limited by the very large size of the highly sparse eFCN. However, it offers interesting insights into the persistence of architectural bias under stochastic gradient dynamics. It shows the existence of some rare basins in the FCN loss landscape associated with very good generalization. These can only be accessed thanks to the CNN prior, which helps navigate the landscape during the early stages of optimization.

cs.LG↗

arXiv2020

Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Model

Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in optimising high-dimensional non-convex functions and why they find good minima instead of being trapped in spurious ones. Here we present a quantitative theory explaining this behaviour in a spiked matrix-tensor model. Our framework is based on the Kac-Rice analysis of stationary points and a closed-form analysis of gradient-flow originating from statistical physics. We show that there is a well defined region of parameters where the gradient-flow algorithm finds a good global minimum despite the presence of exponentially many spurious local minima. We show that this is achieved by surfing on saddles that have strong negative direction towards the global minima, a phenomenon that is connected to a BBP-type threshold in the Hessian describing the critical points of the landscapes.

cs.LG↗

arXiv2020

Attractive versus truncated repulsive supercooled liquids: The dynamics is encoded in the pair correlation function

We compare glassy dynamics in two liquids that differ in the form of their interaction potentials. Both systems have the same repulsive interactions but one has also an attractive part in the potential. These two systems exhibit very different dynamics despite having nearly identical pair correlation functions. We demonstrate that a properly weighted integral of the pair correlation function, which amplifies the subtle differences between the two systems, correctly captures their dynamical differences. The weights are obtained from a standard machine learning algorithm.

cond-mat.stat-mech↗

arXiv2020

Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inference

Gradient-descent-based algorithms and their stochastic versions have widespread applications in machine learning and statistical inference. In this work we perform an analytic study of the performances of one of them, the Langevin algorithm, in the context of noisy high-dimensional inference. We employ the Langevin algorithm to sample the posterior probability measure for the spiked matrix-tensor model. The typical behaviour of this algorithm is described by a system of integro-differential equations that we call the Langevin state evolution, whose solution is compared with the one of the state evolution of approximate message passing (AMP). Our results show that, remarkably, the algorithmic threshold of the Langevin algorithm is sub-optimal with respect to the one given by AMP. We conjecture this phenomenon to be due to the residual glassiness present in that region of parameters. Finally we show how a landscape-annealing protocol, that uses the Langevin algorithm but violate the Bayes-optimality condition, can approach the performance of AMP.

cs.LG↗