arXiv subjects

Pierfrancesco Urbani

Publications and source records attributed to Pierfrancesco Urbani.

At least 37 records · Page 2Linked to original sources

arXiv2021

Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems

The optimization step in many machine learning problems rarely relies on vanilla gradient descent but it is common practice to use momentum-based accelerated methods. Despite these algorithms being widely applied to arbitrary loss functions, their behaviour in generically non-convex, high dimensional landscapes is poorly understood. In this work, we use dynamical mean field theory techniques to describe analytically the average dynamics of these methods in a prototypical non-convex model: the (spiked) matrix-tensor model. We derive a closed set of equations that describe the behaviour of heavy-ball momentum and Nesterov acceleration in the infinite dimensional limit. By numerical integration of these equations, we observe that these methods speed up the dynamics but do not improve the algorithmic threshold with respect to gradient descent in the spiked model.

cond-mat.dis-nn↗

arXiv2021

Marginal stability of soft anharmonic mean field spin glasses

We investigate the properties of the glass phase of a recently introduced spin glass model of soft spins subjected to an anharmonic quartic local potential, which serves as a model of low temperature molecular or soft glasses. We solve the model using mean field theory and show that, at low temperatures, it is described by full replica symmetry breaking (fullRSB). As a consequence, at zero temperature the glass phase is marginally stable. We show that in this case, marginal stability comes from a combination of both soft linear excitations -- appearing in a gapless spectrum of the Hessian of linear excitations -- and pseudogapped non-linear excitations -- corresponding to nearly degenerate two level systems. Therefore, this model is a natural candidate to describe what happens in soft glasses, where quasi localized soft modes in the density of states appear together with non-linear modes triggering avalanches and conjectured to be essential to describe the universal low-temperature anomalies of glasses.

cond-mat.dis-nn↗

arXiv2021

Disordered high-dimensional optimal control

Mean field optimal control problems are a class of optimization problems that arise from optimal control when applied to the many body setting. In the noisy case one has a set of controllable stochastic processes and a cost function that is a functional of their trajectories. The goal of the optimization is to minimize this cost over the control variables. Here we consider the case in which we have $N$ stochastic processes, or agents, with the associated control variables, which interact in a disordered way so that the resulting cost function is random. The goal is to find the average minimal cost for $N\to \infty$, when a typical realization of the quenched random interactions is considered. We introduce a simple model and show how to perform a dimensional reduction from the infinite dimensional case to a set of one dimensional stochastic partial differential equations of the Hamilton-Jacobi-Bellman and Fokker-Planck type. The statistical properties of the corresponding stochastic terms must be computed self-consistently, as we show explicitly.

math.OC↗

arXiv2021

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

In this paper we investigate how gradient-based algorithms such as gradient descent, (multi-pass) stochastic gradient descent, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best generalization error at limited sample complexity. We consider the loss landscape of the high-dimensional phase retrieval problem as a prototypical highly non-convex example. We observe that for phase retrieval the stochastic variants of gradient descent are able to reach perfect generalization for regions of control parameters where the gradient descent algorithm is not. We apply dynamical mean-field theory from statistical physics to characterize analytically the full trajectories of these algorithms in their continuous-time limit, with a warm start, and for large system sizes. We further unveil several intriguing properties of the landscape and the algorithms such as that the gradient descent can obtain better generalization properties from less informed initializations.

cond-mat.dis-nn↗

arXiv2020

Low-frequency vibrational spectrum of mean-field disordered systems

We study a recently introduced and exactly solvable mean-field model for the density of vibrational states $\mathcal{D}(ω)$ of a structurally disordered system. The model is formulated as a collection of disordered anharmonic oscillators, with random stiffness $κ$ drawn from a distribution $p(κ)$, subjected to a constant field $h$ and interacting bilinearly with a coupling of strength $J$. We investigate the vibrational properties of its ground state at zero temperature. When $p(κ)$ is gapped, the emergent $\mathcal{D}(ω)$ is also gapped, for small $J$. Upon increasing $J$, the gap vanishes on a critical line in the $(h,J)$ phase diagram, whereupon replica symmetry is broken. At small $h$, the form of this pseudogap is quadratic, $\mathcal{D}(ω)\simω^2$, and its modes are delocalized, as expected from previously investigated mean-field spin glass models. However, we determine that for large enough $h$, a quartic pseudogap $\mathcal{D}(ω)\simω^4$, populated by localized modes, emerges, the two regimes being separated by a special point on the critical line. We thus uncover that mean-field disordered systems can generically display both a quadratic-delocalized and a quartic-localized spectrum at the glass transition.

cond-mat.dis-nn↗

arXiv2020

Mean-field model of interacting quasilocalized excitations in glasses

Structural glasses feature quasilocalized excitations whose frequencies $ω$ follow a universal density of states ${\cal D}(ω)\!\sim\!ω^4$. Yet, the underlying physics behind this universality is not fully understood. Here we study a mean-field model of quasilocalized excitations in glasses, viewed as groups of particles embedded inside an elastic medium and described collectively as anharmonic oscillators. The oscillators, whose harmonic stiffness is taken from a rather featureless probability distribution (of upper cutoff $κ_0$) in the absence of interactions, interact among themselves through random couplings (characterized by strength $J$) and with the surrounding elastic medium (an interaction characterized by a constant force $h$). We first show that the model gives rise to a gapless density of states ${\cal D}(ω)\!=\!A_{\rm g}\,ω^4$ for a broad range of model parameters, expressed in terms of the strength of stabilizing anharmonicity, which plays a decisive role in the model. Then -- using scaling theory and numerical simulations -- we provide a complete understanding of the non-universal prefactor $A_{\rm g}(h,J,κ_0)$, of the oscillators' interaction-induced mean square displacement and of an emerging characteristic frequency, all in terms of properly identified dimensionless quantities. In particular, we show that $A_{\rm g}(h,J,κ_0)$ is a nonmonotonic function of $J$ for a fixed $h$, varying predominantly exponentially with $-(κ_0 h^{2/3}\!/J^2)$ in the weak interactions (small $J$) regime -- reminiscent of recent observations in computer glasses -- and predominantly decaying as a power-law for larger $J$, in a regime where $h$ plays no role. We discuss the physical interpretation of the model and its possible relations to available observations in structural glasses, along with delineating some future research directions.

cond-mat.dis-nn↗

arXiv2020

Proliferation of non-linear excitations in the piecewise-linear perceptron

We investigate the properties of local minima of the energy landscape of a continuous non-convex optimization problem, the spherical perceptron with piecewise linear cost function and show that they are critical, marginally stable and displaying a set of pseudogaps, singularities and non-linear excitations whose properties appear to be in the same universality class of jammed packings of hard spheres. The piecewise linear perceptron problem appears as an evolution of the purely linear perceptron optimization problem that has been recently investigated in [1]. Its cost function contains two non-analytic points where the derivative has a jump. Correspondingly, in the non-convex/glassy phase, these two points give rise to four pseudogaps in the force distribution and this induces four power laws in the gap distribution as well. In addition one can define an extended notion of isostaticity and show that local minima appear again to be isostatic in this phase. We believe that our results generalize naturally to more complex cases with a proliferation of non-linear excitations as the number of non-analytic points in the cost function is increased.

cond-mat.dis-nn↗

arXiv2020

Surfing on minima of isostatic landscapes: avalanches and unjamming transition

Recently, we showed that optimization problems, both in infinite as well as in finite dimensions, for continuous variables and soft excluded volume constraints, can display entire isostatic phases where local minima of the cost function are marginally stable configurations endowed with non-linear excitations [1,2]. In this work we describe an athermal adiabatic algorithm to explore with continuity the corresponding rough high-dimensional landscape. We concentrate on a prototype problem of this kind, the spherical perceptron optimization problem with linear cost function (hinge loss). This algorithm allows to "surf" between isostatic marginally stable configurations and to investigate some properties of such landscape. In particular we focus on the statistics of avalanches occurring when local minima are destabilized. We show that when perturbing such minima, the system undergoes plastic rearrangements whose size is power law distributed and we characterize the corresponding critical exponent. Finally we investigate the critical properties of the unjamming transition, showing that the linear interaction potential gives rise to logarithmic behavior in the scaling of energy and pressure as a function of the distance from the unjamming point. For some quantities, the logarithmic corrections can be gauged out. This is the case of the number of soft constraints that are violated as a function of the distance from jamming which follows a non-trivial power law behavior.

cond-mat.dis-nn↗

arXiv2020

Searching for the Gardner transition in glassy glycerol

We search for a Gardner transition in glassy glycerol, a standard molecular glass, measuring the third harmonics cubic susceptibility $χ_3^{(3)}$ from slightly below the usual glass transition temperature down to $10K$. According to the mean field picture, if local motion within the glass were becoming highly correlated due to the emergence of a Gardner phase then $χ_3^{(3)}$, which is analogous to the dynamical spin-glass susceptibility, should increase and diverge at the Gardner transition temperature $T_G$. We find instead that upon cooling $| χ_3^{(3)} |$ decreases by several orders of magnitude and becomes roughly constant in the regime $100K-10K$. We rationalize our findings by assuming that the low temperature physics is described by localized excitations weakly interacting via a spin-glass dipolar pairwise interaction in a random magnetic field. Our quantitative estimations show that the spin-glass interaction is twenty to fifty times smaller than the local random field contribution, thus rationalizing the absence of the spin-glass Gardner phase. This hints at the fact that a Gardner phase may be suppressed in standard molecular glasses, but it also suggests ways to favor its existence in other amorphous solids and by changing the preparation protocol.

cond-mat.dis-nn↗

arXiv2020

Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval

Despite the widespread use of gradient-based algorithms for optimizing high-dimensional non-convex functions, understanding their ability of finding good minima instead of being trapped in spurious ones remains to a large extent an open problem. Here we focus on gradient flow dynamics for phase retrieval from random measurements. When the ratio of the number of measurements over the input dimension is small the dynamics remains trapped in spurious minima with large basins of attraction. We find analytically that above a critical ratio those critical points become unstable developing a negative direction toward the signal. By numerical experiments we show that in this regime the gradient flow algorithm is not trapped; it drifts away from the spurious critical points along the unstable direction and succeeds in finding the global minimum. Using tools from statistical physics we characterize this phenomenon, which is related to a BBP-type transition in the Hessian of the spurious minima.

cs.LG↗

arXiv2020

Critical energy landscape of linear soft spheres

We show that soft spheres interacting with a linear ramp potential when overcompressed beyond the jamming point fall in an amorphous solid phase which is critical, mechanically marginally stable and share many features with the jamming point itself. In the whole phase, the relevant local minima of the potential energy landscape display an isostatic contact network of perfectly touching spheres whose statistics is controlled by an infinite lengthscale. Excitations around such energy minima are non-linear, system spanning, and characterized by a set of non-trivial critical exponents. We perform numerical simulations to measure their values and show that, while they coincide, within numerical precision, with the critical exponents appearing at jamming, the nature of the corresponding excitations is richer. Therefore, linear soft spheres appear as a novel class of finite dimensional systems that self-organize into new, critical, marginally stable, states.

cond-mat.dis-nn↗

arXiv2020

Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Models

In this work we analyse quantitatively the interplay between the loss landscape and performance of descent algorithms in a prototypical inference problem, the spiked matrix-tensor model. We study a loss function that is the negative log-likelihood of the model. We analyse the number of local minima at a fixed distance from the signal/spike with the Kac-Rice formula, and locate trivialization of the landscape at large signal-to-noise ratios. We evaluate in a closed form the performance of a gradient flow algorithm using integro-differential PDEs as developed in physics of disordered systems for the Langevin dynamics. We analyze the performance of an approximate message passing algorithm estimating the maximum likelihood configuration via its state evolution. We conclude by comparing the above results: while we observe a drastic slow down of the gradient flow dynamics even in the region where the landscape is trivial, both the analyzed algorithms are shown to perform well even in the part of the region of parameters where spurious local minima are present.

cs.LG↗

arXiv2020

Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inference

Gradient-descent-based algorithms and their stochastic versions have widespread applications in machine learning and statistical inference. In this work we perform an analytic study of the performances of one of them, the Langevin algorithm, in the context of noisy high-dimensional inference. We employ the Langevin algorithm to sample the posterior probability measure for the spiked matrix-tensor model. The typical behaviour of this algorithm is described by a system of integro-differential equations that we call the Langevin state evolution, whose solution is compared with the one of the state evolution of approximate message passing (AMP). Our results show that, remarkably, the algorithmic threshold of the Langevin algorithm is sub-optimal with respect to the one given by AMP. We conjecture this phenomenon to be due to the residual glassiness present in that region of parameters. Finally we show how a landscape-annealing protocol, that uses the Langevin algorithm but violate the Bayes-optimality condition, can approach the performance of AMP.

cs.LG↗

arXiv2019

Critical jammed phase of the linear perceptron

Criticality in statistical physics naturally emerges at isolated points in the phase diagram. Jamming of spheres is not an exception: varying density, it is the critical point that separates the unjammed phase where spheres do not overlap and the jammed phase where they cannot be arranged without overlaps. The same remains true in more general constraint satisfaction problems with continuous variables (CCSP) where jamming coincides with the (protocol dependent) satisfiability transition point. In this work we show that by carefully choosing the cost function to be minimized, the region of criticality extends to occupy a whole region of the jammed phase. As a working example, we consider the spherical perceptron with a linear cost function in the unsatisfiable (UNSAT) jammed phase and we perform numerical simulations which show critical power laws emerging in the configurations obtained minimizing the linear cost function. We develop a scaling theory to compute the emerging critical exponents.

cond-mat.dis-nn↗

arXiv2019

Mean field theory of jamming of nonspherical particles

Recent computer simulations have uncovered the striking difference between the jamming transition of spherical and non-spherical particles. While systems of spherical particles are isostatic at the jamming point, systems of nonspherical particles are not: the contact number and shear modulus of the former exhibit a square root singularity near jamming, while those of the latter are linearly proportional to the distance from jamming. Furthermore, while our theoretical understanding of jamming of spherical particles is well developed, the same is not true for nonspherical particles. To understand jamming of non-spherical particles, in the previous work [Brito, PNAS 115(46), 111736], we extended the perceptron model, whose SAT/UNSAT transition belongs to the same universality class of jamming of spherical particles, to include additional variables accounting for the rotational degrees of freedom of nonspherical particles. In this paper, we give more detailed investigations of the full scaling behavior of the model near the jamming transition point in both convex and non-convex phases.

cond-mat.dis-nn↗

arXiv2019

Can a large packing be assembled from smaller ones?

We consider zero temperature packings of soft spheres, that undergo a jamming to unjamming transition as a function of packing fraction. We compare differences in the structure, as measured from the contact statistics, of a finite subsystem of a large packing to a whole packing with periodic boundaries of an equivalent size and pressure. We find that the fluctuations of the ensemble of whole packings are smaller than those of the ensemble of subsystems. Convergence of these two quantities appears to occur at very large systems, which are usually not attainable in numerical simulations. Finding differences between packings in two dimensions and three dimensions, we also consider four dimensions and mean-field models, and find that they show similar system size dependence. Mean-field critical exponents appear to be consistent with the 3d and 4d packings, suggesting they are above the upper critical dimension. We also find that the convergence as a function of system size to the thermodynamic limit is characterized by two different length scales. We argue that this is the result of the system being above the upper critical dimension.

cond-mat.soft↗

arXiv2019

Jamming in multilayer supervised learning models

Critical jamming transitions are characterized by an astonishing degree of universality. Analytic and numerical evidence points to the existence of a large universality class that encompasses finite and infinite dimensional spheres and continuous constraint satisfaction problems (CCSP) such as the non-convex perceptron and related models. In this paper we investigate multilayer neural networks (MLNN) learning random associations as models for CCSP which could potentially define different jamming universality classes. As opposed to simple perceptrons and infinite dimensional spheres, which are described by a single effective field in terms of which the constraints appear to be one-dimensional, the description of MLNN, involves multiple fields, and the constraints acquire a multidimensional character. We first study the models numerically and show that similarly to the perceptron, whenever jamming is isostatic, the sphere universality class is recovered, we then write the exact mean-field equations for the models and identify a dimensional reduction mechanism that leads to a scaling regime identical to one of the infinite dimensional spheres. We suggest that this mechanism could be general enough to explain finite dimensional universality.

cond-mat.dis-nn↗

arXiv2019

Glassy nature of the hard phase in inference problems

An algorithmically hard phase was described in a range of inference problems: even if the signal can be reconstructed with a small error from an information theoretic point of view, known algorithms fail unless the noise-to-signal ratio is sufficiently small. This hard phase is typically understood as a metastable branch of the dynamical evolution of message passing algorithms. In this work we study the metastable branch for a prototypical inference problem, the low-rank matrix factorization, that presents a hard phase. We show that for noise-to-signal ratios that are below the information theoretic threshold, the posterior measure is composed of an exponential number of metastable glassy states and we compute their entropy, called the complexity. We show that this glassiness extends even slightly below the algorithmic threshold below which the well-known approximate message passing (AMP) algorithm is able to closely reconstruct the signal. Counter-intuitively, we find that the performance of the AMP algorithm is not improved by taking into account the glassy nature of the hard phase. This result provides further evidence that the hard phase in inference problems is algorithmically impenetrable for some deep computational reasons that remain to be uncovered.

cond-mat.dis-nn↗