arXiv subjects

Elena Agliari

Publications and source records attributed to Elena Agliari.

At least 37 records · Page 2Linked to original sources

arXiv2022

Supervised Hebbian Learning

In neural network's Literature, Hebbian learning traditionally refers to the procedure by which the Hopfield model and its generalizations store archetypes (i.e., definite patterns that are experienced just once to form the synaptic matrix). However, the term "Learning" in Machine Learning refers to the ability of the machine to extract features from the supplied dataset (e.g., made of blurred examples of these archetypes), in order to make its own representation of the unavailable archetypes. Here, given a sample of examples, we define a supervised learning protocol by which the Hopfield network can infer the archetypes, and we detect the correct control parameters (including size and quality of the dataset) to depict a phase diagram for the system performance. We also prove that, for structureless datasets, the Hopfield model equipped with this supervised learning rule is equivalent to a restricted Boltzmann machine and this suggests an optimal and interpretable training routine. Finally, this approach is generalized to structured datasets: we highlight a quasi-ultrametric organization (reminiscent of replica-symmetry-breaking) in the analyzed datasets and, consequently, we introduce an additional "replica hidden layer" for its (partial) disentanglement, which is shown to improve MNIST classification from 75% to 95%, and to offer a new perspective on deep architectures.

cond-mat.dis-nn↗

arXiv2022

Pavlov Learning Machines

As well known, Hebb's learning traces its origin in Pavlov's Classical Conditioning, however, while the former has been extensively modelled in the past decades (e.g., by Hopfield model and countless variations on theme), as for the latter modelling has remained largely unaddressed so far; further, a bridge between these two pillars is totally lacking. The main difficulty towards this goal lays in the intrinsically different scales of the information involved: Pavlov's theory is about correlations among \emph{concepts} that are (dynamically) stored in the synaptic matrix as exemplified by the celebrated experiment starring a dog and a ring bell; conversely, Hebb's theory is about correlations among pairs of adjacent neurons as summarized by the famous statement {\em neurons that fire together wire together}. In this paper we rely on stochastic-process theory and model neural and synaptic dynamics via Langevin equations, to prove that -- as long as we keep neurons' and synapses' timescales largely split -- Pavlov mechanism spontaneously takes place and ultimately gives rise to synaptic weights that recover the Hebbian kernel.

cond-mat.dis-nn↗

arXiv2022

Recurrent neural networks that generalize from examples and optimize by dreaming

The gap between the huge volumes of data needed to train artificial neural networks and the relatively small amount of data needed by their biological counterparts is a central puzzle in machine learning. Here, inspired by biological information-processing, we introduce a generalized Hopfield network where pairwise couplings between neurons are built according to Hebb's prescription for on-line learning and allow also for (suitably stylized) off-line sleeping mechanisms. Moreover, in order to retain a learning framework, here the patterns are not assumed to be available, instead, we let the network experience solely a dataset made of a sample of noisy examples for each pattern. We analyze the model by statistical-mechanics tools and we obtain a quantitative picture of its capabilities as functions of its control parameters: the resulting network is an associative memory for pattern recognition that learns from examples on-line, generalizes and optimizes its storage capacity by off-line sleeping. Remarkably, the sleeping mechanisms always significantly reduce (up to $\approx 90\%$) the dataset size required to correctly generalize, further, there are memory loads that are prohibitive to Hebbian networks without sleeping (no matter the size and quality of the provided examples), but that are easily handled by the present "rested" neural networks.

cond-mat.dis-nn↗

arXiv2022

Non-linear PDEs approach to statistical mechanics of Dense Associative Memories

Dense associative memories (DAM), are widespread models in artificial intelligence used for pattern recognition tasks; computationally, they have been proven to be robust against adversarial input and theoretically, leveraging their analogy with spin-glass systems, they are usually treated by means of statistical-mechanics tools. Here we develop analytical methods, based on nonlinear PDEs, to investigate their functioning. In particular, we prove differential identities involving DAM partition function and macroscopic observables useful for a qualitative and quantitative analysis of the system. These results allow for a deeper comprehension of the mechanisms underlying DAMs and provide interdisciplinary tools for their study.

cond-mat.dis-nn↗

arXiv2021

The emergence of a concept in shallow neural networks

We consider restricted Boltzmann machine (RBMs) trained over an unstructured dataset made of blurred copies of definite but unavailable ``archetypes'' and we show that there exists a critical sample size beyond which the RBM can learn archetypes, namely the machine can successfully play as a generative model or as a classifier, according to the operational routine. In general, assessing a critical sample size (possibly in relation to the quality of the dataset) is still an open problem in machine learning. Here, restricting to the random theory, where shallow networks suffice and the grand-mother cell scenario is correct, we leverage the formal equivalence between RBMs and Hopfield networks, to obtain a phase diagram for both the neural architectures which highlights regions, in the space of the control parameters (i.e., number of archetypes, number of neurons, size and quality of the training set), where learning can be accomplished. Our investigations are led by analytical methods based on the statistical-mechanics of disordered systems and results are further corroborated by extensive Monte Carlo simulations.

cond-mat.dis-nn↗

arXiv2021

Pattern recognition in Deep Boltzmann machines

We consider a multi-layer Sherrington-Kirkpatrick spin-glass as a model for deep restricted Boltzmann machines and we solve for its quenched free energy, in the thermodynamic limit and allowing for a first step of replica symmetry breaking. This result is accomplished rigorously exploiting interpolating techniques and recovering the expression already known for the replica-symmetry case. Further, we drop the restriction constraint by introducing intra-layer connections among spins and we show that the resulting system can be mapped into a modular Hopfield network, which is also addressed rigorously via interpolating techniques up to the first step of replica symmetry breaking.

cond-mat.dis-nn↗

arXiv2021

The relativistic Hopfield model with correlated patterns

In this work we introduce and investigate the properties of the "relativistic" Hopfield model endowed with temporally correlated patterns. First, we review the "relativistic" Hopfield model and we briefly describe the experimental evidence underlying correlation among patterns. Then, we face the study of the resulting model exploiting statistical-mechanics tools in a low-load regime. More precisely, we prove the existence of the thermodynamic limit of the related free-energy and we derive the self-consistence equations for its order parameters. These equations are solved numerically to get a phase diagram describing the performance of the system as an associative memory as a function of its intrinsic parameters (i.e., the degree of noise and of correlation among patterns). We find that, beyond the standard retrieval and ergodic phases, the relativistic system exhibits correlated and symmetric regions -- that are genuine effects of temporal correlation -- whose width is, respectively, reduced and increased with respect to the classical case.

math-ph↗

arXiv2020

Tolerance versus synaptic noise in dense associative memories

The retrieval capabilities of associative neural networks can be impaired by different kinds of noise: the fast noise (which makes neurons more prone to failure), the slow noise (stemming from interference among stored memories), and synaptic noise (due to possible flaws during the learning or the storing stage). In this work we consider dense associative neural networks, where neurons can interact in $p$-plets, in the absence of fast noise, and we investigate the interplay of slow and synaptic noise. In particular, leveraging on the duality between associative neural networks and restricted Boltzmann machines, we analyze the effect of corrupted information, imperfect learning and storing errors. For $p=2$ (corresponding to the Hopfield model) any source of synaptic noise breaks-down retrieval if the number of memories $K$ scales as the network size. For $p>2$, in the relatively low-load regime $K \sim N$, synaptic noise is tolerated up to a certain bound, depending on the density of the structure.

cond-mat.dis-nn↗

arXiv2020

Replica symmetry breaking in neural networks: a few steps toward rigorous results

In this paper we adapt the broken replica interpolation technique (developed by Francesco Guerra to deal with the Sherrington-Kirkpatrick model, namely a pairwise mean-field spin-glass whose couplings are i.i.d. standard Gaussian variables) in order to work also with the Hopfield model (i.e., a pairwise mean-field neural-network whose couplings are drawn according to Hebb's learning rule): this is accomplished by grafting Guerra's telescopic averages on the transport equation technique, recently developed by some of the Authors. As an overture, we apply the technique to solve the Sherrington-Kirkpatrick model with i.i.d. Gaussian couplings centered at $J_0$ and with finite variance $J$; the mean $J_0$ plays the role of a signal to be detected in a noisy environment tuned by $J$, hence making this model a natural test-case to be investigated before addressing the Hopfield model. For both the models, an explicit expression of their quenched free energy in terms of their natural order parameters is obtained at the K-th step (K arbitrary, but finite) of replica-symmetry-breaking. In particular, for the Hopfield model, by assuming that the overlaps respect Parisi's decomposition (following the ziqqurat ansatz) and that the Mattis magnetization is self-averaging, we recover previous results obtained via replica-trick by Amit, Crisanti and Gutfreund (1RSB) and by Steffan and Kühn (2RSB).

math-ph↗

arXiv2020

Generalized Guerra's interpolation schemes for dense associative neural networks

In this work we develop analytical techniques to investigate a broad class of associative neural networks set in the high-storage regime. These techniques translate the original statistical-mechanical problem into an analytical-mechanical one which implies solving a set of partial differential equations, rather than tackling the canonical probabilistic route. We test the method on the classical Hopfield model - where the cost function includes only two-body interactions (i.e., quadratic terms) - and on the "relativistic" Hopfield model - where the (expansion of the) cost function includes p-body (i.e., of degree p) contributions. Under the replica symmetric assumption, we paint the phase diagrams of these models by obtaining the explicit expression of their free energy as a function of the model parameters (i.e., noise level and memory storage). Further, since for non-pairwise models ergodicity breaking is non necessarily a critical phenomenon, we develop a fluctuation analysis and find that criticality is preserved in the relativistic model.

cond-mat.dis-nn↗

arXiv2019

A statistical-inference approach to reconstruct inter-cellular interactions in cell-migration experiments

Migration of cells can be characterized by two, prototypical types of motion: individual and collective migration. We propose a statistical-inference approach designed to detect the presence of cell-cell interactions that give rise to collective behaviors in cell-motility experiments. Such inference method has been first successfully tested on synthetic motional data, and then applied to two experiments. In the first experiment, cell migrate in a wound-healing model: when applied to this experiment, the inference method predicts the existence of cell-cell interactions, correctly mirroring the strong intercellular contacts which are present in the experiment. In the second experiment, dendritic cells migrate in a chemokine gradient. Our inference analysis does not provide evidence for interactions, indicating that cells migrate by sensing independently the chemokine source. According to this prediction, we speculate that mature dendritic cells disregard inter-cellular signals that could otherwise delay their arrival to lymph vessels.

q-bio.CB↗

arXiv2019

Neural networks with redundant representation: detecting the undetectable

We consider a three-layer Sejnowski machine and show that features learnt via contrastive divergence have a dual representation as patterns in a dense associative memory of order P=4. The latter is known to be able to Hebbian-store an amount of patterns scaling as N^{P-1}, where N denotes the number of constituting binary neurons interacting P-wisely. We also prove that, by keeping the dense associative network far from the saturation regime (namely, allowing for a number of patterns scaling only linearly with N, while P>2) such a system is able to perform pattern recognition far below the standard signal-to-noise threshold. In particular, a network with P=4 is able to retrieve information whose intensity is O(1) even in the presence of a noise O(\sqrt{N}) in the large N limit. This striking skill stems from a redundancy representation of patterns -- which is afforded given the (relatively) low-load information storage -- and it contributes to explain the impressive abilities in pattern recognition exhibited by new-generation neural networks. The whole theory is developed rigorously, at the replica symmetric level of approximation, and corroborated by signal-to-noise analysis and Monte Carlo simulations.

cond-mat.dis-nn↗

arXiv2019

Exact calculations of first-passage properties on the pseudofractal scale-free web

In this paper, we consider discrete time random walks on the pseudofractal scale-free web (PSFW) and we study analytically the related first passage properties. First, we classify the nodes of the PSFW into different levels and propose a method to derive the generation function of the first passage probability from an arbitrary starting node to the absorbing domain, which is located at one or more nodes of low-level (i.e., nodes with large degree). Then, we calculate exactly the first passage probability, the survival probability, the mean and the variance of first passage time by using the generating functions as a tool. Finally, for some illustrative examples corresponding to given choices of starting node and absorbing domain, we derive exact and explicit results for such first passage properties. The method we propose can as well address the cases where the absorbing domain is located at one or more nodes of high-level on the PSFW, and it can also be used to calculate the first passage properties on other networks with self-similar structure, such as $(u, v)$ flowers and recursive scale-free trees.

cond-mat.stat-mech↗

arXiv2019

Scaling laws for diffusion on (trans)fractal scale-free networks

Fractal (or transfractal) features are common in real-life networks and are known to influence the dynamic processes taking place in the network itself. Here we consider a class of scale-free deterministic networks, called $(u,v)$-flowers, whose topological properties can be controlled by tuning the parameters $u$ and $v$; in particular, for $u>1$, they are fractals endowed with a fractal dimension $d_f$, while for $u=1$, they are transfractal endowed with a transfractal dimension $\tilde{d}_f$. In this work we investigate dynamic processes (i.e., random walks) and topological properties (i.e., the Laplacian spectrum) and we show that, under proper conditions, the same scalings (ruled by the related dimensions), emerge for both fractal and transfractal.

cond-mat.stat-mech↗

arXiv2019

Exact results for the first-passage properties in a class of fractal networks

In this work we consider a class of recursively-grown fractal networks $G_n(t)$, whose topology is controlled by two integer parameters $t$ and $n$. We first analyse the structural properties of $G_n(t)$ (including fractal dimension, modularity and clustering coefficient) and then we move to its transport properties. The latter are studied in terms of first-passage quantities (including the mean trapping time, the global mean first-passage time and the Kemeny's constant) and we highlight that their asymptotic behavior is controlled by network's size and diameter. Remarkably, if we tune $n$ (or, analogously, $t$) while keeping the network size fixed, as $n$ increases ($t$ decreases) the network gets more and more clustered and modular, while its diameter is reduced, implying, ultimately, a better transport performance. The connection between this class of networks and models for polymer architectures is also discussed.

cond-mat.stat-mech↗

arXiv2019

Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit

Restricted Boltzmann machines (RBMs) constitute one of the main models for machine statistical inference and they are widely employed in Artificial Intelligence as powerful tools for (deep) learning. However, in contrast with countless remarkable practical successes, their mathematical formalization has been largely elusive: from a statistical-mechanics perspective these systems display the same (random) Gibbs measure of bi-partite spin-glasses, whose rigorous treatment is notoriously difficult. In this work, beyond providing a brief review on RBMs from both the learning and the retrieval perspectives, we aim to contribute to their analytical investigation, by considering two distinct realizations of their weights (i.e., Boolean and Gaussian) and studying the properties of their related free energies. More precisely, focusing on a RBM characterized by digital couplings, we first extend the Pastur-Shcherbina-Tirozzi method (originally developed for the Hopfield model) to prove the self-averaging property for the free energy, over its quenched expectation, in the infinite volume limit, then we explicitly calculate its simplest approximation, namely its annealed bound. Next, focusing on a RBM characterized by analogical weights, we extend Guerra's interpolating scheme to obtain a control of the quenched free-energy under the assumption of replica symmetry: we get self-consistencies for the order parameters (in full agreement with the existing Literature) as well as the critical line for ergodicity breaking that turns out to be the same obtained in AGS theory. As we discuss, this analogy stems from the slow-noise universality. Finally, glancing beyond replica symmetry, we analyze the fluctuations of the overlaps for an estimate of the (slow) noise affecting the retrieval of the signal, and by a stability analysis we recover the Aizenman-Contucci identities typical of glassy systems.

cond-mat.dis-nn↗

arXiv2018

Dreaming neural networks: rigorous results

Recently a daily routine for associative neural networks has been proposed: the network Hebbian-learns during the awake state (thus behaving as a standard Hopfield model), then, during its sleep state, optimizing information storage, it consolidates pure patterns and removes spurious ones: this forces the synaptic matrix to collapse to the projector one (ultimately approaching the Kanter-Sompolinksy model). This procedure keeps the learning Hebbian-based (a biological must) but, by taking advantage of a (properly stylized) sleep phase, still reaches the maximal critical capacity (for symmetric interactions). So far this emerging picture (as well as the bulk of papers on unlearning techniques) was supported solely by mathematically-challenging routes, e.g. mainly replica-trick analysis and numerical simulations: here we rely extensively on Guerra's interpolation techniques developed for neural networks and, in particular, we extend the generalized stochastic stability approach to the case. Confining our description within the replica symmetric approximation (where the previous ones lie), the picture painted regarding this generalization (and the previously existing variations on theme) is here entirely confirmed. Further, still relying on Guerra's schemes, we develop a systematic fluctuation analysis to check where ergodicity is broken (an analysis entirely absent in previous investigations). We find that, as long as the network is awake, ergodicity is bounded by the Amit-Gutfreund-Sompolinsky critical line (as it should), but, as the network sleeps, sleeping destroys spin glass states by extending both the retrieval as well as the ergodic region: after an entire sleeping session the solely surviving regions are retrieval and ergodic ones and this allows the network to achieve the perfect retrieval regime (the number of storable patterns equals the number of neurons in the network).

cond-mat.dis-nn↗

arXiv2018

A novel derivation of the Marchenko-Pastur law through analog bipartite spin-glasses

In this work we consider the {\em analog bipartite spin-glass} (or {\em real-valued restricted Boltzmann machine} in a neural network jargon), whose variables (those quenched as well as those dynamical) share standard Gaussian distributions. First, via Guerra's interpolation technique, we express its quenched free energy in terms of the natural order parameters of the theory (namely the self- and two-replica overlaps), then, we re-obtain the same result by using the replica-trick: a mandatory tribute, given the special occasion. Next, we show that the quenched free energy of this model is the functional generator of the moments of the correlation matrix among the weights connecting the two layers of the spin-glass (i.e., the Wishart matrix in random matrix theory or the Hebbian coupling in neural networks): as weights are quenched stochastic variables, this plays as a novel tool to inspect random matrices. In particular, we find that the Stieltjes transform of the spectral density of the correlation matrix is determined by the (replica-symmetric) quenched free energy of the bipartite spin-glass model. In this setup, we re-obtain the Marchenko-Pastur law in a very simple way.

cond-mat.dis-nn↗