arXiv subjects

Alberto Fachechi

Publications and source records attributed to Alberto Fachechi.

At least 19 recordsLinked to original sources

arXiv2026

Dense auto-hetero associative memories applied to noisy communication channels

Networks of interacting Hebbian networks have recently been shown to perform a task beyond associative memory, namely \emph{pattern disentanglement}: when fed with a spurious mixture of stored patterns, the different modules spontaneously specialize on, and retrieve, the different constituents of the mixture. So far, this capability has only been established for pairwise interactions, which limits the number of patterns that can be handled. Here we introduce a dense extension of these modular networks, in which both the auto-associative couplings within each module and the hetero-associative couplings among modules are promoted to higher-order Hebbian interactions. We show that, with a suitable choice of the interaction orders, the network disentangles mixtures while storing a number of patterns that scales linearly with the module size, a regime where its pairwise counterpart fails. Through a statistical-mechanical analysis based on Guerra's interpolation, we derive the self-consistency equations for the order parameters and draw the phase diagrams identifying the region where disentanglement is achieved; these predictions are confirmed by Monte Carlo simulations. Finally, we show that disentanglement provides a natural decoding primitive, and we illustrate it with two applications: the explicit reconstruction of all the hidden patterns from the Hebbian tensors and a stream of unlabeled mixtures, and a proof-of-concept communication protocol in which each message token is transmitted as a masked mixture of hidden patterns and decoded by the network dynamics. Owing to its attractor-based decoding, the protocol degrades gracefully under strong channel corruption, where conventional secure-transmission pipelines fail abruptly.

cond-mat.dis-nn↗

arXiv2026

The infinite volume limit of the Hopfield neural networks in the replica-symmetric regime

Despite extensive research over the past five decades, a rigorous determination of the infinite-volume limit of the free energy of the Hopfield model with an extensive number of patterns has remained an outstanding open problem as standard methods from spin-glass theory have so far proven inadequate for this purpose. While a complete proof for finite, non-vanishing, loads is still out of reach, here we develop a novel interpolation strategy that allows us to express the free energy of the Hopfield model in terms of a linear combination of the free energies of hard- and soft-spin glass models along with a remainder contribution. The payoff of this interpolation scheme is twofold: (i) As the remainder involves positive-definite fluctuations of the order parameters (and it vanishes under a concentration assumption on their means where we consistently recover the replica-symmetric picture) such an expression finally constitutes a rigorous {\em bound} on the exact free energy, rather than just an {\em approximation,} as it was the case for earlier interpolation procedures à la Guerra. \newline (ii) As for the spin-glass models the existence of the infinite-volume free energy has been previously determined by the Guerra--Toninelli scheme, this novel interpolation allows us to extend its validity to the Hopfield model too. Yet, as the equivalence between the Hopfield model and mixtures of spin-glasses is currently well established under replica symmetry and up to the first step of replica symmetry breaking, at present our approach can be rigorously implemented only within these regimes. \newline In this work, by inspecting how the order parameters have to self-average around their means, we restrict our attention to the replica-symmetric regime.

cond-mat.dis-nn↗

arXiv2026

Spectral criteria for generalization in unsupervised Hebbian nets

We consider an unsupervised Hebbian network where the pairwise interactions among neurons are built on noisy realizations of hidden ground-truth vectors. Unlike classical Hopfield models, designed as memory devices, this class of networks can be employed to extract latent structure and generalize beyond the "training" set. By combining random matrix theory and replica methods, we derive the asymptotic spectrum of the corresponding interaction matrix and show that the onset of generalization is controlled by a sharp spectral transition. Depending on the quality and the size of the accessible dataset, the spectrum displays either two separated bulks, encoding informative and noisy directions, or a merged single-bulk phase where such distinction is lost. We show that, when coupled with regularization, the emergence of such a spectral split predicts the network's capability to reconstruct the ground-truth vectors from corrupted samples.

cond-mat.dis-nn↗

arXiv2025

The importance of being empty: a spectral approach to Hopfield neural networks with diluted examples

We consider Hopfield networks, where neurons interact pair-wise by Hebbian couplings built over $i$. a set of definite patterns (ground truths), $ii$. a sample of labeled examples (supervised setting), $iii$. a sample of unlabeled examples (unsupervised setting). We focus on the case where ground-truths are Rademacher vectors and examples are noisy versions of these ground-truths, possibly displaying some blank entries (e.g., mimicking missing or dropped data), and we determine the spectral distribution of the coupling matrices in the three scenarios, by exploiting and extending the Marchenko-Pastur theorem. By levering this knowledge, we are able to analytically inspect the stability and attractiveness of the ground truths, as well as the generalization capabilities of the networks. In particular, as corroborated by long-running Monte Carlo simulations, the presence of black entries can have benefits in some specific conditions, suggesting strategies based on data sparsification; the robustness of these results in structured datasets is confirmed numerically. Finally, we demonstrate that the Hebbian matrix, built on sparse examples, can be recovered as the fixed point of a gradient descent algorithm with dropout, over a suitable loss function.

cond-mat.dis-nn↗

arXiv2025

Multi-channel pattern reconstruction through $L$-directional associative memories

We consider $L$-directional associative memories, composed of $L$ Hopfield networks, displaying imitative Hebbian intra-network interactions and anti-imitative Hebbian inter-network interactions, where couplings are built over a set of hidden binary patterns. We evaluate the model's performance in reconstructing the whole set of hidden binary patterns when provided with mixtures of noisy versions of these patterns. Our numerical results demonstrate the model's high effectiveness in the reconstruction task for structureless and structured datasets.

cond-mat.dis-nn↗

arXiv2024

Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines

We consider restricted Boltzmann machines with a binary visible layer and a Gaussian hidden layer trained by an unlabelled dataset composed of noisy realizations of a single ground pattern. We develop a statistical mechanics framework to describe the network generative capabilities, by exploiting the replica trick and assuming self-averaging of the underlying order parameters (i.e., replica symmetry). In particular, we outline the effective control parameters (e.g., the relative number of weights to be trained, the regularization parameter), whose tuning can yield qualitatively-different operative regimes. Further, we provide analytical and numerical evidence for the existence of a sub-region in the space of the hyperparameters where replica-symmetry breaking occurs.

cond-mat.dis-nn↗

arXiv2024

Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting

In this work we approach attractor neural networks from a machine learning perspective: we look for optimal network parameters by applying a gradient descent over a regularized loss function. Within this framework, the optimal neuron-interaction matrices turn out to be a class of matrices which correspond to Hebbian kernels revised by a reiterated unlearning protocol. Remarkably, the extent of such unlearning is proved to be related to the regularization hyperparameter of the loss function and to the training time. Thus, we can design strategies to avoid overfitting that are formulated in terms of regularization and early-stopping tuning. The generalization capabilities of these attractor networks are also investigated: analytical results are obtained for random synthetic datasets, next, the emerging picture is corroborated by numerical experiments that highlight the existence of several regimes (i.e., overfitting, failure and success) as the dataset parameters are varied.

cs.LG↗

arXiv2024

A spectral approach to Hebbian-like neural networks

We consider the Hopfield neural network as a model of associative memory and we define its neuronal interaction matrix $\mathbf{J}$ as a function of a set of $K \times M$ binary vectors $\{\mathbfξ^{μ, A} \}_{μ=1,...,K}^{A=1,...,M}$ representing a sample of the reality that we want to retrieve. In particular, any item $\mathbfξ^{μ, A}$ is meant as a corrupted version of an unknown ground pattern $\mathbfζ^μ$, that is the target of our retrieval process. We consider and compare two definitions for $\mathbf{J}$, referred to as supervised and unsupervised, according to whether the class $μ$, each example belongs to, is unveiled or not, also, these definitions recover the paradigmatic Hebb's rule under suitable limits. The spectral properties of the resulting matrices are studied and used to inspect the retrieval capabilities of the related models as a function of their control parameters.

math-ph↗

arXiv2022

Pavlov Learning Machines

As well known, Hebb's learning traces its origin in Pavlov's Classical Conditioning, however, while the former has been extensively modelled in the past decades (e.g., by Hopfield model and countless variations on theme), as for the latter modelling has remained largely unaddressed so far; further, a bridge between these two pillars is totally lacking. The main difficulty towards this goal lays in the intrinsically different scales of the information involved: Pavlov's theory is about correlations among \emph{concepts} that are (dynamically) stored in the synaptic matrix as exemplified by the celebrated experiment starring a dog and a ring bell; conversely, Hebb's theory is about correlations among pairs of adjacent neurons as summarized by the famous statement {\em neurons that fire together wire together}. In this paper we rely on stochastic-process theory and model neural and synaptic dynamics via Langevin equations, to prove that -- as long as we keep neurons' and synapses' timescales largely split -- Pavlov mechanism spontaneously takes place and ultimately gives rise to synaptic weights that recover the Hebbian kernel.

cond-mat.dis-nn↗

arXiv2022

Non-linear PDEs approach to statistical mechanics of Dense Associative Memories

Dense associative memories (DAM), are widespread models in artificial intelligence used for pattern recognition tasks; computationally, they have been proven to be robust against adversarial input and theoretically, leveraging their analogy with spin-glass systems, they are usually treated by means of statistical-mechanics tools. Here we develop analytical methods, based on nonlinear PDEs, to investigate their functioning. In particular, we prove differential identities involving DAM partition function and macroscopic observables useful for a qualitative and quantitative analysis of the system. These results allow for a deeper comprehension of the mechanisms underlying DAMs and provide interdisciplinary tools for their study.

cond-mat.dis-nn↗

arXiv2021

Pattern recognition in Deep Boltzmann machines

We consider a multi-layer Sherrington-Kirkpatrick spin-glass as a model for deep restricted Boltzmann machines and we solve for its quenched free energy, in the thermodynamic limit and allowing for a first step of replica symmetry breaking. This result is accomplished rigorously exploiting interpolating techniques and recovering the expression already known for the replica-symmetry case. Further, we drop the restriction constraint by introducing intra-layer connections among spins and we show that the resulting system can be mapped into a modular Hopfield network, which is also addressed rigorously via interpolating techniques up to the first step of replica symmetry breaking.

cond-mat.dis-nn↗

arXiv2021

PDE/statistical mechanics duality: relation between Guerra's interpolated $p$-spin ferromagnets and the Burgers hierarchy

We examine the duality relating the equilibrium dynamics of the mean-field $p$-spin ferromagnets at finite size in the Guerra's interpolation scheme and the Burgers hierarchy. In particular, we prove that - for fixed $p$ - the expectation value of the order parameter on the first side w.r.t. the generalized partition function satisfies the $p-1$-th element in the aforementioned class of nonlinear equations. In the light of this duality, we interpret the phase transitions in the thermodynamic limit of the statistical mechanics model with the development of shock waves in the PDE side. We also obtain the solutions for the $p$-spin ferromagnets at fixed $N$, allowing us to easily generate specific solutions of the corresponding equation in the Burgers hierarchy. Finally, we obtain an effective description of the finite $N$ equilibrium dynamics of the $p=2$ model with some standard tools in PDE side.

cond-mat.stat-mech↗

arXiv2021

The relativistic Hopfield model with correlated patterns

In this work we introduce and investigate the properties of the "relativistic" Hopfield model endowed with temporally correlated patterns. First, we review the "relativistic" Hopfield model and we briefly describe the experimental evidence underlying correlation among patterns. Then, we face the study of the resulting model exploiting statistical-mechanics tools in a low-load regime. More precisely, we prove the existence of the thermodynamic limit of the related free-energy and we derive the self-consistence equations for its order parameters. These equations are solved numerically to get a phase diagram describing the performance of the system as an associative memory as a function of its intrinsic parameters (i.e., the degree of noise and of correlation among patterns). We find that, beyond the standard retrieval and ergodic phases, the relativistic system exhibits correlated and symmetric regions -- that are genuine effects of temporal correlation -- whose width is, respectively, reduced and increased with respect to the classical case.

math-ph↗

arXiv2020

Generalized Guerra's interpolation schemes for dense associative neural networks

In this work we develop analytical techniques to investigate a broad class of associative neural networks set in the high-storage regime. These techniques translate the original statistical-mechanical problem into an analytical-mechanical one which implies solving a set of partial differential equations, rather than tackling the canonical probabilistic route. We test the method on the classical Hopfield model - where the cost function includes only two-body interactions (i.e., quadratic terms) - and on the "relativistic" Hopfield model - where the (expansion of the) cost function includes p-body (i.e., of degree p) contributions. Under the replica symmetric assumption, we paint the phase diagrams of these models by obtaining the explicit expression of their free energy as a function of the model parameters (i.e., noise level and memory storage). Further, since for non-pairwise models ergodicity breaking is non necessarily a critical phenomenon, we develop a fluctuation analysis and find that criticality is preserved in the relativistic model.

cond-mat.dis-nn↗

arXiv2019

Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories

Recently, Hopfield and Krotov introduced the concept of {\em dense associative memories} [DAM] (close to spin-glasses with $P$-wise interactions in a disordered statistical mechanical jargon): they proved a number of remarkable features these networks share and suggested their use to (partially) explain the success of the new generation of Artificial Intelligence. Thanks to a remarkable ante-litteram analysis by Baldi \& Venkatesh, among these properties, it is known these networks can handle a maximal amount of stored patterns $K$ scaling as $K \sim N^{P-1}$.\\ In this paper, once introduced a {\em minimal dense associative network} as one of the most elementary cost-functions falling in this class of DAM, we sacrifice this high-load regime -namely we force the storage of {\em solely} a linear amount of patterns, i.e. $K = αN$ (with $α>0$)- to prove that, in this regime, these networks can correctly perform pattern recognition even if pattern signal is $O(1)$ and is embedded in a sea of noise $O(\sqrt{N})$, also in the large $N$ limit. To prove this statement, by extremizing the quenched free-energy of the model over its natural order-parameters (the various magnetizations and overlaps), we derived its phase diagram, at the replica symmetric level of description and in the thermodynamic limit: as a sideline, we stress that, to achieve this task, aiming at cross-fertilization among disciplines, we pave two hegemon routes in the statistical mechanics of spin glasses, namely the replica trick and the interpolation technique.\\ Both the approaches reach the same conclusion: there is a not-empty region, in the noise-$T$ vs load-$α$ phase diagram plane, where these networks can actually work in this challenging regime; in particular we obtained a quite high critical (linear) load in the (fast) noiseless case resulting in $\lim_{β\to \infty}α_c(β)=0.65$.

cond-mat.dis-nn↗

arXiv2019

Neural networks with redundant representation: detecting the undetectable

We consider a three-layer Sejnowski machine and show that features learnt via contrastive divergence have a dual representation as patterns in a dense associative memory of order P=4. The latter is known to be able to Hebbian-store an amount of patterns scaling as N^{P-1}, where N denotes the number of constituting binary neurons interacting P-wisely. We also prove that, by keeping the dense associative network far from the saturation regime (namely, allowing for a number of patterns scaling only linearly with N, while P>2) such a system is able to perform pattern recognition far below the standard signal-to-noise threshold. In particular, a network with P=4 is able to retrieve information whose intensity is O(1) even in the presence of a noise O(\sqrt{N}) in the large N limit. This striking skill stems from a redundancy representation of patterns -- which is afforded given the (relatively) low-load information storage -- and it contributes to explain the impressive abilities in pattern recognition exhibited by new-generation neural networks. The whole theory is developed rigorously, at the replica symmetric level of approximation, and corroborated by signal-to-noise analysis and Monte Carlo simulations.

cond-mat.dis-nn↗

arXiv2018

Dreaming neural networks: rigorous results

Recently a daily routine for associative neural networks has been proposed: the network Hebbian-learns during the awake state (thus behaving as a standard Hopfield model), then, during its sleep state, optimizing information storage, it consolidates pure patterns and removes spurious ones: this forces the synaptic matrix to collapse to the projector one (ultimately approaching the Kanter-Sompolinksy model). This procedure keeps the learning Hebbian-based (a biological must) but, by taking advantage of a (properly stylized) sleep phase, still reaches the maximal critical capacity (for symmetric interactions). So far this emerging picture (as well as the bulk of papers on unlearning techniques) was supported solely by mathematically-challenging routes, e.g. mainly replica-trick analysis and numerical simulations: here we rely extensively on Guerra's interpolation techniques developed for neural networks and, in particular, we extend the generalized stochastic stability approach to the case. Confining our description within the replica symmetric approximation (where the previous ones lie), the picture painted regarding this generalization (and the previously existing variations on theme) is here entirely confirmed. Further, still relying on Guerra's schemes, we develop a systematic fluctuation analysis to check where ergodicity is broken (an analysis entirely absent in previous investigations). We find that, as long as the network is awake, ergodicity is bounded by the Amit-Gutfreund-Sompolinsky critical line (as it should), but, as the network sleeps, sleeping destroys spin glass states by extending both the retrieval as well as the ergodic region: after an entire sleeping session the solely surviving regions are retrieval and ergodic ones and this allows the network to achieve the perfect retrieval regime (the number of storable patterns equals the number of neurons in the network).

cond-mat.dis-nn↗

arXiv2018

A novel derivation of the Marchenko-Pastur law through analog bipartite spin-glasses

In this work we consider the {\em analog bipartite spin-glass} (or {\em real-valued restricted Boltzmann machine} in a neural network jargon), whose variables (those quenched as well as those dynamical) share standard Gaussian distributions. First, via Guerra's interpolation technique, we express its quenched free energy in terms of the natural order parameters of the theory (namely the self- and two-replica overlaps), then, we re-obtain the same result by using the replica-trick: a mandatory tribute, given the special occasion. Next, we show that the quenched free energy of this model is the functional generator of the moments of the correlation matrix among the weights connecting the two layers of the spin-glass (i.e., the Wishart matrix in random matrix theory or the Hebbian coupling in neural networks): as weights are quenched stochastic variables, this plays as a novel tool to inspect random matrices. In particular, we find that the Stieltjes transform of the spectral density of the correlation matrix is determined by the (replica-symmetric) quenched free energy of the bipartite spin-glass model. In this setup, we re-obtain the Marchenko-Pastur law in a very simple way.

cond-mat.dis-nn↗