arXiv subjects

Aurélien Decelle

Publications and source records attributed to Aurélien Decelle.

At least 37 records · Page 2Linked to original sources

arXiv2023

Explaining the effects of non-convergent sampling in the training of Energy-Based Models

In this paper, we quantify the impact of using non-convergent Markov chains to train Energy-Based models (EBMs). In particular, we show analytically that EBMs trained with non-persistent short runs to estimate the gradient can perfectly reproduce a set of empirical statistics of the data, not at the level of the equilibrium measure, but through a precise dynamical process. Our results provide a first-principles explanation for the observations of recent works proposing the strategy of using short runs starting from random initial conditions as an efficient way to generate high-quality samples in EBMs, and lay the groundwork for using EBMs as diffusion models. After explaining this effect in generic EBMs, we analyze two solvable models in which the effect of the non-convergent sampling in the trained parameters can be described in detail. Finally, we test these predictions numerically on a ConvNet EBM and a Boltzmann machine.

cs.LG↗

arXiv2023

Thermodynamics of bidirectional associative memories

In this paper we investigate the equilibrium properties of bidirectional associative memories (BAMs). Introduced by Kosko in 1988 as a generalization of the Hopfield model to a bipartite structure, the simplest architecture is defined by two layers of neurons, with synaptic connections only between units of different layers: even without internal connections within each layer, information storage and retrieval are still possible through the reverberation of neural activities passing from one layer to another. We characterize the computational capabilities of a stochastic extension of this model in the thermodynamic limit, by applying rigorous techniques from statistical physics. A detailed picture of the phase diagram at the replica symmetric level is provided, both at finite temperature and in the noiseless regimes. Also for the latter, the critical load is further investigated up to one step of replica symmetry breaking. An analytical and numerical inspection of the transition curves (namely critical lines splitting the various modes of operation of the machine) is carried out as the control parameters - noise, load and asymmetry between the two layer sizes - are tuned. In particular, with a finite asymmetry between the two layers, it is shown how the BAM can store information more efficiently than the Hopfield model by requiring less parameters to encode a fixed number of patterns. Comparisons are made with numerical simulations of neural dynamics. Finally, a low-load analysis is carried out to explain the retrieval mechanism in the BAM by analogy with two interacting Hopfield models. A potential equivalence with two coupled Restricted Boltmzann Machines is also discussed.

cond-mat.dis-nn↗

arXiv2022

Cosmology with cosmic web environments II. Redshift-space auto and cross power spectra

Degeneracies among parameters of the cosmological model are known to drastically limit the information contained in the matter distribution. In the first paper of this series, we shown that the cosmic web environments; namely the voids, walls, filaments and nodes; can be used as a leverage to improve the real-space constraints on a set of six cosmological parameters, including the summed neutrino mass. Following-upon these results, we propose to study the achievable constraints of environment-dependent power spectra in redshift space where the velocities add up information to the standard two-point statistics by breaking the isotropy of the matter density field. A Fisher analysis based on a set of thousands of Quijote simulations allows us to conclude that the combination of power spectra computed in the several cosmic web environments is able to break some degeneracies. Compared to the matter monopole and quadrupole information alone, the combination of environment-dependent spectra tightens down the constraints on key parameters like the matter density or the summed neutrino mass by up to a factor of $5.5$. Additionally, while the information contained in the matter statistic quickly saturates at mildly non-linear scales in redshift space, the combination of power spectra in the environments appears as a goldmine of information able to improve the constraints at all the studied scales from $0.1$ to $0.5$ $h$/Mpc and suggests that further improvements are reachable at even finer scales.

astro-ph.CO↗

arXiv2022

Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling

Restricted Boltzmann Machines are simple and powerful generative models that can encode any complex dataset. Despite all their advantages, in practice the trainings are often unstable and it is difficult to assess their quality because the dynamics are affected by extremely slow time dependencies. This situation becomes critical when dealing with low-dimensional clustered datasets, where the time required to sample ergodically the trained models becomes computationally prohibitive. In this work, we show that this divergence of Monte Carlo mixing times is related to a phenomenon of phase coexistence, similar to that which occurs in physics near a first-order phase transition. We show that sampling the equilibrium distribution using the Markov chain Monte Carlo method can be dramatically accelerated when using biased sampling techniques, in particular the Tethered Monte Carlo (TMC) method. This sampling technique efficiently solves the problem of evaluating the quality of a given trained model and generating new samples in a reasonable amount of time. Moreover, we show that this sampling technique can also be used to improve the computation of the log-likelihood gradient during training, leading to dramatic improvements in training RBMs with artificial clustered datasets. On real low-dimensional datasets, this new training method fits RBM models with significantly faster relaxation dynamics than those obtained with standard PCD recipes. We also show that TMC sampling can be used to recover the free-energy profile of the RBM. This proves to be extremely useful to compute the probability distribution of a given model and to improve the generation of new decorrelated samples in slow PCD-trained models.

cs.LG↗

arXiv2022

Fundamental problems in statistical physics XIV: Lecture on Machine Learning

The recent progresses in Machine Learning opened the door to actual applications of learning algorithms but also to new research directions both in the field of Machine Learning directly and, at the edges with other disciplines. The case that interests us is the interface with physics, and more specifically Statistical Physics. In this short lecture, I will try to present first a brief introduction to Machine Learning from the angle of neural networks. After explaining quickly some fundamental models and global aspects of the training procedure, I will discuss into more detail two examples illustrate what can be done from the Statistical Physics perspective.

cond-mat.dis-nn↗

arXiv2021

Cosmology with cosmic web environments I. Real-space power spectra

We undertake the first comprehensive and quantitative real-space analysis of the cosmological information content in the environments of the cosmic web (voids, filaments, walls, and nodes) up to non-linear scales, $k = 0.5$ $h$/Mpc. Relying on the large set of $N$-body simulations from the Quijote suite, the environments are defined through the eigenvalues of the tidal tensor and the Fisher formalism is used to assess the constraining power of the power spectra derived in each of the four environments and their combination. Our results show that there is more information available in the environment-dependent power spectra, both individually and when combined all together, than in the matter power spectrum. By breaking some key degeneracies between parameters of the cosmological model such as $M_ν$--$σ_\mathrm{8}$ or $Ω_\mathrm{m}$--$σ_8$, the power spectra computed in identified environments improve the constraints on cosmological parameters by factors $\sim 15$ for the summed neutrino mass $M_ν$ and $\sim 8$ for the matter density $Ω_\mathrm{m}$ over those derived from the matter power spectrum. We show that these tighter constraints are obtained for a wide range of the maximum scale, from $k_\mathrm{max} = 0.1$ $h$/Mpc to highly non-linear regimes with $k_\mathrm{max} = 0.5$ $h$/Mpc. We also report an eight times higher value of the signal-to-noise ratio for the combination of spectra compared to the matter one. Importantly, we show that all the presented results are robust to variations of the parameters defining the environments hence suggesting a robustness to the definition we chose to define them.

astro-ph.CO↗

arXiv2021

Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines

Training Restricted Boltzmann Machines (RBMs) has been challenging for a long time due to the difficulty of computing precisely the log-likelihood gradient. Over the past decades, many works have proposed more or less successful training recipes but without studying the crucial quantity of the problem: the mixing time, i.e. the number of Monte Carlo iterations needed to sample new configurations from a model. In this work, we show that this mixing time plays a crucial role in the dynamics and stability of the trained model, and that RBMs operate in two well-defined regimes, namely equilibrium and out-of-equilibrium, depending on the interplay between this mixing time of the model and the number of steps, $k$, used to approximate the gradient. We further show empirically that this mixing time increases with the learning, which often implies a transition from one regime to another as soon as $k$ becomes smaller than this time. In particular, we show that using the popular $k$ (persistent) contrastive divergence approaches, with $k$ small, the dynamics of the learned model are extremely slow and often dominated by strong out-of-equilibrium effects. On the contrary, RBMs trained in equilibrium display faster dynamics, and a smooth convergence to dataset-like configurations during the sampling. Finally we discuss how to exploit in practice both regimes depending on the task one aims to fulfill: (i) short $k$ can be used to generate convincing samples in short learning times, (ii) large $k$ (or increasingly large) is needed to learn the correct equilibrium distribution of the RBM. Finally, the existence of these two operational regimes seems to be a general property of energy based models trained via likelihood maximization.

cs.LG↗

arXiv2021

Exact training of Restricted Boltzmann machines on intrinsically low dimensional data

The restricted Boltzmann machine is a basic machine learning tool able, in principle, to model the distribution of some arbitrary dataset. Its standard training procedure appears however delicate and obscure in many respects. We bring some new insights to it by considering the situation where the data have low intrinsic dimension, offering the possibility of an exact treatment and revealing a fundamental failure of the standard training procedure. The reasons for this failure \textemdash~like the occurrence of first-order phase transitions during training~\textemdash \ are clarified thanks to a Coulomb interactions reformulation of the model. In addition a convex relaxation of the original optimization problem is formulated thereby resulting in a unique solution, obtained in precise numerical form on $d=1,2$ study cases, while a constrained linear regression solution can be conjectured on the basis of an information theory argument.

cond-mat.dis-nn↗

arXiv2021

Restricted Boltzmann Machine, recent advances and mean-field theory

This review deals with Restricted Boltzmann Machine (RBM) under the light of statistical physics. The RBM is a classical family of Machine learning (ML) models which played a central role in the development of deep learning. Viewing it as a Spin Glass model and exhibiting various links with other models of statistical physics, we gather recent results dealing with mean-field theory in this context. First the functioning of the RBM can be analyzed via the phase diagrams obtained for various statistical ensembles of RBM leading in particular to identify a {\it compositional phase} where a small number of features or modes are combined to form complex patterns. Then we discuss recent works either able to devise mean-field based learning algorithms; either able to reproduce generic aspects of the learning process from some {\it ensemble dynamics equations} or/and from linear stability arguments.

cond-mat.dis-nn↗

arXiv2020

Encoding large scale cosmological structure with Generative Adversarial Networks

Recently a type of neural networks called Generative Adversarial Networks (GANs) has been proposed as a solution for fast generation of simulation-like datasets, in an attempt to bypass heavy computations and expensive cosmological simulations to run in terms of time and computing power. In the present work, we build and train a GAN to look further into the strengths and limitations of such an approach. We then propose a novel method in which we make use of a trained GAN to construct a simple autoencoder (AE) as a first step towards building a predictive model. Both the GAN and AE are trained on images issued from two types of N-body simulations, namely 2D and 3D simulations. We find that the GAN successfully generates new images that are statistically consistent with the images it was trained on. We then show that the AE manages to efficiently extract information from simulation images, satisfyingly inferring the latent encoding of the GAN to generate an image with similar large scale structures.

astro-ph.CO↗

arXiv2019

Robust Multi-Output Learning with Highly Incomplete Data via Restricted Boltzmann Machines

In a standard multi-output classification scenario, both features and labels of training data are partially observed. This challenging issue is widely witnessed due to sensor or database failures, crowd-sourcing and noisy communication channels in industrial data analytic services. Classic methods for handling multi-output classification with incomplete supervision information usually decompose the problem into an imputation stage that reconstructs the missing training information, and a learning stage that builds a classifier based on the imputed training set. These methods fail to fully leverage the dependencies between features and labels. In order to take full advantage of these dependencies we consider a purely probabilistic setting in which the features imputation and multi-label classification problems are jointly solved. Indeed, we show that a simple Restricted Boltzmann Machine can be trained with an adapted algorithm based on mean-field equations to efficiently solve problems of inductive and transductive learning in which both features and labels are missing at random. The effectiveness of the approach is demonstrated empirically on various datasets, with particular focus on a real-world Internet-of-Things security dataset.

cs.LG↗

arXiv2019

Gaussian-Spherical Restricted Boltzmann Machines

We consider a special type of Restricted Boltzmann machine (RBM), namely a Gaussian-spherical RBM where the visible units have Gaussian priors while the vector of hidden variables is constrained to stay on an ${\mathbbm L}_2$ sphere. The spherical constraint having the advantage to admit exact asymptotic treatments, various scaling regimes are explicitly identified based solely on the spectral properties of the coupling matrix (also called weight matrix of the RBM). Incidentally these happen to be formally related to similar scaling behaviours obtained in a different context dealing with spatial condensation of zero range processes. More specifically, when the spectrum of the coupling matrix is doubly degenerated an exact treatment can be proposed to deal with finite size effects. Interestingly the known parallel between the ferromagnetic transition of the spherical model and the Bose-Einstein condensation can be made explicit in that case. More importantly this gives us the ability to extract all needed response functions with arbitrary precision for the training algorithm of the RBM. This allows us then to numerically integrate the dynamics of the spectrum of the weight matrix during learning in a precise way. This dynamics reveals in particular a sequential emergence of modes from the Marchenko-Pastur bulk of singular vectors of the coupling matrix.

cond-mat.dis-nn↗

arXiv2019

Inverse problems for structured datasets using parallel TAP equations and RBM

We propose an efficient algorithm to solve inverse problems in the presence of binary clustered datasets. We consider the paradigmatic Hopfield model in a teacher student scenario, where this situation is found in the retrieval phase. This problem has been widely analyzed through various methods such as mean-field approaches or the pseudo-likelihood optimization. Our approach is based on the estimation of the posterior using the Thouless-Anderson-Palmer (TAP) equations in a parallel updating scheme. At the difference with other methods, it allows to retrieve the exact patterns of the teacher and the parallel update makes it possible to apply it for large system sizes. We also observe that the Approximate Message Passing (AMP) equations do not reproduce the expected behavior in the direct problem, questioning the standard practice used to obtain time indexes coming from Belief Propagation (BP). We tackle the same problem using a Restricted Boltzmann Machine (RBM) and discuss the analogies between the two algorithms.

cond-mat.dis-nn↗

arXiv2019

Learning a Local Symmetry with Neural-Networks

We explore the capacity of neural networks to detect a symmetry with complex local and non-local patterns : the gauge symmetry Z 2 . This symmetry is present in physical problems from topological transitions to QCD, and controls the computational hardness of instances of spin-glasses. Here, we show how to design a neural network, and a dataset, able to learn this symmetry and to find compressed latent representations of the gauge orbits. Our method pays special attention to system-wrapping loops, the so-called Polyakov loops, known to be particularly relevant for computational complexity.

cond-mat.dis-nn↗

arXiv2018

Thermodynamics of Restricted Boltzmann Machines and related learning dynamics

We investigate the thermodynamic properties of a Restricted Boltzmann Machine (RBM), a simple energy-based generative model used in the context of unsupervised learning. Assuming the information content of this model to be mainly reflected by the spectral properties of its weight matrix $W$, we try to make a realistic analysis by averaging over an appropriate statistical ensemble of RBMs. First, a phase diagram is derived. Otherwise similar to that of the Sherrington- Kirkpatrick (SK) model with ferromagnetic couplings, the RBM's phase diagram presents a ferromagnetic phase which may or may not be of compositional type depending on the kurtosis of the distribution of the components of the singular vectors of $W$. Subsequently, the learning dynamics of the RBM is studied in the thermodynamic limit. A "typical" learning trajectory is shown to solve an effective dynamical equation, based on the aforementioned ensemble average and explicitly involving order parameters obtained from the thermodynamic analysis. In particular, this let us show how the evolution of the dominant singular values of $W$, and thus of the unstable modes, is driven by the input data. At the beginning of the training, in which the RBM is found to operate in the linear regime, the unstable modes reflect the dominant covariance modes of the data. In the non-linear regime, instead, the selected modes interact and eventually impose a matching of the order parameters to their empirical counterparts estimated from the data. Finally, we illustrate our considerations by performing experiments on both artificial and real data, showing in particular how the RBM operates in the ferromagnetic compositional phase.

cond-mat.dis-nn↗

arXiv2017

Spectral Dynamics of Learning Restricted Boltzmann Machines

The Restricted Boltzmann Machine (RBM), an important tool used in machine learning in particular for unsupervized learning tasks, is investigated from the perspective of its spectral properties. Starting from empirical observations, we propose a generic statistical ensemble for the weight matrix of the RBM and characterize its mean evolution. This let us show how in the linear regime, in which the RBM is found to operate at the beginning of the training, the statistical properties of the data drive the selection of the unstable modes of the weight matrix. A set of equations characterizing the non-linear regime is then derived, unveiling in some way how the selected modes interact in later stages of the learning procedure and defining a deterministic learning curve for the RBM.

cond-mat.dis-nn↗

arXiv2016

Data quality for the inverse Ising problem

There are many methods proposed for inferring parameters of the Ising model from given data, that is a set of configurations generated according to the model itself. However little attention has been paid until now to the data, e.g. how the data is generated, whether the inference error using one set of data could be smaller than using another set of data, etc. In this paper we address the data quality problem in the kinetic inverse Ising problem. We quantify the quality of data using effective rank of the correlation matrix, and show that data gathered in a out of-equilibrium regime has a better quality than data gathered in equilibrium for coupling reconstruction. We also propose a matrix-perturbation based method for tuning the quality of given data and for removing bad-quality (i.e. redundant) configurations from data.

cond-mat.stat-mech↗

arXiv2016

Inference of the sparse kinetic Ising model using the decimation method

In this paper we study the inference of the kinetic Ising model on sparse graphs by the decimation method. The decimation method, which was first proposed in [Phys. Rev. Lett. 112, 070603] for the static inverse Ising problem, tries to recover the topology of the inferred system by setting the weakest couplings to zero iteratively. During the decimation process the likelihood function is maximized over the remaining couplings. Unlike the $\ell_1$-optimization based methods, the decimation method does not use the Laplace distribution as a heuristic choice of prior to select a sparse solution. In our case, the whole process can be done automatically without fixing any parameters by hand. We show that in the dynamical inference problem, where the task is to reconstruct the couplings of an Ising model given the data, the decimation process can be applied naturally into a maximum-likelihood optimization algorithm, as opposed to the static case where pseudo-likelihood method needs to be adopted. We also use extensive numerical studies to validate the accuracy of our methods in dynamical inference problems. Our results illustrate that on various topologies and with different distribution of couplings, the decimation method outperforms the widely-used $\ell _1$-optimization based methods.

cond-mat.dis-nn↗