Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function

Abstract

We study the expectation value of the logarithm of the partition function of large binary-to-binary lattice-gas Restricted Boltzmann Machines (RBMs) within a replica-symmetric ansatz, averaging over the disorder represented by the parameters of the RBM Hamiltonian. Averaging over the Hamiltonian parameters is done with a diagonal covariance matrix. Due to the diagonal form of the parameter covariance matrix not being preserved under the isomorphism between the Ising and lattice-gas forms of the RBM, we find differences in the behaviour of the quenched log partition function of the lattice-gas RBM compared to that of the Ising RBM form usually studied. We obtain explicit expressions for the expectation and variance of the lattice-gas RBM log partition function per node in the thermodynamic limit. We also obtain explicit expressions for the leading order finite size correction to the expected log partition function per node, and the threshold for the stability of the replica-symmetric approximation. We show that the stability threshold of the replica-symmetric approximation is equivalent, in the thermodynamic limit, to the stability threshold of a recent message-passing algorithm used to construct a mean-field Bethe approximation to the RBM free energy. Given the replica-symmetry assumption breaks down as the level of disorder in the spin-spin couplings increases, we obtain asymptotic expansions, in terms of the variance controlling this disorder, for the replica-symmetric log partition function and the replica-symmetric stability threshold. We confirm the various results derived using simulation.