arXiv subjects

Zohar Ringel

Publications and source records attributed to Zohar Ringel.

At least 37 records · Page 2Linked to original sources

arXiv2022

Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs

Deep neural networks (DNNs) are powerful tools for compressing and distilling information. Their scale and complexity, often involving billions of inter-dependent parameters, render direct microscopic analysis difficult. Under such circumstances, a common strategy is to identify slow variables that average the erratic behavior of the fast microscopic variables. Here, we identify a similar separation of scales occurring in fully trained finitely over-parameterized deep convolutional neural networks (CNNs) and fully connected networks (FCNs). Specifically, we show that DNN layers couple only through the second moment (kernels) of their activations and pre-activations. Moreover, the latter fluctuates in a nearly Gaussian manner. For infinite width DNNs, these kernels are inert, while for finite ones they adapt to the data and yield a tractable data-aware Gaussian Process. The resulting thermodynamic theory of deep learning yields accurate predictions in various settings. In addition, it provides new ways of analyzing and understanding DNNs in general.

stat.ML↗

arXiv2021

Statistical physics through the lens of real-space mutual information

Identifying the relevant coarse-grained degrees of freedom in a complex physical system is a key stage in developing powerful effective theories in and out of equilibrium. The celebrated renormalization group provides a framework for this task, but its practical execution in unfamiliar systems is fraught with ad hoc choices, whereas machine learning approaches, though promising, often lack formal interpretability. Recently, the optimal coarse-graining in a statistical system was shown to exist, based on a universal, but computationally difficult information-theoretic variational principle. This limited its applicability to but the simplest systems; moreover, the relation to standard formalism of field theory was unclear. Here we present an algorithm employing state-of-art results in machine-learning-based estimation of information-theoretic quantities, overcoming these challenges. We use this advance to develop a new paradigm in identifying the most relevant field theory operators describing properties of the system, going beyond the existing approaches to real-space renormalization. We evidence its power on an interacting model, where the emergent degrees of freedom are qualitatively different from the microscopic building blocks of the theory. Our results push the boundary of formally interpretable applications of machine learning, conceptually paving the way towards automated theory building.

cond-mat.stat-mech↗

arXiv2021

Symmetries and phase diagrams with real-space mutual information neural estimation

Real-space mutual information (RSMI) was shown to be an important quantity, formally and from a numerical standpoint, in finding coarse-grained descriptions of physical systems. It very generally quantifies spatial correlations, and can give rise to constructive algorithms extracting relevant degrees of freedom. Efficient and reliable estimation or maximization of RSMI is, however, numerically challenging. A recent breakthrough in theoretical machine learning has been the introduction of variational lower bounds for mutual information, parametrized by neural networks. Here we describe in detail how these results can be combined with differentiable coarse-graining operations to develop a single unsupervised neural-network based algorithm, the RSMI-NE, efficiently extracting the relevant degrees of freedom in the form of the operators of effective field theories, directly from real-space configurations. We study the information contained in the statistical ensemble of constructed coarse-graining transformations, and its recovery from partial input data using a secondary machine learning analysis applied to this ensemble. In particular, we show how symmetries, also emergent, can be identified. We demonstrate the extraction of the phase diagram and the order parameters for equilibrium systems, and consider also an example of a non-equilibrium problem.

cond-mat.stat-mech↗

arXiv2021

Predicting the outputs of finite deep neural networks trained with noisy gradients

A recent line of works studied wide deep neural networks (DNNs) by approximating them as Gaussian Processes (GPs). A DNN trained with gradient flow was shown to map to a GP governed by the Neural Tangent Kernel (NTK), whereas earlier works showed that a DNN with an i.i.d. prior over its weights maps to the so-called Neural Network Gaussian Process (NNGP). Here we consider a DNN training protocol, involving noise, weight decay and finite width, whose outcome corresponds to a certain non-Gaussian stochastic process. An analytical framework is then introduced to analyze this non-Gaussian process, whose deviation from a GP is controlled by the finite width. Our contribution is three-fold: (i) In the infinite width limit, we establish a correspondence between DNNs trained with noisy gradients and the NNGP, not the NTK. (ii) We provide a general analytical form for the finite width correction (FWC) for DNNs with arbitrary activation functions and depth and use it to predict the outputs of empirical finite networks with high accuracy. Analyzing the FWC behavior as a function of $n$, the training set size, we find that it is negligible for both the very small $n$ regime, and, surprisingly, for the large $n$ regime (where the GP error scales as $O(1/n)$). (iii) We flesh out algebraically how these FWCs can improve the performance of finite convolutional neural networks (CNNs) relative to their GP counterparts on image classification tasks.

stat.ML↗

arXiv2021

A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs

Deep neural networks (DNNs) in the infinite width/channel limit have received much attention recently, as they provide a clear analytical window to deep learning via mappings to Gaussian Processes (GPs). Despite its theoretical appeal, this viewpoint lacks a crucial ingredient of deep learning in finite DNNs, laying at the heart of their success -- feature learning. Here we consider DNNs trained with noisy gradient descent on a large training set and derive a self consistent Gaussian Process theory accounting for strong finite-DNN and feature learning effects. Applying this to a toy model of a two-layer linear convolutional neural network (CNN) shows good agreement with experiments. We further identify, both analytical and numerically, a sharp transition between a feature learning regime and a lazy learning regime in this model. Strong finite-DNN effects are also derived for a non-linear two-layer fully connected network. Our self consistent theory provides a rich and versatile analytical framework for studying feature learning and other non-lazy effects in finite DNNs.

cs.LG↗

arXiv2020

Relevance in the Renormalization Group and in Information Theory

The analysis of complex physical systems hinges on the ability to extract the relevant degrees of freedom from among the many others. Though much hope is placed in machine learning, it also brings challenges, chief of which is interpretability. It is often unclear what relation, if any, the architecture- and training-dependent learned "relevant" features bear to standard objects of physical theory. Here we report on theoretical results which may help to systematically address this issue: we establish equivalence between the information-theoretic notion of relevance defined in the Information Bottleneck (IB) formalism of compression theory, and the field-theoretic relevance of the Renormalization Group. We show analytically that for statistical physical systems described by a field theory the "relevant" degrees of freedom found using IB compression indeed correspond to operators with the lowest scaling dimensions. We confirm our field theoretic predictions numerically. We study dependence of the IB solutions on the physical symmetries of the data. Our findings provide a dictionary connecting two distinct theoretical toolboxes, and an example of constructively incorporating physical interpretability in applications of deep learning in physics.

cond-mat.stat-mech↗

arXiv2020

Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective

In the past decade, deep neural networks (DNNs) came to the fore as the leading machine learning algorithms for a variety of tasks. Their raise was founded on market needs and engineering craftsmanship, the latter based more on trial and error than on theory. While still far behind the application forefront, the theoretical study of DNNs has recently made important advancements in analyzing the highly over-parameterized regime where some exact results have been obtained. Leveraging these ideas and adopting a more physics-like approach, here we construct a versatile field-theory formalism for supervised deep learning, involving renormalization group, Feynman diagrams and replicas. In particular we show that our approach leads to highly accurate predictions of learning curves of truly deep DNNs trained on polynomial regression tasks and that these predictions can be used for efficient hyper-parameter optimization. In addition, they explain how DNNs generalize well despite being highly over-parameterized, this due to an entropic bias to simple functions which, for the case of fully-connected DNNs with data sampled on the hypersphere, are low order polynomials in the input vector. Being a complex interacting system of artificial neurons, we believe that such tools and methodologies borrowed from condensed matter physics would prove essential for obtaining an accurate quantitative understanding of deep learning.

cs.LG↗

arXiv2020

Intrinsic sign problems in topological quantum field theories

The sign problem is a widespread numerical hurdle preventing us from simulating the equilibrium behavior of various problems at the forefront of physics. Focusing on an important sub-class of such problems, bosonic $(2+1)$-dimensional topological quantum field theories, here we provide a simple criterion to diagnose intrinsic sign problems---that is, sign problems that are inherent to that phase of matter and cannot be removed by any local unitary transformation. Explicitly, \textit{if the exchange statistics of the anyonic excitations do not form complete sets of roots of unity, then the model has an intrinsic sign problem}. This establishes a concrete connection between the statistics of anyons, contained in the modular $S$ and $T$ matrices, and the presence of a sign problem in a microscopic Hamiltonian. Furthermore, it places constraints on the phases that can be realised by stoquastic Hamiltonians. We prove this and a more restrictive criterion for the large set of gapped bosonic models described by an abelian topological quantum field theory at low-energy, and offer evidence that it applies more generally with analogous results for non-abelian and chiral theories.

cond-mat.str-el↗

arXiv2020

Intrinsic sign problem in fermionic and bosonic chiral topological matter

The infamous sign problem leads to an exponential complexity in Monte Carlo simulations of generic many-body quantum systems. Nevertheless, many phases of matter are known to admit a sign-problem-free representative, allowing efficient simulations on classical computers. Motivated by long standing open problems in many-body physics, as well as fundamental questions in quantum complexity, the possibility of intrinsic sign problems, where a phase of matter admits no sign-problem-free representative, was recently raised but remains largely unexplored. Here, we establish the existence of an intrinsic sign problem in a broad class of gapped, chiral, topological phases of matter. Within this class, we exclude the possibility of stoquastic Hamiltonians for bosons (or 'qudits'), and of sign-problem-free determinantal Monte Carlo algorithms for fermions. The intrinsically sign-problematic class of phases we identify is defined in terms of topological invariants with clear observable signatures: the chiral central charge, and the topological spins of anyons. We obtain analogous results for phases that are spontaneously chiral, and present evidence for an extension of our results that applies to both chiral and non-chiral topological matter.

cond-mat.str-el↗

arXiv2019

NV center based nano-NMR enhanced by deep learning

The growing field of nano nuclear magnetic resonance (nano-NMR) seeks to estimate spectra or discriminate between spectra of minuscule amounts of complex molecules. While this field holds great promise, nano-NMR experiments suffer from detrimental inherent noise. This strong noise masks to the weak signal and results in a very low signal-to-noise ratio. Moreover, the noise model is usually complex and unknown, which renders the data processing of the measurement results very complicated. Hence, spectra discrimination is hard to achieve and in particular, it is difficult to reach the optimal discrimination. In this work we present strong indications that this difficulty can be overcome by deep learning (DL) algorithms. The DL algorithms can mitigate the adversarial effects of the noise efficiently by effectively learning the noise model. We show that in the case of frequency discrimination DL algorithms reach the optimal discrimination without having any pre-knowledge of the physical model. Moreover, the DL discrimination scheme outperform Bayesian methods when verified on noisy experimental data obtained by a single Nitrogen-Vacancy (NV) center. In the case of frequency resolution we show that this approach outperforms Bayesian methods even when the latter have full pre-knowledge of the noise model and the former has none. These DL algorithms also emerge as much more efficient in terms of computational resources and run times. Since in many real-world scenarios the noise is complex and difficult to model, we argue that DL is likely to become a dominant tool in the field.

quant-ph↗

arXiv2019

Optimal Renormalization Group Transformation from Information Theory

Recently a novel real-space RG algorithm was introduced, identifying the relevant degrees of freedom of a system by maximizing an information-theoretic quantity, the real-space mutual information (RSMI), with machine learning methods. Motivated by this, we investigate the information theoretic properties of coarse-graining procedures, for both translationally invariant and disordered systems. We prove that a perfect RSMI coarse-graining does not increase the range of interactions in the renormalized Hamiltonian, and, for disordered systems, suppresses generation of correlations in the renormalized disorder distribution, being in this sense optimal. We empirically verify decay of those measures of complexity, as a function of information retained by the RG, on the examples of arbitrary coarse-grainings of the clean and random Ising chain. The results establish a direct and quantifiable connection between properties of RG viewed as a compression scheme, and those of physical objects i.e. Hamiltonians and disorder distributions. We also study the effect of constraints on the number and type of coarse-grained degrees of freedom on a generic RG procedure.

cond-mat.stat-mech↗

arXiv2019

The role of a layer in deep neural networks: a Gaussian Process perspective

A fundamental question in deep learning concerns the role played by individual layers in a deep neural network (DNN) and the transferable properties of the data representations which they learn. To the extent that layers have clear roles, one should be able to optimize them separately using layer-wise loss functions. Such loss functions would describe what is the set of good data representations at each depth of the network and provide a target for layer-wise greedy optimization (LEGO). Here we derive a novel correspondence between Gaussian Processes and SGD trained deep neural networks. Leveraging this correspondence, we derive the Deep Gaussian Layer-wise loss functions (DGLs) which, we believe, are the first supervised layer-wise loss functions which are both explicit and competitive in terms of accuracy. Being highly structured and symmetric, the DGLs provide a promising analytic route to understanding the internal representations generated by DNNs.

cs.LG↗

arXiv2018

Mutual Information, Neural Networks and the Renormalization Group

Physical systems differring in their microscopic details often display strikingly similar behaviour when probed at macroscopic scales. Those universal properties, largely determining their physical characteristics, are revealed by the powerful renormalization group (RG) procedure, which systematically retains "slow" degrees of freedom and integrates out the rest. However, the important degrees of freedom may be difficult to identify. Here we demonstrate a machine learning algorithm capable of identifying the relevant degrees of freedom and executing RG steps iteratively without any prior knowledge about the system. We introduce an artificial neural network based on a model-independent, information-theoretic characterization of a real-space RG procedure, performing this task. We apply the algorithm to classical statistical physics problems in one and two dimensions. We demonstrate RG flow and extract the Ising critical exponent. Our results demonstrate that machine learning techniques can extract abstract physical concepts and consequently become an integral part of theory- and model-building.

cond-mat.dis-nn↗

arXiv2018

Critical Percolation as a Framework to Analyze the Training of Deep Networks

In this paper we approach two relevant deep learning topics: i) tackling of graph structured input data and ii) a better understanding and analysis of deep networks and related learning algorithms. With this in mind we focus on the topological classification of reachability in a particular subset of planar graphs (Mazes). Doing so, we are able to model the topology of data while staying in Euclidean space, thus allowing its processing with standard CNN architectures. We suggest a suitable architecture for this problem and show that it can express a perfect solution to the classification task. The shape of the cost function around this solution is also derived and, remarkably, does not depend on the size of the maze in the large maze limit. Responsible for this behavior are rare events in the dataset which strongly regulate the shape of the cost function near this global minimum. We further identify an obstacle to learning in the form of poorly performing local minima in which the network chooses to ignore some of the inputs. We further support our claims with training experiments and numerical analysis of the cost function on networks with up to $128$ layers.

cond-mat.dis-nn↗

arXiv2017

Classical topological paramagnetism

Topological phases of matter are one of the hallmarks of quantum condensed matter physics. One of their striking features is a bulk-boundary correspondence wherein the topological nature of the bulk manifests itself on boundaries via exotic massless phases. In classical wave phenomena analogous effects may arise; however, these cannot be viewed as equilibrium phases of matter. Here we identify a set of rules under which robust equilibrium classical topological phenomena exist. We write down simple and analytically tractable classical lattice models of spins and rotors in two and three dimensions which, at suitable parameter ranges, are paramagnetic in the bulk but nonetheless exhibit some unusual long-range or critical order on their boundaries. We point out the role of simplicial cohomology as a means of classifying, writing-down, and analyzing such models. This opens a new experimental route for studying strongly interacting topological phases of spins.

cond-mat.stat-mech↗

arXiv2016

Wavefunctions of Symmetry Protected Topological Phases from Conformal Field Theories

We propose a method for analyzing two-dimensional symmetry protected topological (SPT) wavefunctions using a correspondence with conformal field theories (CFTs) and integrable lattice models. This method generalizes the CFT approach for the fractional quantum Hall effect wherein the wavefunction amplitude is written as a many-operator correlator in the CFT. Adopting a bottom-up approach, we start from various known microscopic wavefunctions of SPTs with discrete symmetries and show how the CFT description emerges at large scale, thereby revealing a deep connection between group cocyles and critical, sometimes integrable, models. We show that the CFT describing the bulk wavefunction is often also the one describing the entanglement spectrum, but not always. Using a plasma analogy, we also prove the existence of hidden quasi-long-range order for a large class of SPTs. Finally, we show how response to symmetry fluxes is easily described in terms of the CFT.

cond-mat.str-el↗

arXiv2015

Sharp correlations in the ARPES spectra of strongly disordered topological boundary modes

Data from angle resolved photo-emission spectroscopy (ARPES) often serves as a smoking-gun evidence for the existence of topological materials. It provides the energy dispersion curves of the topological boundary modes which characterize these phases. Unfortunately this method requires a sufficiently regular boundary such that these boundary modes remain sharp in momentum space. Here the seemingly random data obtained from performing ARPES on strongly disordered topological insulators and Weyl semimetals is analyzed theoretically and numerically. Expectedly the disorder averaged ARPES spectra appear featureless. Surprisingly however, correlations in these spectra between different energies and momenta reveal delta-sharp features in momentum space. Measuring such correlations using nano-ARPES may verify the topological nature of the suggested weak topological insulator ($Bi_{14} Rh_3 I_9$) which thus far was not studied using ARPES due to the rough nature of its metallic surfaces.

cond-mat.mes-hall↗

arXiv2015

Using weak measurements to extract the $Z_2$ index of a topological insulator

Recently there has been an interest in applying the concept of weak values and weak measurements to condensed matter systems. Here a weak measurement protocol is proposed for obtaining the $Z_2$ index of a topological insulator. The setup consists of a topological insulator with a hole pierced by a time dependent Aharonov-Bohm flux. A certain weak value ($A_{gs}$) associated with the time-integrated magnetization in the hole has a universal response to a small ambient magnetic field ($B$), namely $A_{gs}B = 2 \hbar$. This result is unaffected by disorder, interactions, and, to a large extent, the speed of the flux threading. It hinges mainly on preventing the flux from leaking outside the hole, as well as being able to detect magnetization at a resolution of a few spins. A similar result may be obtained using only charge measurements, in a setup consisting of a double quantum dot weakly coupled to an LC circuit. Here one obtains $\langle ϕ\rangle_{weak} Q_0 =2\hbar$, where $\langle ϕ\rangle_{weak}$ is a weak value associated with the flux on the inductor and $Q_0$ is the average capacitor charging. The universality of these results suggests that they may be used as a testbed for weak values in condensed matter physics.

cond-mat.mes-hall↗