arXiv subjects

Saeed Saremi

Publications and source records attributed to Saeed Saremi.

30 records · Page 2Linked to original sources

arXiv2020

Unnormalized Variational Bayes

We unify empirical Bayes and variational Bayes for approximating unnormalized densities. This framework, named unnormalized variational Bayes (UVB), is based on formulating a latent variable model for the random variable $Y=X+N(0,σ^2 I_d)$ and using the evidence lower bound (ELBO), computed by a variational autoencoder, as a parametrization of the energy function of $Y$ which is then used to estimate $X$ with the empirical Bayes least-squares estimator. In this intriguing setup, the $\textit{gradient}$ of the ELBO with respect to noisy inputs plays the central role in learning the energy function. Empirically, we demonstrate that UVB has a higher capacity to approximate energy functions than the parametrization with MLPs as done in neural empirical Bayes (DEEN). We especially showcase $σ=1$, where the differences between UVB and DEEN become visible and qualitative in the denoising experiments. For this high level of noise, the distribution of $Y$ is very smoothed and we demonstrate that one can traverse in a single run $-$ without a restart $-$ all MNIST classes in a variety of styles via walk-jump sampling with a fast-mixing Langevin MCMC sampler. We finish by probing the encoder/decoder of the trained models and confirm UVB $\neq$ VAE.

stat.ML↗

arXiv2020

Learning and Inference in Imaginary Noise Models

Inspired by recent developments in learning smoothed densities with empirical Bayes, we study variational autoencoders with a decoder that is tailored for the random variable $Y=X+N(0,σ^2 I_d)$. A notion of smoothed variational inference emerges where the smoothing is implicitly enforced by the noise model of the decoder; "implicit", since during training the encoder only sees clean samples. This is the concept of imaginary noise model, where the noise model dictates the functional form of the variational lower bound $\mathcal{L}(σ)$, but the noisy data are never seen during learning. The model is named $σ$-VAE. We prove that all $σ$-VAEs are equivalent to each other via a simple $β$-VAE expansion: $\mathcal{L}(σ_2) \equiv \mathcal{L}(σ_1,β)$, where $β=σ_2^2/σ_1^2$. We prove a similar result for the Laplace distribution in exponential families. Empirically, we report an intriguing power law $\mathcal{D}_{\rm KL} \sim σ^{-ν}$ for the learned models and we study the inference in the $σ$-VAE for unseen noisy data. The experiments were performed on MNIST, where we show that quite remarkably the model can make reasonable inferences on extremely noisy samples even though it has not seen any during training. The vanilla VAE completely breaks down in this regime. We finish with a hypothesis (the XYZ hypothesis) on the findings here.

stat.ML↗

arXiv2020

Provable Robust Classification via Learned Smoothed Densities

Smoothing classifiers and probability density functions with Gaussian kernels appear unrelated, but in this work, they are unified for the problem of robust classification. The key building block is approximating the $\textit{energy function}$ of the random variable $Y=X+N(0,σ^2 I_d)$ with a neural network which we use to formulate the problem of robust classification in terms of $\widehat{x}(Y)$, the $\textit{Bayes estimator}$ of $X$ given the noisy measurements $Y$. We introduce $\textit{empirical Bayes smoothed classifiers}$ within the framework of $\textit{randomized smoothing}$ and study it theoretically for the two-class linear classifier, where we show one can improve their robustness above $\textit{the margin}$. We test the theory on MNIST and we show that with a learned smoothed energy function and a linear classifier we can achieve provable $\ell_2$ robust accuracies that are competitive with empirical defenses. This setup can be significantly improved by $\textit{learning}$ empirical Bayes smoothed classifiers with adversarial training and on MNIST we show that we can achieve provable robust accuracies higher than the state-of-the-art empirical defenses in a range of radii. We discuss some fundamental challenges of randomized smoothing based on a geometric interpretation due to concentration of Gaussians in high dimensions, and we finish the paper with a proposal for using walk-jump sampling, itself based on learned smoothed densities, for robust classification.

stat.ML↗

arXiv2020

No Representation without Transformation

We extend the framework of variational autoencoders to represent transformations explicitly in the latent space. In the family of hierarchical graphical models that emerges, the latent space is populated by higher order objects that are inferred jointly with the latent representations they act on. To explicitly demonstrate the effect of these higher order objects, we show that the inferred latent transformations reflect interpretable properties in the observation space. Furthermore, the model is structured in such a way that in the absence of transformations, we can run inference and obtain generative capabilities comparable with standard variational autoencoders. Finally, utilizing the trained encoder, we outperform the baselines by a wide margin on a challenging out-of-distribution classification task.

cs.LG↗

arXiv2020

Neural Empirical Bayes

We unify $\textit{kernel density estimation}$ and $\textit{empirical Bayes}$ and address a set of problems in unsupervised learning with a geometric interpretation of those methods, rooted in the $\textit{concentration of measure}$ phenomenon. Kernel density is viewed symbolically as $X\rightharpoonup Y$ where the random variable $X$ is smoothed to $Y= X+N(0,σ^2 I_d)$, and empirical Bayes is the machinery to denoise in a least-squares sense, which we express as $X \leftharpoondown Y$. A learning objective is derived by combining these two, symbolically captured by $X \rightleftharpoons Y$. Crucially, instead of using the original nonparametric estimators, we parametrize $\textit{the energy function}$ with a neural network denoted by $ϕ$; at optimality, $\nabla ϕ\approx -\nabla \log f$ where $f$ is the density of $Y$. The optimization problem is abstracted as interactions of high-dimensional spheres which emerge due to the concentration of isotropic gaussians. We introduce two algorithmic frameworks based on this machinery: (i) a "walk-jump" sampling scheme that combines Langevin MCMC (walks) and empirical Bayes (jumps), and (ii) a probabilistic framework for $\textit{associative memory}$, called NEBULA, defined à la Hopfield by the $\textit{gradient flow}$ of the learned energy to a set of attractors. We finish the paper by reporting the emergence of very rich "creative memories" as attractors of NEBULA for highly-overlapping spheres.

stat.ML↗

arXiv2019

On approximating $\nabla f$ with neural networks

Consider a feedforward neural network $ψ: \mathbb{R}^d\rightarrow \mathbb{R}^d$ such that $ψ\approx \nabla f$, where $f:\mathbb{R}^d \rightarrow \mathbb{R}$ is a smooth function, therefore $ψ$ must satisfy $\partial_j ψ_i = \partial_i ψ_j$ pointwise. We prove a theorem that a $ψ$ network with more than one hidden layer can only represent one feature in its first hidden layer; this is a dramatic departure from the well-known results for one hidden layer. The proof of the theorem is straightforward, where two backward paths and a weight-tying matrix play the key roles. We then present the alternative, the implicit parametrization, where the neural network is $ϕ: \mathbb{R}^d \rightarrow \mathbb{R}$ and $\nabla ϕ\approx \nabla f$; in addition, a "soft analysis" of $\nabla ϕ$ gives a dual perspective on the theorem. Throughout, we come back to recent probabilistic models that are formulated as $\nabla ϕ\approx \nabla f$, and conclude with a critique of denoising autoencoders.

stat.ML↗

arXiv2018

Deep Energy Estimator Networks

Density estimation is a fundamental problem in statistical learning. This problem is especially challenging for complex high-dimensional data due to the curse of dimensionality. A promising solution to this problem is given here in an inference-free hierarchical framework that is built on score matching. We revisit the Bayesian interpretation of the score function and the Parzen score matching, and construct a multilayer perceptron with a scalable objective for learning the energy (i.e. the unnormalized log-density), which is then optimized with stochastic gradient descent. In addition, the resulting deep energy estimator network (DEEN) is designed as products of experts. We present the utility of DEEN in learning the energy, the score function, and in single-step denoising experiments for synthetic and high-dimensional data. We also diagnose stability problems in the direct estimation of the score function that had been observed for denoising autoencoders.

stat.ML↗

arXiv2017

Annealed Generative Adversarial Networks

We introduce a novel framework for adversarial training where the target distribution is annealed between the uniform distribution and the data distribution. We posited a conjecture that learning under continuous annealing in the nonparametric regime is stable irrespective of the divergence measures in the objective function and proposed an algorithm, dubbed ß-GAN, in corollary. In this framework, the fact that the initial support of the generative network is the whole ambient space combined with annealing are key to balancing the minimax game. In our experiments on synthetic data, MNIST, and CelebA, ß-GAN with a fixed annealing schedule was stable and did not suffer from mode collapse.

stat.ML↗

arXiv2015

The Wilson Machine for Image Modeling

Learning the distribution of natural images is one of the hardest and most important problems in machine learning. The problem remains open, because the enormous complexity of the structures in natural images spans all length scales. We break down the complexity of the problem and show that the hierarchy of structures in natural images fuels a new class of learning algorithms based on the theory of critical phenomena and stochastic processes. We approach this problem from the perspective of the theory of critical phenomena, which was developed in condensed matter physics to address problems with infinite length-scale fluctuations, and build a framework to integrate the criticality of natural images into a learning algorithm. The problem is broken down by mapping images into a hierarchy of binary images, called bitplanes. In this representation, the top bitplane is critical, having fluctuations in structures over a vast range of scales. The bitplanes below go through a gradual stochastic heating process to disorder. We turn this representation into a directed probabilistic graphical model, transforming the learning problem into the unsupervised learning of the distribution of the critical bitplane and the supervised learning of the conditional distributions for the remaining bitplanes. We learnt the conditional distributions by logistic regression in a convolutional architecture. Conditioned on the critical binary image, this simple architecture can generate large, natural-looking images, with many shades of gray, without the use of hidden units, unprecedented in the studies of natural images. The framework presented here is a major step in bringing criticality and stochastic processes to machine learning and in studying natural image statistics.

stat.ML↗

arXiv2009

Kondo Vortices, Zero Modes, and Magnetic Ordering in a Kondo Lattice Model

Motivated by the mysteries of the heavy fermion quantum critical point, we investigate the competition between Kondo screening and magnetic ordering in the honeycomb Kondo lattice at half filling. We examine the destruction of the Kondo phase by proliferating vortex configurations in the Kondo hybridization order parameter. We find that there are zero modes associated with Kondo vortices. Condensing these vortices can lead to the antiferromagnetic phase.

cond-mat.str-el↗

arXiv2007

RKKY in half-filled bipartite lattices: graphene as an example

We first present a simple proof that for any bipartite lattice at half filling the RKKY interaction is antiferromagnetic between impurities on opposite (i.e., A and B) sublattices and is ferromagnetic between impurities on the same sublattices. This result is valid on all length scales. We then focus on the honeycomb lattice and examine the theorem in the long distance limit by performing the low energy calculation using Dirac electrons. To find the universal (cutoff free) result we perform the calculation in smooth cutoff schemes, as we show that the calculation based on a sharp cutoff leads to wrong results. We also find the long distance behavior of the RKKY interaction between "plaquette" impurities in both coherent and incoherent regimes.

cond-mat.str-el↗

arXiv2007

Quantum critical point in the Kondo-Heisenberg model on the honeycomb lattice

We study the Kondo--Heisenberg model on the honeycomb lattice at half-filling. Due to the vanishing of the density of states at the fermi level, the Kondo insulator disappears at a finite Kondo coupling even in the absence of the Heisenberg exchange. We adopt a large-N formulation of this model and use the renormalization group machinery to study systematically the second order phase transition of the Kondo insulator (KI) to the algebraic spin liquid (ASL). We note that neither phase breaks any physical symmetry, so that the transition is not described by the standard Ginzburg-Landau-Wilson critical point. We find a stable Lorentz-invariant fixed point that controls this second order phase transition. We calculate the exponent $ν$ of the diverging length scale near the transition. The quasi-particle weight of the conduction electron vanishes at this KI--ASL fixed point, indicating non-Fermi liquid behavior. The algebraic decay exponent of the staggered spin correlation is calculated at the fixed point and in the ASL phase. We find a jump in this exponent at the transition point.

cond-mat.str-el↗