arXiv subjects

Karol Gregor

Publications and source records attributed to Karol Gregor.

29 records · Page 2Linked to original sources

arXiv2015

DRAW: A Recurrent Neural Network For Image Generation

This paper introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism that mimics the foveation of the human eye, with a sequential variational auto-encoding framework that allows for the iterative construction of complex images. The system substantially improves on the state of the art for generative models on MNIST, and, when trained on the Street View House Numbers dataset, it generates images that cannot be distinguished from real data with the naked eye.

cs.CV↗

arXiv2014

Neural Variational Inference and Learning in Belief Networks

Highly expressive directed latent variable models, such as sigmoid belief networks, are difficult to train on large datasets because exact inference in them is intractable and none of the approximate inference methods that have been applied to them scale well. We propose a fast non-iterative approximate inference method that uses a feedforward network to implement efficient exact sampling from the variational posterior. The model and this inference network are trained jointly by maximizing a variational lower bound on the log-likelihood. Although the naive estimator of the inference model gradient is too high-variance to be useful, we make it practical by applying several straightforward model-independent variance reduction techniques. Applying our approach to training sigmoid belief networks and deep autoregressive networks, we show that it outperforms the wake-sleep algorithm on MNIST and achieves state-of-the-art results on the Reuters RCV1 document dataset.

cs.LG↗

arXiv2014

Deep AutoRegressive Networks

We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approximate parameter estimation method based on the minimum description length (MDL) principle, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference. We demonstrate state-of-the-art generative performance on a number of classic data sets: several UCI data sets, MNIST and Atari 2600 games.

cs.LG↗

arXiv2012

Fast approximations to structured sparse coding and applications to object classification

We describe a method for fast approximation of sparse coding. The input space is subdivided by a binary decision tree, and we simultaneously learn a dictionary and assignment of allowed dictionary elements for each leaf of the tree. We store a lookup table with the assignments and the pseudoinverses for each node, allowing for very fast inference. We give an algorithm for learning the tree, the dictionary and the dictionary element assignment, and In the process of describing this algorithm, we discuss the more general problem of learning the groups in group structured sparse modelling. We show that our method creates good sparse representations by using it in the object recognition framework of \cite{lazebnik06,yang-cvpr-09}. Implementing our own fast version of the SIFT descriptor the whole system runs at 20 frames per second on $321 \times 481$ sized images on a laptop with a quad-core cpu, while sacrificing very little accuracy on the Caltech 101 and 15 scenes benchmarks.

cs.CV↗

arXiv2011

Learning Representations by Maximizing Compression

We give an algorithm that learns a representation of data through compression. The algorithm 1) predicts bits sequentially from those previously seen and 2) has a structure and a number of computations similar to an autoencoder. The likelihood under the model can be calculated exactly, and arithmetic coding can be used directly for compression. When training on digits the algorithm learns filters similar to those of restricted boltzman machines and denoising autoencoders. Independent samples can be drawn from the model by a single sweep through the pixels. The algorithm has a good compression performance when compared to other methods that work under random ordering of pixels.

cs.CV↗

arXiv2011

Efficient Learning of Sparse Invariant Representations

We propose a simple and efficient algorithm for learning sparse invariant representations from unlabeled data with fast inference. When trained on short movies sequences, the learned features are selective to a range of orientations and spatial frequencies, but robust to a wide range of positions, similar to complex cells in the primary visual cortex. We give a hierarchical version of the algorithm, and give guarantees of fast convergence under certain conditions.

cs.CV↗

arXiv2009

Behavior and performance of the deep belief networks on image classification

We apply deep belief networks of restricted Boltzmann machines to bags of words of sift features obtained from databases of 13 Scenes, 15 Scenes and Caltech 256 and study experimentally their behavior and performance. We find that the final performance in the supervised phase is reached much faster if the system is pre-trained. Pre-training the system on a larger dataset keeping the supervised dataset fixed improves the performance (for the 13 Scenes case). After the unsupervised pre-training, neurons arise that form approximate explicit representations for several categories (meaning they are mostly active for this category). The last three facts suggest that unsupervised training really discovers structure in these data. Pre-training can be done on a completely different dataset (we use Corel dataset) and we find that the supervised phase performs just as good (on the 15 Scenes dataset). This leads us to conjecture that one can pre-train the system once (e.g. in a factory) and subsequently apply it to many supervised problems which then learn much faster. The best performance is obtained with single hidden layer system suggesting that the histogram of sift features doesn't have much high level structure. The overall performance is almost equal, but slightly worse then that of the support vector machine and the spatial pyramidal matching.

cs.NE↗

arXiv2008

Impurities in triangular lattice spin 1/2 antiferromagnet

We study effects of nonmagnetic impurities in a spin-1/2 frustrated triangular antiferromagnet with the aim of understanding the observed broadening of $^{13}$C NMR lines in the organic spin liquid material $κ$-(ET)$_2$Cu$_2$(CN)$_3$. For high temperatures down to $J/3$, we calculate local susceptibility near a nonmagnetic impurity and near a grain boundary for the nearest neighbor Heisenberg model in high temperature series expansion. We find that the local susceptibility decays to the uniform one in few lattice spacings, and for a low density of impurities we would not be able to explain the line broadening present in the experiments already at elevated temperatures. At low temperatures, we assume a gapless spin liquid with a Fermi surface of spinons. We calculate the local susceptibility in the mean field and also go beyond the mean field by Gutzwiller projection. The zero temperature local susceptibility decays as a power law and oscillates at $2 k_F$. As in the high temperature analysis we find that a low density of impurities is not able to explain the observed broadening of the lines. We are thus led to conclude that there is more disorder in the system. We find that a large density of point-like disorder gives broadening that is consistent with the experiment down to about 5K, but that below this temperature additional mechanism is likely needed.

cond-mat.str-el↗

arXiv2008

Studies of non-magnetic impurities in the spin-1/2 Kagome Antiferromagnet

Motivated by recent experiments on ZnCu$_3$(OH)$_6$Cl$_2$, we study the inhomogeneous Knight shifts (local susceptibilities) of spin 1/2 Kagome antiferromagnet in the presence of nonmagnetic impurities. Using high temperature series expansion, we calculate the local susceptibility and its histogram down to about T=0.4J. At low temperatures, we explore a Dirac spin liquid proposal and calculate the local susceptibility in the mean field and beyond mean field using Gutzwiller projection, finding the overall picture to be consistent with the NMR experiments.

cond-mat.str-el↗

arXiv2007

Spin solid phases of spin 1 and spin 3/2 antiferromagnets on a cubic lattice

We study spin S=1 and S=3/2 Heisenberg antiferromagnets on a cubic lattice focusing on spin solid states. Using Schwinger boson formulation for spins, we start in a U(1) spin liquid phase proximate to Neel phase and explore possible confining paramagnetic phases as we transition away from the spin liquid by the process of monopole condensation. Electromagnetic duality is used to rewrite the theory in terms of monopoles. For spin 1 we find several candidate phases of which the most natural one is a phase with spins organized into parallel Haldane chains. For spin 3/2 we find that the most natural phase has spins organized into parallel ladders. As a by-product, we also write a Landau theory of the ordering in two special classical frustrated XY models on the cubic lattice, one of which is the fully frustrated XY model. In a particular limit our approach maps to a dimer model with 2S dimers coming out of every site, and we find the same spin solid phases in this regime as well.

cond-mat.str-el↗

arXiv2005

Spin-nematic order in the frustrated pyrochlore-lattice quantum rotor model

As an example of ordering due to quantum fluctuations, we examine the nearest-neighbor antiferromagnetic quantum O(n) rotor model on the pyrochlore lattice. Classically, this system remains disordered even at zero temperature; we find that adding quantum fluctuations induces an ordered phase that survives to positive temperature, and we determine how its phase diagram scales with the coupling constant and the number of spin components. We demonstrate, using quantum Monte Carlo simulations, that this phase has long-range spin-nematic order, and that the phase transition into it appears to be first order.

cond-mat.str-el↗