Searcharxiv⌕ Search

arXiv subjects

Stephan Mandt

Publications and source records attributed to Stephan Mandt.

121 records · Page 7Linked to original sources
arXiv2017

Sparse Probit Linear Mixed Model

Linear Mixed Models (LMMs) are important tools in statistical genetics. When used for feature selection, they allow to find a sparse set of genetic traits that best predict a continuous phenotype of interest, while simultaneously correcting for various confounding factors such as age, ethnicity and population structure. Formulated as models for linear regression, LMMs have been restricted to continuous phenotypes. We introduce the Sparse Probit Linear Mixed Model (Probit-LMM), where we generalize the LMM modeling paradigm to binary phenotypes. As a technical challenge, the model no longer possesses a closed-form likelihood function. In this paper, we present a scalable approximate inference algorithm that lets us fit the model to high-dimensional data sets. We show on three real-world examples from different domains that in the setup of binary labels, our algorithm leads to better prediction accuracies and also selects features which show less correlation with the confounding factors.

stat.ML↗
arXiv2017

Structured Black Box Variational Inference for Latent Time Series Models

Continuous latent time series models are prevalent in Bayesian modeling; examples include the Kalman filter, dynamic collaborative filtering, or dynamic topic models. These models often benefit from structured, non mean field variational approximations that capture correlations between time steps. Black box variational inference with reparameterization gradients (BBVI) allows us to explore a rich new class of Bayesian non-conjugate latent time series models; however, a naive application of BBVI to such structured variational models would scale quadratically in the number of time steps. We describe a BBVI algorithm analogous to the forward-backward algorithm which instead scales linearly in time. It allows us to efficiently sample from the variational distribution and estimate the gradients of the ELBO. Finally, we show results on the recently proposed dynamic word embedding model, which was trained using our method.

stat.ML↗
arXiv2016

Exponential Family Embeddings

Word embeddings are a powerful approach for capturing semantic similarity among terms in a vocabulary. In this paper, we develop exponential family embeddings, a class of methods that extends the idea of word embeddings to other types of high-dimensional data. As examples, we studied neural data with real-valued observations, count data from a market basket analysis, and ratings data from a movie recommendation system. The main idea is to model each observation conditioned on a set of other observations. This set is called the context, and the way the context is defined is a modeling choice that depends on the problem. In language the context is the surrounding words; in neuroscience the context is close-by neurons; in market basket data the context is other items in the shopping cart. Each type of embedding model defines the context, the exponential family of conditional distributions, and how the latent embedding vectors are shared across data. We infer the embeddings with a scalable algorithm based on stochastic gradient descent. On all three applications - neural activity of zebrafish, users' shopping behavior, and movie ratings - we found exponential family embedding models to be more effective than other types of dimension reduction. They better reconstruct held-out data and find interesting qualitative structure.

stat.ML↗
arXiv2016

Variational Tempering

Variational inference (VI) combined with data subsampling enables approximate posterior inference over large data sets, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This approach uses a decreasing temperature parameter which deterministically deforms the objective during the course of the optimization. A well-known drawback to this annealing approach is the choice of the cooling schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data.

stat.ML↗
arXiv2016

A Variational Analysis of Stochastic Gradient Algorithms

Stochastic Gradient Descent (SGD) is an important algorithm in machine learning. With constant learning rates, it is a stochastic process that, after an initial phase of convergence, generates samples from a stationary distribution. We show that SGD with constant rates can be effectively used as an approximate posterior inference algorithm for probabilistic modeling. Specifically, we show how to adjust the tuning parameters of SGD such as to match the resulting stationary distribution to the posterior. This analysis rests on interpreting SGD as a continuous-time stochastic process and then minimizing the Kullback-Leibler divergence between its stationary distribution and the target posterior. (This is in the spirit of variational inference.) In more detail, we model SGD as a multivariate Ornstein-Uhlenbeck process and then use properties of this process to derive the optimal parameters. This theoretical framework also connects SGD to modern scalable inference algorithms; we analyze the recently proposed stochastic gradient Fisher scoring under this perspective. We demonstrate that SGD with properly chosen constant rates gives a new way to optimize hyperparameters in probabilistic models.

stat.ML↗
arXiv2014

Damping of Bloch oscillations: Variational solutions of the Boltzmann equation beyond linear response

Variational solutions of the Boltzmann equation usually rely on the concept of linear response. We extend the variational approach for tight-binding models at high entropies to a regime far beyond linear response. We analyze both weakly interacting fermions and incoherent bosons on a lattice. We consider a case where the particles are driven by a constant force, leading to the well-known Bloch oscillations, and we consider interactions that are weak enough not to overdamp these oscillations. This regime is computationally demanding and relevant for ultracold atoms in optical lattices. We derive a simple theory in terms of coupled dynamic equations for the particle density, energy density, current and heat current, allowing for analytic solutions. As an application, we identify damping coefficients for Bloch oscillations in the Hubbard model at weak interactions and compute them for a one-dimensional toy model. We also approximately solve the long-time dynamics of a weakly interacting, strongly Bloch-oscillating cloud of fermionic particles in a tilted lattice, leading to a subdiffusive scaling exponent.

cond-mat.quant-gas↗
arXiv2014

Smoothed Gradients for Stochastic Variational Inference

Stochastic variational inference (SVI) lets us scale up Bayesian computation to massive data. It uses stochastic optimization to fit a variational distribution, following easy-to-compute noisy natural gradients. As with most traditional stochastic optimization methods, SVI takes precautions to use unbiased stochastic gradients whose expectations are equal to the true gradients. In this paper, we explore the idea of following biased stochastic gradients in SVI. Our method replaces the natural gradient with a similarly constructed vector that uses a fixed-window moving average of some of its previous terms. We will demonstrate the many advantages of this technique. First, its computational cost is the same as for SVI and storage requirements only multiply by a constant factor. Second, it enjoys significant variance reduction over the unbiased estimates, smaller bias than averaged gradients, and leads to smaller mean-squared error against the full gradient. We test our method on latent Dirichlet allocation with three large corpora.

stat.ML↗
arXiv2014

Stochastic Differential Equations for Quantum Dynamics of Spin-Boson Networks

The quantum dynamics of open many-body systems poses a challenge for computational approaches. Here we develop a stochastic scheme based on the positive P phase-space representation to study the nonequilibrium dynamics of coupled spin-boson networks that are driven and dissipative. Such problems are at the forefront of experimental research in cavity and solid state realizations of quantum optics, as well as cold atom physics, trapped ions and superconducting circuits. We demonstrate and test our method on a driven, dissipative two-site system, each site involving a spin coupled to a photonic mode, with photons hopping between the sites, where we find good agreement with Monte Carlo Wavefunction simulations. In addition to numerically reproducing features recently observed in an experiment [Phys. Rev. X 4, 031043 (2014)], we also predict a novel steady state quantum dynamical phase transition for an asymmetric configuration of drive and dissipation.

quant-ph↗
arXiv2013

Relaxation towards negative temperatures in bosonic systems: Generalized Gibbs ensembles and beyond integrability

Motivated by the recent experimental observation of negative absolute temperature states in systems of ultracold atomic gases in optical lattices [Braun et al., Science 339, 52 (2013)], we investigate theoretically the formation of these states. More specifically, we consider the relaxation after a sudden inversion of the external parabolic confining potential in the one-dimensional inhomogeneous Bose-Hubbard model. First, we focus on the integrable hard-core boson limit which allows us to treat large systems and arbitrarily long times, providing convincing numerical evidence for relaxation to a generalized Gibbs ensemble at negative temperature T<0, a notion we define in this context. Second, going beyond one dimension, we demonstrate that the emergence of negative temperature states can be understood in a dual way in terms of positive temperatures, which relies on a dynamic symmetry of the Hubbard model. We complement the study by exact diagonalization simulations at finite values of the on-site interaction.

cond-mat.quant-gas↗
arXiv2012

Fermionic transport in a homogeneous Hubbard model: Out-of-equilibrium dynamics with ultracold atoms

Transport properties are among the defining characteristics of many important phases in condensed matter physics. In the presence of strong correlations they are difficult to predict even for model systems like the Hubbard model. In real materials they are in general obscured by additional complications including impurities, lattice defects or multi-band effects. Ultracold atoms in contrast offer the possibility to study transport and out-of-equilibrium phenomena in a clean and well-controlled environment and can therefore act as a quantum simulator for condensed matter systems. Here we studied the expansion of an initially confined fermionic quantum gas in the lowest band of a homogeneous optical lattice. While we observe ballistic transport for non-interacting atoms, even small interactions render the expansion almost bimodal with a dramatically reduced expansion velocity. The dynamics is independent of the sign of the interaction, revealing a novel, dynamic symmetry of the Hubbard model.

cond-mat.quant-gas↗
arXiv2011

Interacting Fermionic Atoms in Optical Lattices Diffuse Symmetrically Upwards and Downwards in a Gravitational Potential

We consider a cloud of fermionic atoms in an optical lattice described by a Hubbard model with an additional linear potential. While homogeneous interacting systems mainly show damped Bloch oscillations and heating, a finite cloud behaves differently: It expands symmetrically such that gains of potential energy at the top are compensated by losses at the bottom. Interactions stabilize the necessary heat currents by inducing gradients of the inverse temperature 1/T, with T<0 at the bottom of the cloud. An analytic solution of hydrodynamic equations shows that the width of the cloud increases with t^(1/3) for long times consistent with results from our Boltzmann simulations.

cond-mat.quant-gas↗
arXiv2010

Equilibration rates and negative absolute temperatures for ultracold atoms in optical lattices

As highly tunable interacting systems, cold atoms in optical lattices are ideal to realize and observe negative absolute temperatures, T < 0. We show theoretically that by reversing the confining potential, stable superfluid condensates at finite momentum and T < 0 can be created with low entropy production for attractive bosons. They may serve as `smoking gun' signatures of equilibrated T < 0. For fermions, we analyze the time scales needed to equilibrate to T < 0. For moderate interactions, the equilibration time is proportional to the square of the radius of the cloud and grows with increasing interaction strengths as atoms and energy are transported by diffusive processes.

cond-mat.quant-gas↗