arXiv subjects

Alexander A. Alemi

Publications and source records attributed to Alexander A. Alemi.

31 records · Page 2Linked to original sources

arXiv2019

On the Use of ArXiv as a Dataset

The arXiv has collected 1.5 million pre-print articles over 28 years, hosting literature from scientific fields including Physics, Mathematics, and Computer Science. Each pre-print features text, figures, authors, citations, categories, and other metadata. These rich, multi-modal features, combined with the natural graph structure---created by citation, affiliation, and co-authorship---makes the arXiv an exciting candidate for benchmarking next-generation models. Here we take the first necessary steps toward this goal, by providing a pipeline which standardizes and simplifies access to the arXiv's publicly available data. We use this pipeline to extract and analyze a 6.7 million edge citation graph, with an 11 billion word corpus of full-text research articles. We present some baseline classification results, and motivate application of more exciting generative graph models.

cs.IR↗

arXiv2019

GILBO: One Metric to Measure Them All

We propose a simple, tractable lower bound on the mutual information contained in the joint generative density of any latent variable generative model: the GILBO (Generative Information Lower BOund). It offers a data-independent measure of the complexity of the learned latent variable description, giving the log of the effective description length. It is well-defined for both VAEs and GANs. We compute the GILBO for 800 GANs and VAEs each trained on four datasets (MNIST, FashionMNIST, CIFAR-10 and CelebA) and discuss the results.

stat.ML↗

arXiv2018

$β$-VAEs can retain label information even at high compression

In this paper, we investigate the degree to which the encoding of a $β$-VAE captures label information across multiple architectures on Binary Static MNIST and Omniglot. Even though they are trained in a completely unsupervised manner, we demonstrate that a $β$-VAE can retain a large amount of label information, even when asked to learn a highly compressed representation.

cs.LG↗

arXiv2018

TherML: Thermodynamics of Machine Learning

In this work we offer a framework for reasoning about a wide class of existing objectives in machine learning. We develop a formal correspondence between this work and thermodynamics and discuss its implications.

cs.LG↗

arXiv2018

Uncertainty in the Variational Information Bottleneck

We present a simple case study, demonstrating that Variational Information Bottleneck (VIB) can improve a network's classification calibration as well as its ability to detect out-of-distribution data. Without explicitly being designed to do so, VIB gives two natural metrics for handling and quantifying uncertainty.

cs.LG↗

arXiv2018

Fixing a Broken ELBO

Recent work in unsupervised representation learning has focused on learning deep directed latent-variable models. Fitting these models by maximizing the marginal likelihood or evidence is typically intractable, thus a common approximation is to maximize the evidence lower bound (ELBO) instead. However, maximum likelihood training (whether exact or approximate) does not necessarily result in a good latent representation, as we demonstrate both theoretically and empirically. In particular, we derive variational lower and upper bounds on the mutual information between the input and the latent variable, and use these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Using this framework, we demonstrate that there is a family of models with identical ELBO, but different quantitative and qualitative characteristics. Our framework also suggests a simple new method to ensure that latent variable models with powerful stochastic decoders do not ignore their latent code.

cs.LG↗

arXiv2018

Canonical Sectors and Evolution of Firms in the US Stock Markets

A classification of companies into sectors of the economy is important for macroeconomic analysis and for investments into the sector-specific financial indices and exchange traded funds (ETFs). Major industrial classification systems and financial indices have historically been based on expert opinion and developed manually. Here we show how unsupervised machine learning can provide a more objective and comprehensive broad-level sector decomposition of stocks. An emergent low-dimensional structure in the space of historical stock price returns automatically identifies "canonical sectors" in the market, and assigns every stock a participation weight into these sectors. Furthermore, by analyzing data from different periods, we show how these weights for listed firms have evolved over time.

q-fin.ST↗

arXiv2017

Jeffrey's prior sampling of deep sigmoidal networks

Neural networks have been shown to have a remarkable ability to uncover low dimensional structure in data: the space of possible reconstructed images form a reduced model manifold in image space. We explore this idea directly by analyzing the manifold learned by Deep Belief Networks and Stacked Denoising Autoencoders using Monte Carlo sampling. The model manifold forms an only slightly elongated hyperball with actual reconstructed data appearing predominantly on the boundaries of the manifold. In connection with the results we present, we discuss problems of sampling high-dimensional manifolds as well as recent work [M. Transtrum, G. Hart, and P. Qiu, Submitted (2014)] discussing the relation between high dimensional geometry and model reduction.

cond-mat.dis-nn↗

arXiv2017

Light Microscopy at Maximal Precision

Microscopy is the workhorse of the physical and life sciences, producing crisp images of everything from atoms to cells well beyond the capabilities of the human eye. However, the analysis of these images is frequently little better than automated manual marking. Here, we revolutionize the analysis of microscopy images, extracting all the information theoretically contained in a complex microscope image. Using a generic, methodological approach, we extract the information by fitting experimental images with a detailed optical model of the microscope, a method we call Parameter Extraction from Reconstructing Images (PERI). As a proof of principle, we demonstrate this approach with a confocal image of colloidal spheres, improving measurements of particle positions and radii by 100x over current methods and attaining the maximum possible accuracy. With this unprecedented resolution, we measure nanometer-scale colloidal interactions in dense suspensions solely with light microscopy, a previously impossible feat. Our approach is generic and applicable to imaging methods from brightfield to electron microscopy, where we expect accuracies of 1 nm and 0.1 pm, respectively.

cond-mat.soft↗

arXiv2016

Improved generator objectives for GANs

We present a framework to understand GAN training as alternating density ratio estimation and approximate divergence minimization. This provides an interpretation for the mismatched GAN generator and discriminator objectives often used in practice, and explains the problem of poor sample diversity. We also derive a family of generator objectives that target arbitrary $f$-divergences without minimizing a lower bound, and use them to train generative image models that target either improved sample quality or greater sample diversity.

cs.LG↗

arXiv2015

You Can Run, You Can Hide: The Epidemiology and Statistical Mechanics of Zombies

We use a popular fictional disease, zombies, in order to introduce techniques used in modern epidemiology modelling, and ideas and techniques used in the numerical study of critical phenomena. We consider variants of zombie models, from fully connected continuous time dynamics to a full scale exact stochastic dynamic simulation of a zombie outbreak on the continental United States. Along the way, we offer a closed form analytical expression for the fully connected differential equation, and demonstrate that the single person per site two dimensional square lattice version of zombies lies in the percolation universality class. We end with a quantitative study of the full scale US outbreak, including the average susceptibility of different geographical regions.

q-bio.PE↗

arXiv2015

Clustering via Content-Augmented Stochastic Blockmodels

Much of the data being created on the web contains interactions between users and items. Stochastic blockmodels, and other methods for community detection and clustering of bipartite graphs, can infer latent user communities and latent item clusters from this interaction data. These methods, however, typically ignore the items' contents and the information they provide about item clusters, despite the tendency of items in the same latent cluster to share commonalities in content. We introduce content-augmented stochastic blockmodels (CASB), which use item content together with user-item interaction data to enhance the user communities and item clusters learned. Comparisons to several state-of-the-art benchmark methods, on datasets arising from scientists interacting with scientific articles, show that content-augmented stochastic blockmodels provide highly accurate clusters with respect to metrics representative of the underlying community structure.

stat.ML↗

arXiv2014

Mechanical Properties of Growing Melanocytic Nevi and the Progression to Melanoma

Melanocytic nevi are benign proliferations that sometimes turn into malignant melanoma in a way that is still unclear from the biochemical and genetic point of view. Diagnostic and prognostic tools are then mostly based on dermoscopic examination and morphological analysis of histological tissues. To investigate the role of mechanics and geometry in the morpholgical dynamics of melanocytic nevi, we study a computation model for cell proliferation in a layered non-linear elastic tissue. Numerical simulations suggest that the morphology of the nevus is correlated to the initial location of the proliferating cell starting the growth process and to the mechanical properties of the tissue. Our results also support that melanocytes are subject to compressive stresses that fluctuate widely in the nevus and depend on the growth stage. Numerical simulations of cells in the epidermis releasing matrix metalloproteinases display an accelerated invasion of the dermis by destroying the basal membrane. Moreover, we suggest experimentally that osmotic stress and collagen inhibit growth in primary melanoma cells while the effect is much weaker in metastatic cells. Knowing that morphological features of nevi might also reflect geometry and mechanics rather than malignancy could be relevant for diagnostic purposes

q-bio.TO↗