arXiv subjects

George Stein

Publications and source records attributed to George Stein.

At least 19 recordsLinked to original sources

arXiv2024

A Geometric Framework for Understanding Memorization in Generative Models

As deep generative models have progressed, recent work has shown them to be capable of memorizing and reproducing training datapoints when deployed. These findings call into question the usability of generative models, especially in light of the legal and privacy risks brought about by memorization. To better understand this phenomenon, we propose the manifold memorization hypothesis (MMH), a geometric framework which leverages the manifold hypothesis into a clear language in which to reason about memorization. We propose to analyze memorization in terms of the relationship between the dimensionalities of (i) the ground truth data manifold and (ii) the manifold learned by the model. This framework provides a formal standard for "how memorized" a datapoint is and systematically categorizes memorized data into two types: memorization driven by overfitting and memorization driven by the underlying data distribution. By analyzing prior work in the context of the MMH, we explain and unify assorted observations in the literature. We empirically validate the MMH using synthetic data and image datasets up to the scale of Stable Diffusion, developing new tools for detecting and preventing generation of memorized samples in the process.

stat.ML↗

arXiv2024

Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks

Large-scale vision models have become integral in many applications due to their unprecedented performance and versatility across downstream tasks. However, the robustness of these foundation models has primarily been explored for a single task, namely image classification. The vulnerability of other common vision tasks, such as semantic segmentation and depth estimation, remains largely unknown. We present a comprehensive empirical evaluation of the adversarial robustness of self-supervised vision encoders across multiple downstream tasks. Our attacks operate in the encoder embedding space and at the downstream task output level. In both cases, current state-of-the-art adversarial fine-tuning techniques tested only for classification significantly degrade clean and robust performance on other tasks. Since the purpose of a foundation model is to cater to multiple applications at once, our findings reveal the need to enhance encoder robustness more broadly. Our code is available at ${github.com/layer6ai-labs/ssl-robustness}$.

cs.CV↗

arXiv2024

TabPFGen -- Tabular Data Generation with TabPFN

Advances in deep generative modelling have not translated well to tabular data. We argue that this is caused by a mismatch in structure between popular generative models and discriminative models of tabular data. We thus devise a technique to turn TabPFN -- a highly performant transformer initially designed for in-context discriminative tabular tasks -- into an energy-based generative model, which we dub TabPFGen. This novel framework leverages the pre-trained TabPFN as part of the energy function and does not require any additional training or hyperparameter tuning, thus inheriting TabPFN's in-context learning capability. We can sample from TabPFGen analogously to other energy-based models. We demonstrate strong results on standard generative modelling tasks, including data augmentation, class-balancing, and imputation, unlocking a new frontier of tabular data generation.

cs.LG↗

arXiv2024

Self-supervised Representation Learning From Random Data Projectors

Self-supervised representation learning~(SSRL) has advanced considerably by exploiting the transformation invariance assumption under artificially designed data augmentations. While augmentation-based SSRL algorithms push the boundaries of performance in computer vision and natural language processing, they are often not directly applicable to other data modalities, and can conflict with application-specific data augmentation constraints. This paper presents an SSRL approach that can be applied to any data modality and network architecture because it does not rely on augmentations or masking. Specifically, we show that high-quality data representations can be learned by reconstructing random data projections. We evaluate the proposed approach on a wide range of representation learning tasks that span diverse modalities and real-world applications. We show that it outperforms multiple state-of-the-art SSRL baselines. Due to its wide applicability and strong empirical results, we argue that learning from randomness is a fruitful research direction worthy of attention and further study.

cs.LG↗

arXiv2024

Exploring the Non-Gaussianity of the Cosmic Infrared Background and Its Weak Gravitational Lensing

Gravitational lensing deflects the paths of photons, altering the statistics of cosmic backgrounds and distorting their information content. We take the Cosmic Infrared Background (CIB), which provides plentiful information about galaxy formation and evolution, as an example to probe the effect of lensing on non-Gaussian statistics. Using the Websky simulations, we first quantify the non-Gaussianity of the CIB, revealing additional detail on top of its well-measured power spectrum. To achieve this, we use needlet-like multipole-band-filters to calculate the variance and higher-point correlations. Using our simulations, we show the 2-point, 3-point and 4-point spectra, and compare our calculated power spectra and bispectra to Planck values. We then lens the CIB, shell-by-shell with corresponding convergence maps, to capture the broad redshift extent of both the CIB and its lensing convergence. The lensing of the CIB changes the 3-point and 4-point functions by a few tens of percent at large scales, unlike with the power spectrum, which changes by less than two percent. We expand our analyses to encompass the full intensity probability distribution functions (PDFs) involving all n-point correlations as a function of scale. In particular, we use the relative entropy between lensed and unlensed PDFs to create a spectrum of templates that can allow estimation of lensing. The underlying CIB model is missing the important role of star-bursting, which we test by adding a stochastic log-normal term to the intensity distributions. The novel aspects of our filtering and lensing pipeline should prove useful for any radiant background, including line intensity maps.

astro-ph.CO↗

arXiv2023

Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models

We systematically study a wide variety of generative models spanning semantically-diverse image datasets to understand and improve the feature extractors and metrics used to evaluate them. Using best practices in psychophysics, we measure human perception of image realism for generated samples by conducting the largest experiment evaluating generative models to date, and find that no existing metric strongly correlates with human evaluations. Comparing to 17 modern metrics for evaluating the overall performance, fidelity, diversity, rarity, and memorization of generative models, we find that the state-of-the-art perceptual realism of diffusion models as judged by humans is not reflected in commonly reported metrics such as FID. This discrepancy is not explained by diversity in generated samples, though one cause is over-reliance on Inception-V3. We address these flaws through a study of alternative self-supervised feature extractors, find that the semantic information encoded by individual networks strongly depends on their training procedure, and show that DINOv2-ViT-L/14 allows for much richer evaluation of generative models. Next, we investigate data memorization, and find that generative models do memorize training examples on simple, smaller datasets like CIFAR10, but not necessarily on more complex datasets like ImageNet. However, our experiments show that current metrics do not properly detect memorization: none in the literature is able to separate memorization from other phenomena such as underfitting or mode shrinkage. To facilitate further development of generative models and their evaluation we release all generated image datasets, human evaluation data, and a modular library to compute 17 common metrics for 9 different encoders at https://github.com/layer6ai-labs/dgm-eval.

cs.LG↗

arXiv2022

A Probabilistic Autoencoder for Type Ia Supernovae Spectral Time Series

We construct a physically-parameterized probabilistic autoencoder (PAE) to learn the intrinsic diversity of type Ia supernovae (SNe Ia) from a sparse set of spectral time series. The PAE is a two-stage generative model, composed of an Auto-Encoder (AE) which is interpreted probabilistically after training using a Normalizing Flow (NF). We demonstrate that the PAE learns a low-dimensional latent space that captures the nonlinear range of features that exists within the population, and can accurately model the spectral evolution of SNe Ia across the full range of wavelength and observation times directly from the data. By introducing a correlation penalty term and multi-stage training setup alongside our physically-parameterized network we show that intrinsic and extrinsic modes of variability can be separated during training, removing the need for the additional models to perform magnitude standardization. We then use our PAE in a number of downstream tasks on SNe Ia for increasingly precise cosmological analyses, including automatic detection of SN outliers, the generation of samples consistent with the data distribution, and solving the inverse problem in the presence of noisy and incomplete data to constrain cosmological distance measurements. We find that the optimal number of intrinsic model parameters appears to be three, in line with previous studies, and show that we can standardize our test sample of SNe Ia with an RMS of $0.091 \pm 0.010$ mag, which corresponds to $0.074 \pm 0.010$ mag if peculiar velocity contributions are removed. Trained models and codes are released at \href{https://github.com/georgestein/suPAErnova}{github.com/georgestein/suPAErnova}

astro-ph.CO↗

arXiv2022

Mining for Strong Gravitational Lenses with Self-supervised Learning

We employ self-supervised representation learning to distill information from 76 million galaxy images from the Dark Energy Spectroscopic Instrument Legacy Imaging Surveys' Data Release 9. Targeting the identification of new strong gravitational lens candidates, we first create a rapid similarity search tool to discover new strong lenses given only a single labelled example. We then show how training a simple linear classifier on the self-supervised representations, requiring only a few minutes on a CPU, can automatically classify strong lenses with great efficiency. We present 1192 new strong lens candidates that we identified through a brief visual identification campaign, and release an interactive web-based similarity search tool and the top network predictions to facilitate crowd-sourcing rapid discovery of additional strong gravitational lenses and other rare objects: https://github.com/georgestein/ssl-legacysurvey.

astro-ph.IM↗

arXiv2021

Self-supervised similarity search for large scientific datasets

We present the use of self-supervised learning to explore and exploit large unlabeled datasets. Focusing on 42 million galaxy images from the latest data release of the Dark Energy Spectroscopic Instrument (DESI) Legacy Imaging Surveys, we first train a self-supervised model to distill low-dimensional representations that are robust to symmetries, uncertainties, and noise in each image. We then use the representations to construct and publicly release an interactive semantic similarity search tool. We demonstrate how our tool can be used to rapidly discover rare objects given only a single example, increase the speed of crowd-sourcing campaigns, and construct and improve training sets for supervised applications. While we focus on images from sky surveys, the technique is straightforward to apply to any scientific dataset of any dimensionality. The similarity search web app can be found at https://github.com/georgestein/galaxy_search

astro-ph.IM↗

arXiv2021

Self-Supervised Representation Learning for Astronomical Images

Sky surveys are the largest data generators in astronomy, making automated tools for extracting meaningful scientific information an absolute necessity. We show that, without the need for labels, self-supervised learning recovers representations of sky survey images that are semantically useful for a variety of scientific tasks. These representations can be directly used as features, or fine-tuned, to outperform supervised methods trained only on labeled data. We apply a contrastive learning framework on multi-band galaxy photometry from the Sloan Digital Sky Survey (SDSS) to learn image representations. We then use them for galaxy morphology classification, and fine-tune them for photometric redshift estimation, using labels from the Galaxy Zoo 2 dataset and SDSS spectroscopy. In both downstream tasks, using the same learned representations, we outperform the supervised state-of-the-art results, and we show that our approach can achieve the accuracy of supervised models while using 2-4 times fewer labels for training.

astro-ph.IM↗

arXiv2021

Statistical exploration of halo anisotropic clustering and intrinsic alignments with the mass-Peak Patch algorithm

The anisotropy or triaxiality of massive dark matter haloes largely defines the structure of the cosmic web, in particular the filaments that join the haloes together. Here we investigate such oriented correlations in mass-Peak Patch halo catalogues by using the initial strain tensor of spherical proto-halo regions to orient the haloes. To go beyond the spherically averaged two-point correlation function of haloes we use oriented stacks to compute oriented two-point correlations: we explicitly break isotropy by imposing a local frame set by the strain tensor of the reference halo before stacking neighbouring haloes. Beyond the exclusion zone of the reference halo, clustering is found to be strongly enhanced along the major direction of the strain tensor as expected. This anisotropic clustering of haloes along filaments is further quantified by using a spherical harmonics decomposition. Furthermore, we compute the evolution of cluster-scale halo principal directions relative to those of their neighbours and show that there are strong correlations extending up to very large scales. In order to provide calculations more suitable to observational confrontations, we also utilize 2D projected versions of some equivalent correlation functions. Finally, we show that the multipole structure of the mass-peak patch halo's anisotropic clustering can be qualitatively captured in an analytic treatment based on peak theory. Though highly informative, analytic evaluation involves extensive use of Monte Carlo methods, which is also what the simulated catalogue uses, taking into account as they do the adaptive nature of the mass-peak patch mass hierarchy and all non-local complexities associated with the exclusion of smaller haloes overlapping with larger ones: there is no substitute for the mass-Peak Patch simulation-based determination of oriented and anisotropic correlations.

astro-ph.CO↗

arXiv2021

The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics

A new paradigm for data-driven, model-agnostic new physics searches at colliders is emerging, and aims to leverage recent breakthroughs in anomaly detection and machine learning. In order to develop and benchmark new anomaly detection methods within this framework, it is essential to have standard datasets. To this end, we have created the LHC Olympics 2020, a community challenge accompanied by a set of simulated collider events. Participants in these Olympics have developed their methods using an R&D dataset and then tested them on black boxes: datasets with an unknown anomaly (or not). This paper will review the LHC Olympics 2020 challenge, including an overview of the competition, a description of methods deployed in the competition, lessons learned from the experience, and implications for data analyses with future datasets as well as future colliders.

hep-ph↗

arXiv2021

Estimating Galactic Distances From Images Using Self-supervised Representation Learning

We use a contrastive self-supervised learning framework to estimate distances to galaxies from their photometric images. We incorporate data augmentations from computer vision as well as an application-specific augmentation accounting for galactic dust. We find that the resulting visual representations of galaxy images are semantically useful and allow for fast similarity searches, and can be successfully fine-tuned for the task of redshift estimation. We show that (1) pretraining on a large corpus of unlabeled data followed by fine-tuning on some labels can attain the accuracy of a fully-supervised model which requires 2-4x more labeled data, and (2) that by fine-tuning our self-supervised representations using all available data labels in the Main Galaxy Sample of the Sloan Digital Sky Survey (SDSS), we outperform the state-of-the-art supervised learning method.

astro-ph.IM↗

arXiv2020

Unsupervised in-distribution anomaly detection of new physics through conditional density estimation

Anomaly detection is a key application of machine learning, but is generally focused on the detection of outlying samples in the low probability density regions of data. Here we instead present and motivate a method for unsupervised in-distribution anomaly detection using a conditional density estimator, designed to find unique, yet completely unknown, sets of samples residing in high probability density regions. We apply this method towards the detection of new physics in simulated Large Hadron Collider (LHC) particle collisions as part of the 2020 LHC Olympics blind challenge, and show how we detected a new particle appearing in only 0.08% of 1 million collision events. The results we present are our original blind submission to the 2020 LHC Olympics, where it achieved the state-of-the-art performance.

cs.LG↗

arXiv2020

Nonlinear 3D Cosmic Web Simulation with Heavy-Tailed Generative Adversarial Networks

Fast and accurate simulations of the non-linear evolution of the cosmic density field are a major component of many cosmological analyses, but the computational time and storage required to run them can be exceedingly large. For this reason, we use generative adversarial networks (GANs) to learn a compressed representation of the 3D matter density field that is fast and easy to sample, and for the first time show that GANs are capable of generating samples at the level of accuracy of other conventional methods. Using sub-volumes from a suite of GADGET-2 N-body simulations, we demonstrate that a deep-convolutional GAN can generate samples that capture both large- and small-scale features of the matter density field, as validated through a variety of n-point statistics. The use of a data scaling that preserves high-density features and a heavy-tailed latent space prior allow us to obtain state of the art results for fast 3D cosmic web generation. In particular, the mean power spectra from generated samples agree to within 5% up to k=3 and within 10% for k<5 when compared with N-body simulations, and similar accuracy is obtained for a variety of bispectra. By modeling the latent space with a heavy-tailed prior rather than a standard Gaussian, we better capture sample variance in the high-density voxel PDF and reduce errors in power spectrum and bispectrum covariance on all scales. Furthermore, we show that a conditional GAN can smoothly interpolate between samples conditioned on redshift. Deep generative models, such as the ones described in this work, provide great promise as fast, low-memory, high-fidelity forward models of large-scale structure.

astro-ph.CO↗

arXiv2020

The Websky Extragalactic CMB Simulations

We present a new pipeline for the efficient generation of synthetic observations of the extragalactic microwave sky, tailored to large ground-based CMB experiments such as the Simons Observatory, Advanced ACTPol, SPT-3G, and CMB-S4. Such simulated observations are a key technical challenge in cosmology because of the dynamic range and accuracy required. The first part of the pipeline generates a random cosmological realization in the form of a dark matter halo catalog and matter displacement field, as seen from a given position. The halo catalog and displacement field are modeled with ellipsoidal collapse dynamics and Lagrangian perturbation theory, respectively. In the second part, the cosmological realization is converted into a set of intensity maps over the range 10 - 10^3 GHz using models based on existing observations and hydrodynamical simulations. These maps include infrared emission from dusty star forming galaxies (CIB), Comptonization of CMB photons by hot gas in groups and clusters through the thermal Sunyaev-Zel'dovich effect (tSZ), Doppler boosting by Thomson scattering of the CMB by bulk flows through the kinetic Sunyaev-Zel'dovich effect (kSZ), and weak gravitational lensing of primary CMB anisotropies by the large-scale distribution of matter in the universe. After describing the pipeline and its implementation, we present the Websky maps, created from a realization of the cosmic web on our past light cone in the redshift interval 0<z<4.6 over the full-sky and a volume of ~(600 Gpc/h)^3 resolved with ~10^12 resolution elements. The Websky maps and halo catalog are publicly available at mocks.cita.utoronto.ca/websky.

astro-ph.CO↗

arXiv2019

Deconfusing intensity maps with neural networks

Line intensity maps (LIMs) are in principle sensitive to a large amount of information about faint, distant galaxies which are invisible to conventional surveys. However, actually extracting that information from a confused, foreground-contaminated map can be challenging. In this work we present the first application of convolutional neural network (CNN) to directly determine the underlying luminosity function of a LIM, including a treatment of extragalactic foregrounds and instrumental noise. We apply the CNN to simulations of mock Carbon Monoxide (CO) line intensity maps similar to those which will be produced by the currently-active COMAP experiment. We evaluate the trained CNN on a number of noise scenarios in order to determine how robust the network predictions are for application to realistic data. We find that, in the ideal case where the mock data capture all of the features of the real data, the CNN performs comparably to or better than conventional analyses. However, the network's accuracy degrades considerably when tested on signals and systematics outside of those it was trained on. For both intensity mapping and cosmology as a whole, this motivates a broad-based study of whether simulated data can ever be generated with sufficient detail to realize the enormous potential of machine learning methods.

astro-ph.CO↗

arXiv2019

Comparing approximate methods for mock catalogues and covariance matrices I: correlation function

This paper is the first in a set that analyses the covariance matrices of clustering statistics obtained from several approximate methods for gravitational structure formation. We focus here on the covariance matrices of anisotropic two-point correlation function measurements. Our comparison includes seven approximate methods, which can be divided into three categories: predictive methods that follow the evolution of the linear density field deterministically (ICE-COLA, Peak Patch, and Pinocchio), methods that require a calibration with N-body simulations (Patchy and Halogen), and simpler recipes based on assumptions regarding the shape of the probability distribution function (PDF) of density fluctuations (log-normal and Gaussian density fields). We analyse the impact of using covariance estimates obtained from these approximate methods on cosmological analyses of galaxy clustering measurements, using as a reference the covariances inferred from a set of full N-body simulations. We find that all approximate methods can accurately recover the mean parameter values inferred using the N-body covariances. The obtained parameter uncertainties typically agree with the corresponding N-body results within 5% for our lower mass threshold, and 10% for our higher mass threshold. Furthermore, we find that the constraints for some methods can differ by up to 20% depending on whether the halo samples used to define the covariance matrices are defined by matching the mass, number density, or clustering amplitude of the parent N-body samples. The results of our configuration-space analysis indicate that most approximate methods provide similar results, with no single method clearly outperforming the others.

astro-ph.CO↗