arXiv subjects

Michael S. Warren

Publications and source records attributed to Michael S. Warren.

At least 19 recordsLinked to original sources

arXiv2020

Mobility Changes in Response to COVID-19

In response to the COVID-19 pandemic, both voluntary changes in behavior and administrative restrictions on human interactions have occurred. These actions are intended to reduce the transmission rate of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). We use anonymized and/or de-identified mobile device locations to measure mobility, a statistic representing the distance a typical member of a given population moves in a day. Results indicate that a large reduction in mobility has taken place, both in the US and globally. In the United States, large mobility reductions have been detected associated with the onset of the COVID-19 threat and specific government directives. Mobility data at the US admin1 (state) and admin2 (county) level have been made freely available under a Creative Commons Attribution (CC BY 4.0) license via the GitHub repository https://github.com/descarteslabs/DL-COVID-19/

cs.SI↗

arXiv2020

Visual search over billions of aerial and satellite images

We present a system for performing visual search over billions of aerial and satellite images. The purpose of visual search is to find images that are visually similar to a query image. We define visual similarity using 512 abstract visual features generated by a convolutional neural network that has been trained on aerial and satellite imagery. The features are converted to binary values to reduce data and compute requirements. We employ a hash-based search using Bigtable, a scalable database service from Google Cloud. Searching the continental United States at 1-meter pixel resolution, corresponding to approximately 2 billion images, takes approximately 0.1 seconds. This system enables real-time visual search over the surface of the earth, and an interactive demo is available at https://search.descarteslabs.com.

cs.CV↗

arXiv2020

BUDD: Multi-modal Bayesian Updating Deforestation Detections

The global phenomenon of forest degradation is a pressing issue with severe implications for climate stability and biodiversity protection. In this work we generate Bayesian updating deforestation detection (BUDD) algorithms by incorporating Sentinel-1 backscatter and interferometric coherence with Sentinel-2 normalized vegetation index data. We show that the algorithm provides good performance in validation AOIs. We compare the effectiveness of different combinations of the three data modalities as inputs into the BUDD algorithm and compare against existing benchmarks based on optical imagery.

stat.AP↗

arXiv2017

Data-Intensive Supercomputing in the Cloud: Global Analytics for Satellite Imagery

We present our experiences using cloud computing to support data-intensive analytics on satellite imagery for commercial applications. Drawing from our background in high-performance computing, we draw parallels between the early days of clustered computing systems and the current state of cloud computing and its potential to disrupt the HPC market. Using our own virtual file system layer on top of cloud remote object storage, we demonstrate aggregate read bandwidth of 230 gigabytes per second using 512 Google Compute Engine (GCE) nodes accessing a USA multi-region standard storage bucket. This figure is comparable to the best HPC storage systems in existence. We also present several of our application results, including the identification of field boundaries in Ukraine, and the generation of a global cloud-free base layer from Landsat imagery.

cs.DC↗

arXiv2016

Disentangling redshift-space distortions and nonlinear bias using the 2D power spectrum

We present the nonlinear 2D galaxy power spectrum, $P(k,μ)$, in redshift space, measured from the Dark Sky simulations, using galaxy catalogs constructed with both halo occupation distribution and subhalo abundance matching methods, chosen to represent an intermediate redshift sample of luminous red galaxies. We find that the information content in individual $μ$ (cosine of the angle to the line of sight) bins is substantially richer then multipole moments, and show that this can be used to isolate the impact of nonlinear growth and redshift space distortion (RSD) effects. Using the $μ<0.2$ simulation data, which we show is not impacted by RSD effects, we can successfully measure the nonlinear bias to an accuracy of $\sim 5$% at $k<0.6 h$Mpc$^{-1}$. This use of individual $μ$ bins to extract the nonlinear bias successfully removes a large parameter degeneracy when constraining the linear growth rate of structure. We carry out a joint parameter estimation, using the low $μ$ simulation data to constrain the nonlinear bias, and $μ\ge0.2$ to constrain the growth rate and show that $f$ can be constrained to $\sim 26\, (22)$% to a $k_{\rm max}< 0.4\, (0.6) h$Mpc$^{-1}$ from clustering alone using a simple dispersion model, for a range of galaxy models. Our analysis of individual $μ$ bins also reveals interesting physical effects which arise simply from different methods of populating halos with galaxies. We find a prominent turnaround scale, at which RSD damping effects are greater then the nonlinear growth, which differs not only for each $μ$ bin but also for each galaxy model. These features may provide unique signatures which could be used to shed light on the galaxy-dark matter connection.

astro-ph.CO↗

arXiv2014

Reconstructing the velocity field beyond the local universe

We present a maximum probability approach to reconstructing spatial maps of the peculiar velocity field at redshifts $z\sim0.1$, where the velocities have been measured from distance indicators (DI) such as $D_n-σ$ relations or Tully-Fisher. With the large statistical uncertainties associated with DIs, our reconstruction method aims to recover the underlying true peculiar velocity field by reducing these errors with the use of two physically motivated filtering prior terms. The first constructs an estimate of the velocity field derived from the galaxy over-density {\deltag} and the second makes use of the matter linear density power spectrum {\pk}. Using $N$-body simulations we find, with an SDSS-like sample ($N_{gal}\simeq33$ per deg$^2$), an average correlation coefficient value of $r=0.55\pm{0.02}$ between our reconstructed velocity field and that of the true velocity field from the simulation. However, with a suitably high number density of galaxies from the next generation surveys (e.g. $N_{gal}\simeq140$ per deg$^2$) we can achieve an average $r=0.70\pm{0.02}$ out to moderate redshifts $z\sim0.1$. This will prove useful for future tests of gravity, as these relatively deep maps are complementary to weak lensing maps at the same redshift.

astro-ph.CO↗

arXiv2014

VIDE: The Void IDentification and Examination toolkit

We present VIDE, the Void IDentification and Examination toolkit, an open-source Python/C++ code for finding cosmic voids in galaxy redshift surveys and N-body simulations, characterizing their properties, and providing a platform for more detailed analysis. At its core, VIDE uses a substantially enhanced version of ZOBOV (Neyinck 2008) to calculate a Voronoi tessellation for estimating the density field and a performing a watershed transform to construct voids. Additionally, VIDE provides significant functionality for both pre- and post-processing: for example, vide can work with volume- or magnitude-limited galaxy samples with arbitrary survey geometries, or dark matter particles or halo catalogs in a variety of common formats. It can also randomly subsample inputs and includes a Halo Occupation Distribution model for constructing mock galaxy populations. VIDE uses the watershed levels to place voids in a hierarchical tree, outputs a summary of void properties in plain ASCII, and provides a Python API to perform many analysis tasks, such as loading and manipulating void catalogs and particle members, filtering, plotting, computing clustering statistics, stacking, comparing catalogs, and fitting density profiles. While centered around ZOBOV, the toolkit is designed to be as modular as possible and accommodate other void finders. VIDE has been in development for several years and has already been used to produce a wealth of results, which we summarize in this work to highlight the capabilities of the toolkit. VIDE is publicly available at http://bitbucket.org/cosmicvoids/vide public and http://www.cosmicvoids.net.

astro-ph.CO↗

arXiv2014

Dark Sky Simulations: Early Data Release

The Dark Sky Simulations are an ongoing series of cosmological N-body simulations designed to provide a quantitative and accessible model of the evolution of the large-scale Universe. Such models are essential for many aspects of the study of dark matter and dark energy, since we lack a sufficiently accurate analytic model of non-linear gravitational clustering. In July 2014, we made available to the general community our early data release, consisting of over 55 Terabytes of simulation data products, including our largest simulation to date, which used $1.07 \times 10^{12}~(10240^3)$ particles in a volume $8h^{-1}\mathrm{Gpc}$ across. Our simulations were performed with 2HOT, a purely tree-based adaptive N-body method, running on 200,000 processors of the Titan supercomputer, with data analysis enabled by yt. We provide an overview of the derived halo catalogs, mass function, power spectra and light cone data. We show self-consistency in the mass function and mass power spectrum at the 1% level over a range of more than 1000 in particle mass. We also present a novel method to distribute and access very large datasets, based on an abstraction of the World Wide Web (WWW) as a file system, remote memory-mapped file access semantics, and a space-filling curve index. This method has been implemented for our data release, and provides a means to not only query stored results such as halo catalogs, but also to design and deploy new analysis techniques on large distributed datasets.

astro-ph.CO↗

arXiv2014

Voids in the SDSS DR9: observations, simulations, and the impact of the survey mask

We present and study cosmic voids identified using the watershed void finder VIDE in the Sloan Digital Sky Survey Data Release 9, compare these voids to ones identified in mock catalogs, and assess the impact of the survey mask on void statistics such as number functions, ellipticity distributions, and radial density profiles. The nearly 1,000 identified voids span three nearly volume-limited samples from redshift z = 0.43 to 0.7. For comparison we use 98 of the publicly available 2LPT-based mock galaxy catalogs of Manera et al., and also generate our own mock catalogs by applying a Halo Occupation Distribution model to an N-body simulation. We find that the mask reduces the number density of voids at all scales by a factor of three and slightly skews the relative size distributions. This engenders an increase in the mean ellipticity by roughly 30%. However, we find that radial density profiles are largely robust to the effects of the mask. We see excellent agreement between the data and both mock catalogs, and find no tension between the observed void properties and the properties derived from ΛCDM simulations. We have added the void catalogs from both data and mock galaxy populations discussed in this work to the Public Cosmic Void Catalog at http://www.cosmicvoids.net.

astro-ph.CO↗

arXiv2014

Sparse sampling, galaxy bias, and voids

To study the impact of sparsity and galaxy bias on void statistics, we use a single large-volume, high-resolution N-body simulation to compare voids in multiple levels of subsampled dark matter, halo populations, and mock galaxies from a Halo Occupation Distribution model tuned to different galaxy survey densities. We focus our comparison on three key observational statistics: number functions, ellipticity distributions, and radial density profiles. We use the hierarchical tree structure of voids to interpret the impacts of sampling density and galaxy bias, and theoretical and empirical functions to describe the statistics in all our sample populations. We are able to make simple adjustments to theoretical expectations to offer prescriptions for translating from analytics to the void properties measured in realistic observations. We find that sampling density has a much larger effect on void sizes than galaxy bias. At lower tracer density, small voids disappear and the remaining voids are larger, more spherical, and have slightly steeper profiles. When a proper lower mass threshold is chosen, voids in halo distributions largely mimic those found in galaxy populations, except for ellipticities, where galaxy bias leads to higher values. We use the void density profile of Hamaus et al. (2014) to show that voids follow a self-similar and universal trend, allowing simple translations between voids studied in dark matter and voids identified in galaxy surveys. We have added the mock void catalogs used in this work to the Public Cosmic Void Catalog at http://www.cosmicvoids.net.

astro-ph.CO↗

arXiv2014

Cosmology with Void-Galaxy Correlations

Galaxy bias, the unknown relationship between the clustering of galaxies and the underlying dark matter density field is a major hurdle for cosmological inference from large-scale structure. While traditional analyses focus on the absolute clustering amplitude of high-density regions mapped out by galaxy surveys, we propose a relative measurement that compares those to the underdense regions, cosmic voids. On the basis of realistic mock catalogs we demonstrate that cross correlating galaxies and voids opens up the possibility to calibrate galaxy bias and to define a static ruler thanks to the observable geometric nature of voids. We illustrate how the clustering of voids is related to mass compensation and show that volume-exclusion significantly reduces the degree of stochasticity in their spatial distribution. Extracting the spherically averaged distribution of galaxies inside voids from their cross correlations reveals a remarkable concordance with the mass-density profile of voids.

astro-ph.CO↗

arXiv2014

The dark matter of galaxy voids

How do observed voids relate to the underlying dark matter distribution? To examine the spatial distribution of dark matter contained within voids identified in galaxy surveys, we apply Halo Occupation Distribution models representing sparsely and densely sampled galaxy surveys to a high-resolution N-body simulation. We compare these galaxy voids to voids found in the halo distribution, low-resolution dark matter, and high-resolution dark matter. We find that voids at all scales in densely sampled surveys - and medium- to large-scale voids in sparse surveys - trace the same underdensities as dark matter, but they are larger in radius by ~20%, they have somewhat shallower density profiles, and they have centers offset by ~0.4Rv rms. However, in void-to-void comparison we find that shape estimators are less robust to sampling, and the largest voids in sparsely sampled surveys suffer fragmentation at their edges. We find that voids in galaxy surveys always correspond to underdensities in the dark matter, though the centers may be offset. When this offset is taken into account, we recover almost identical radial density profiles between galaxies and dark matter. All mock catalogs used in this work are available at http://www.cosmicvoids.net.

astro-ph.CO↗

arXiv2013

2HOT: An Improved Parallel Hashed Oct-Tree N-Body Algorithm for Cosmological Simulation

We report on improvements made over the past two decades to our adaptive treecode N-body method (HOT). A mathematical and computational approach to the cosmological N-body problem is described, with performance and scalability measured up to 256k ($2^{18}$) processors. We present error analysis and scientific application results from a series of more than ten 69 billion ($4096^3$) particle cosmological simulations, accounting for $4 \times 10^{20}$ floating point operations. These results include the first simulations using the new constraints on the standard model of cosmology from the Planck satellite. Our simulations set a new standard for accuracy and scientific throughput, while meeting or exceeding the computational efficiency of the latest generation of hybrid TreePM N-body methods.

astro-ph.IM↗

arXiv2012

On the Rates of Type Ia Supernovae in Dwarf and Giant Hosts with ROTSE-IIIb

We present a sample of 23 spectroscopically confirmed Type Ia supernovae that were discovered in the background of galaxy clusters targeted by ROTSE-IIIb and use up to 18 of these to determine the local (z = 0.05) volumetric rate. Since our survey is flux limited and thus biased against fainter objects, the pseudo-absolute magnitude distribution (pAMD) of SNeIa in a given volume is an important concern, especially the relative frequency of high to low-luminosity SNeIa. We find that the pAMD derived from the volume limited Lick Observatory Supernova Search (LOSS) sample is incompatible with the distribution of SNeIa in a volume limited (z<0.12) sub sample of the SDSS-II. The LOSS sample requires far more low-luminosity SNeIa than the SDSS-II can accommodate. Even though LOSS and SDSS-II have sampled different SNeIa populations, their volumetric rates are surprisingly similar. Using the same model pAMD adopted in the SDSS-II SNeIa rate calculation and excluding two high-luminosity SNeIa from our sample, we derive a rate that is marginally higher than previous low-redshift determinations. With our full sample and the LOSS pAMD our rate is more than double the canonical value. We also find that 5 of our 18 SNeIa are hosted by very low-luminosity (M_B > -16) galaxies, whereas only 1 out 79 nearby SDSS-II SNeIa have such faint hosts. It is possible that previous works have under-counted either low luminosity SNeIa, SNeIa in low luminosity hosts, or peculiar SNeIa (sometimes explicitly), and the total SNeIa rate may be higher than the canonical value.

astro-ph.CO↗

arXiv2011

A Parametrized Galaxy Catalog Simulator For Testing Cluster Finding, Mass Estimation and Photometric Redshift Estimation in Optical and Near Infrared Surveys

We present a galaxy catalog simulator which turns N-body simulations with subhalos into multiband photometric mocks. The simulator assigns galaxy properties to each subhalo to reproduce the observed cluster galaxy halo occupation distribution, the radial and mass dependent variation in fractions of blue galaxies, the luminosity functions in clusters and the field, and the red-sequence in clusters. Moreover, the evolution of these parameters is tuned to match existing observational constraints. Field galaxies are sampled from existing multiband photometric surveys using derived galaxy photometric redshifts. Parametrizing an ensemble of cluster galaxy properties enables us to create mock catalogs with variations in those properties, which in turn allows us to quantify the sensitivity of cluster finding to current observational uncertainties in these properties. We present an application of the catalog simulator to characterize the selection function of a galaxy cluster finder that utilizes the cluster red-sequence galaxy clustering on the sky, in terms of completeness and contamination. We estimate systematic uncertainties due to the observational uncertainties on our simulator parameters in determining the selection function using five different sets of modified catalogs. Our estimates indicate that these uncertainties are at the $\le15$% level with current observational constraints on cluster galaxy populations and their evolution. In addition, we examine the $B_{gc}$ parameter as an optical mass indicator and measure the intrinsic scatter of the $B_{gc}$--mass relation to be approximately log normal with $σ_{\log_{10}M}\sim0.25$. Finally, we present tests of a red sequence overdensity redshift estimator using both simulated and real data, showing that it delivers redshifts for massive clusters with $\sim$2% accuracy out to redshifts $z\sim0.5$ with SDSS-like datasets.

astro-ph.CO↗

arXiv2010

The Large Scale Bias of Dark Matter Halos: Numerical Calibration and Model Tests

We measure the clustering of dark matter halos in a large set of collisionless cosmological simulations of the flat LCDM cosmology. Halos are identified using the spherical overdensity algorithm, which finds the mass around isolated peaks in the density field such that the mean density is Delta times the background. We calibrate fitting functions for the large scale bias that are adaptable to any value of Delta we examine. We find a ~6% scatter about our best fit bias relation. Our fitting functions couple to the halo mass functions of Tinker et. al. (2008) such that bias of all dark matter is normalized to unity. We demonstrate that the bias of massive, rare halos is higher than that predicted in the modified ellipsoidal collapse model of Sheth, Mo, & Tormen (2001), and approaches the predictions of the spherical collapse model for the rarest halos. Halo bias results based on friends-of-friends halos identified with linking length 0.2 are systematically lower than for halos with the canonical Delta=200 overdensity by ~10%. In contrast to our previous results on the mass function, we find that the universal bias function evolves very weakly with redshift, if at all. We use our numerical results, both for the mass function and the bias relation, to test the peak-background split model for halo bias. We find that the peak-background split achieves a reasonable agreement with the numerical results, but ~20% residuals remain, both at high and low masses.

astro-ph.CO↗

arXiv2008

Peculiar Velocities into the Next Generation: Cosmological Parameters From Large Surveys without Bias from Nonlinear Structure

We investigate methods to best estimate the normalisation of the mass density fluctuation power spectrum (sigma_8) using peculiar velocity data from a survey like the Six degree Field Galaxy Velocity Survey (6dFGSv). We focus on two potential problems (i) biases from nonlinear growth of structure and (ii) the large number of velocities in the survey. Simulations of LambdaCDM-like models are used to test the methods. We calculate the likelihood from a full covariance matrix of velocities averaged in grid cells. This simultaneously reduces the number of data points and smooths out nonlinearities which tend to dominate on small scales. We show how the averaging can be taken into account in the predictions in a practical way, and show the effect of the choice of cell size. We find that a cell size can be chosen that significantly reduces the nonlinearities without significantly increasing the error bars on cosmological parameters. We compare our results with those from a principal components analysis following Watkins et al (2002) and Feldman et al (2003) to select a set of optimal moments constructed from linear combinations of the peculiar velocities that are least sensitive to the nonlinear scales. We conclude that averaging in grid cells performs equally well. We find that for a survey such as 6dFGSv we can estimate sigma_8 with less than 3% bias from nonlinearities. The expected error on sigma_8 after marginalising over Omega_m is approximately 16 percent.

astro-ph↗

arXiv2008

Extending Recovery of the Primordial Matter Power Spectrum

The shape of the primordial matter power spectrum Plin(k) encodes critical information on cosmological parameters. At large scales, the observable galaxy power spectrum Pobs(k) is expected to follow the shape of Plin(k), but on smaller scales the effects of nonlinearity and galaxy bias make the ratio Pobs(k)/Plin(k) scale-dependent. We develop a method that can extend the dynamic range of the Plin(k) recovery by incorporating constraints on the galaxy halo occupation distribution (HOD) from the projected galaxy correlation function wp. We devise an analytic model to calculate Pobs(k) in real-space and redshift-space. Once HOD parameters are determined by matching wp for a given cosmological model, galaxy bias is completely specified, and our analytic model predicts both the shape and normalization of Pobs(k). Applying our method to SDSS main galaxy samples, we find that the real-space Pobs(k) follows the shape of the nonlinear matter power spectrum at the 1-2% level up to k=0.2 h/Mpc. When we apply our method to SDSS LRG samples, the linear bias approximation is accurate to 5% at k<0.08 h/Mpc, but the scale-dependence of LRG bias prevents the use of linear theory at k>0.08 h/Mpc. Our HOD model prediction is in good agreement with the recent SDSS LRG Pobs(k) measurements at all measured scales (k<0.2 h/Mpc), naturally explaining the shape of Pobs(k). The "Q-model" prescription is a poor description of galaxy bias for the LRG samples, and it can lead to biased cosmological parameter estimates when measurements at k>0.1 h/Mpc are included in the analysis. We quantify the potential bias and constraints on cosmological parameters that arise from applying linear theory and Q-model fitting, and we demonstrate the utility of HOD modeling of future high precision measurements of Pobs(k) on quasi-linear scales.

astro-ph↗