arXiv subjects

K. Simon Krughoff

Publications and source records attributed to K. Simon Krughoff.

14 recordsLinked to original sources

arXiv2025

DESC DC2 Data Release Note

In preparation for cosmological analyses of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST), the LSST Dark Energy Science Collaboration (LSST DESC) has created a 300 deg$^2$ simulated survey as part of an effort called Data Challenge 2 (DC2). The DC2 simulated sky survey, in six optical bands with observations following a reference LSST observing cadence, was processed with the LSST Science Pipelines (19.0.0). In this Note, we describe the public data release of the resulting object catalogs for the coadded images of five years of simulated observations along with associated truth catalogs. We include a brief description of the major features of the available data sets. To enable convenient access to the data products, we have developed a web portal connected to Globus data services. We describe how to access the data and provide example Jupyter Notebooks in Python to aid first interactions with the data. We welcome feedback and questions about the data release via a GitHub repository.

astro-ph.CO↗

arXiv2022

Software Architecture and System Design of Rubin Observatory

Starting from a description of the Rubin Observatory Data Management System Architecture, and drawing on our experience with and involvement in a range of other projects including Gaia, SDSS, UKIRT, and JCMT, we derive a series of generic design patterns and lessons learned.

astro-ph.IM↗

arXiv2022

Faro: A framework for measuring the scientific performance of petascale Rubin Observatory data products

The Vera C. Rubin Observatory will advance many areas of astronomy over the next decade with its unique wide-fast-deep multi-color imaging survey, the Legacy Survey of Space and Time (LSST). The LSST will produce approximately 20TB of raw data per night, which will be automatically processed by the LSST Science Pipelines to generate science-ready data products -- processed images, catalogs and alerts. To ensure that these data products enable transformative science with LSST, stringent requirements have been placed on their quality and scientific fidelity, for example on image quality and depth, astrometric and photometric performance, and object recovery completeness. In this paper we introduce faro, a framework for automatically and efficiently computing scientific performance metrics on the LSST data products for units of data of varying granularity, ranging from single-detector to full-survey summary statistics. By measuring and monitoring metrics, we are able to evaluate trends in algorithmic performance and conduct regression testing during development, compare the performance of one algorithm against another, and verify that the LSST data products will meet performance requirements by comparing to specifications. We present initial results using faro to characterize the performance of the data products produced on simulated and precursor data sets, and discuss plans to use faro to verify the performance of the LSST commissioning data products.

astro-ph.IM↗

arXiv2021

Rubin Science Platform on Google: the story so far

We describe Rubin Observatory's experience with offering a data access facility (and associated services including our Science Platform) deployed on Google Cloud infrastructure as part of our pre-Operations Data Preview program.

astro-ph.IM↗

arXiv2021

The LSST DESC DC2 Simulated Sky Survey

We describe the simulated sky survey underlying the second data challenge (DC2) carried out in preparation for analysis of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) by the LSST Dark Energy Science Collaboration (LSST DESC). Significant connections across multiple science domains will be a hallmark of LSST; the DC2 program represents a unique modeling effort that stresses this interconnectivity in a way that has not been attempted before. This effort encompasses a full end-to-end approach: starting from a large N-body simulation, through setting up LSST-like observations including realistic cadences, through image simulations, and finally processing with Rubin's LSST Science Pipelines. This last step ensures that we generate data products resembling those to be delivered by the Rubin Observatory as closely as is currently possible. The simulated DC2 sky survey covers six optical bands in a wide-fast-deep (WFD) area of approximately 300 deg^2 as well as a deep drilling field (DDF) of approximately 1 deg^2. We simulate 5 years of the planned 10-year survey. The DC2 sky survey has multiple purposes. First, the LSST DESC working groups can use the dataset to develop a range of DESC analysis pipelines to prepare for the advent of actual data. Second, it serves as a realistic testbed for the image processing software under development for LSST by the Rubin Observatory. In particular, simulated data provide a controlled way to investigate certain image-level systematic effects. Finally, the DC2 sky survey enables the exploration of new scientific ideas in both static and time-domain cosmology.

astro-ph.IM↗

arXiv2019

Why is the LSST Science Platform built on Kubernetes?

LSST has chosen Kubernetes as the platform for deploying and operating the LSST Science Platform. We first present the background reasoning behind this decision, including both instrument-agnostic as well as LSST-specific requirements. We then discuss the basic principles of Kubernetes and Helm, and how they are used as the deployment base for the LSST Science Platform. Furthermore, we provide an example of how an external group may use these publicly available software resources to deploy their own instance of the LSST Science Platform, and customize it to their needs. Finally, we discuss how more astronomy software can follow these patterns to gain similar benefits.

astro-ph.IM↗

arXiv2019

The Zwicky Transient Facility Alert Distribution System

The Zwicky Transient Facility (ZTF) survey generates real-time alerts for optical transients, variables, and moving objects discovered in its wide-field survey. We describe the ZTF alert stream distribution and processing (filtering) system. The system uses existing open-source technologies developed in industry: Kafka, a real-time streaming platform, and Avro, a binary serialization format. The technologies used in this system provide a number of advantages for the ZTF use case, including (1) built-in replication, scalability, and stream rewind for the distribution mechanism; (2) structured messages with strictly enforced schemas and dynamic typing for fast parsing; and (3) a Python-based stream processing interface that is similar to batch for a familiar and user-friendly plug-in filter system, all in a modular, primarily containerized system. The production deployment has successfully supported streaming up to 1.2 million alerts or roughly 70 GB of data per night, with each alert available to a consumer within about 10 s of alert candidate production. Data transfer rates of about 80,000 alerts/minute have been observed. In this paper, we discuss this alert distribution and processing system, the design motivations for the technology choices for the framework, performance in production, and how this system may be generally suitable for other alert stream use cases, including the upcoming Large Synoptic Survey Telescope.

astro-ph.IM↗

arXiv2018

An Overview of the LSST Image Processing Pipelines

The Large Synoptic Survey Telescope (LSST) is an ambitious astronomical survey with a similarly ambitious Data Management component. Data Management for LSST includes processing on both nightly and yearly cadences to generate transient alerts, deep catalogs of the static sky, and forced photometry light-curves for billions of objects at hundreds of epochs, spanning at least a decade. The algorithms running in these pipelines are individually sophisticated and interact in subtle ways. This paper provides an overview of those pipelines, focusing more on those interactions than the details of any individual algorithm.

astro-ph.IM↗

arXiv2018

DESCQA: An Automated Validation Framework for Synthetic Sky Catalogs

The use of high-quality simulated sky catalogs is essential for the success of cosmological surveys. The catalogs have diverse applications, such as investigating signatures of fundamental physics in cosmological observables, understanding the effect of systematic uncertainties on measured signals and testing mitigation strategies for reducing these uncertainties, aiding analysis pipeline development and testing, and survey strategy optimization. The list of applications is growing with improvements in the quality of the catalogs and the details that they can provide. Given the importance of simulated catalogs, it is critical to provide rigorous validation protocols that enable both catalog providers and users to assess the quality of the catalogs in a straightforward and comprehensive way. For this purpose, we have developed the DESCQA framework for the Large Synoptic Survey Telescope Dark Energy Science Collaboration as well as for the broader community. The goal of DESCQA is to enable the inspection, validation, and comparison of an inhomogeneous set of synthetic catalogs via the provision of a common interface within an automated framework. In this paper, we present the design concept and first implementation of DESCQA. In order to establish and demonstrate its full functionality we use a set of interim catalogs and validation tests. We highlight several important aspects, both technical and scientific, that require thoughtful consideration when designing a validation framework, including validation metrics and how these metrics impose requirements on the synthetic sky catalogs.

astro-ph.IM↗

arXiv2017

The Hyper Suprime-Cam Software Pipeline

In this paper, we describe the optical imaging data processing pipeline developed for the Subaru Telescope's Hyper Suprime-Cam (HSC) instrument. The HSC Pipeline builds on the prototype pipeline being developed by the Large Synoptic Survey Telescope's Data Management system, adding customizations for HSC, large-scale processing capabilities, and novel algorithms that have since been reincorporated into the LSST codebase. While designed primarily to reduce HSC Subaru Strategic Program (SSP) data, it is also the recommended pipeline for reducing general-observer HSC data. The HSC pipeline includes high level processing steps that generate coadded images and science-ready catalogs as well as low-level detrending and image characterizations.

astro-ph.IM↗

arXiv2015

The LSST Data Management System

The Large Synoptic Survey Telescope (LSST) is a large-aperture, wide-field, ground-based survey system that will image the sky in six optical bands from 320 to 1050 nm, uniformly covering approximately $18,000$deg$^2$ of the sky over 800 times. The LSST is currently under construction on Cerro Pachón in Chile, and expected to enter operations in 2022. Once operational, the LSST will explore a wide range of astrophysical questions, from discovering "killer" asteroids to examining the nature of Dark Energy. The LSST will generate on average 15 TB of data per night, and will require a comprehensive Data Management system to reduce the raw data to scientifically useful catalogs and images with minimum human intervention. These reductions will result in a real-time alert stream, and eleven data releases over the 10-year duration of LSST operations. To enable this processing, the LSST project is developing a new, general-purpose, high-performance, scalable, well documented, open source data processing software stack for O/IR surveys. Prototypes of this stack are already capable of processing data from existing cameras (e.g., SDSS, DECam, MegaCam), and form the basis of the Hyper-Suprime Cam (HSC) Survey data reduction pipeline.

astro-ph.IM↗

arXiv2015

Improving the LSST dithering pattern and cadence for dark energy studies

The Large Synoptic Survey Telescope (LSST) will explore the entire southern sky over 10 years starting in 2022 with unprecedented depth and time sampling in six filters, $ugrizy$. Artificial power on the scale of the 3.5 deg LSST field-of-view will contaminate measurements of baryonic acoustic oscillations (BAO), which fall at the same angular scale at redshift $z \sim 1$. Using the HEALPix framework, we demonstrate the impact of an "un-dithered" survey, in which $17\%$ of each LSST field-of-view is overlapped by neighboring observations, generating a honeycomb pattern of strongly varying survey depth and significant artificial power on BAO angular scales. We find that adopting large dithers (i.e., telescope pointing offsets) of amplitude close to the LSST field-of-view radius reduces artificial structure in the galaxy distribution by a factor of $\sim$10. We propose an observing strategy utilizing large dithers within the main survey and minimal dithers for the LSST Deep Drilling Fields. We show that applying various magnitude cutoffs can further increase survey uniformity. We find that a magnitude cut of $r < 27.3$ removes significant spurious power from the angular power spectrum with a minimal reduction in the total number of observed galaxies over the ten-year LSST run. We also determine the effectiveness of the observing strategy for Type Ia SNe and predict that the main survey will contribute $\sim$100,000 Type Ia SNe. We propose a concentrated survey where LSST observes one-third of its main survey area each year, increasing the number of main survey Type Ia SNe by a factor of $\sim$1.5, while still enabling the successful pursuit of other science drivers.

astro-ph.IM↗

arXiv2011

Spectroscopic Determination of the Low Redshift Type Ia Supernova Rate from the Sloan Digital Sky Survey

Supernova rates are directly coupled to high mass stellar birth and evolution. As such, they are one of the few direct measures of the history of cosmic stellar evolution. In this paper we describe an probabilistic technique for identifying supernovae within spectroscopic samples of galaxies. We present a study of 52 type Ia supernovae ranging in age from -14 days to +40 days extracted from a parent sample of \simeq 50,000 spectra from the SDSS DR5. We find a Supernova Rate (SNR) of 0.472^{+0.048}_{-0.039}(Systematic)^{+0.081}_{-0.071}(Statistical)SNu at a redshift of = 0.1. This value is higher than other values at low redshift at the 1σ, but is consistent at the 3σ level. The 52 supernova candidates used in this study comprise the third largest sample of supernovae used in a type Ia rate determination to date. In this paper we demonstrate the potential for the described approach for detecting supernovae in future spectroscopic surveys.

astro-ph.CO↗

arXiv2005

Inter-cluster filaments in a $Λ$CDM Universe

The large--scale structure (LSS) in the Universe comprises a complicated filamentary network of matter. We study this network using a high--resolution simulation of structure formation of a $Λ$ Cold Dark Matter cosmology. We investigate the distribution of matter between neighbouring large haloes whose masses are comparable to massive clusters of galaxies. We identify a total of 228 filaments between neighbouring clusters. Roughly half of the filaments are either warped or lie off the cluster--cluster axis. We find that straight filaments on the average are shorter than warped ones. More massive clusters are connected to more filaments than less massive ones on average. This finding indicates that the most massive clusters form at the intersections of the filamentary backbone of LSS. For straight filaments, we compute mass profiles. Radial profiles show a fairly well--defined radius, $r_s$, beyond which the profiles follow an $r^{-2}$ power law fairly closely. For the majority of filaments, $r_s$ lies between 1.5 $h^{-1}$ Mpc and 2.0 $h^{-1}$ Mpc. The enclosed overdensity inside $r_s$ varies between a few times up to 25 times mean density, independent of the length of the filaments. Along the filaments' axes, material is not distributed uniformly. Towards the clusters, the density rises, indicating the presence of the cluster infall regions. In addition, we also find some sheet--like connections between clusters. In roughly a fifth of all cluster--cluster connections where we could not identify a filament or sheet, projection effects lead to filamentary structures in the projected mass distribution. (abridged)

astro-ph↗