arXiv subjects

G. Bruce Berriman

Publications and source records attributed to G. Bruce Berriman.

At least 19 recordsLinked to original sources

arXiv2026

A New Plotly-Dash-based Query Infrastructure for the Keck Observatory Archive

The Keck Observatory Archive (KOA) curates all observational data acquired at the W. M. Keck Observatory. The archive is expected to grow rapidly as complex new instruments are commissioned and as the expectations of archive users have expanded. In response, KOA has implemented a new Python-based, VO-compliant query infrastructure. This work is a continuation of the architectural design and technology selection identified at ADASS 2024. We have deployed real-time ingestion of newly acquired data and a dedicated interface for observers to manage these data. Our ADASS 2024 poster identified the new technologies chosen: Plotly-Dash, a low-code framework that exploits event-driven callbacks to simplify the handling of user interactions; R-tree spatial indexing to speed up spatial searches by x20; a VO-compliant TAP middleware, already in use at the NASA Exoplanet Archive and NEID archive; and mViewer, a visualization engine in the Montage Image Mosaic toolkit that is optimized for astronomy images. These technologies will underpin new services that can be hosted on web pages or in Jupyter notebooks, and when completed, will replace the current query infrastructure. We have completed two new services now in beta release. The first is the Data Discovery Service, a web-based dashboard that returns spatial and temporal queries of the entire archive in seconds. It supports filtering observations by keywords, previewing results in an interactive data grid, and visualizing images, and it offers data downloads. The second is a Jupyter notebook that performs interactive visualization of Keck observations of protostars in the Rho Oph Dark Cloud and uses data from CDS and IRSA, as well as KOA.

astro-ph.IM↗

arXiv2026

An Interactive Metrics Dashboard for the Keck Observatory Archive

Since 2004, the Keck Observatory Archive (KOA) has operated as a NASA-funded collaboration between the NASA Exoplanet Science Institute ( NExScI) and the W.M. Keck Observatory. It ingests and serves all data acquired by the twin 10-meter Keck telescopes on Mauna Kea, Hawaii. In the past three years, KOA has begun a modernization program to replace the architecture and systems used since the archive's creation with a new modern Python-based infrastructure. This infrastructure will position KOA to respond to the rapid growth of new and complex data sets that will be acquired by new instruments now in development, and enable follow-up to identify the deluge of alerts of transient sources expected by new survey telescopes such as the Vera C. Rubin Observatory. Since 2022, KOA has ingested new data in near-real time, generally within one minute of creation, and has made them immediately accessible to observers through a dedicated web interface. The archive is now deploying a new, scalable, Python-based, VO-compliant query infrastructure built with the Plotly-Dash framework and R-tree indices to speed-up queries by a factor of 20. The project described here exploits the new query infrastructure to develop a dashboard that will return live metrics on the performance and growth of the archive. These metrics assess the current health of the archive and guide planning future hardware and software upgrades. This single dashboard will enable, for example, monitoring of real-time ingestion, as well as studying the long-term growth of the archive. Current methods of gathering metrics that have been in place since the archive opened will not support the archive as it continues to scale. These methods suffer from high latency, are not optimized for on-demand metrics, are scattered among various tools, and are cumbersome to use.

astro-ph.IM↗

arXiv2024

New Python-based Architecture for the Keck Observatory Archive

We describe the development of the Keck Observatory Archive (KOA) Data Discovery Service, a web-based dashboard that returns metadata for wide-area queries of the entire archive in seconds. Currently in beta, this dashboard will support exploration, visualization, and data access across multiple instruments. This effort is underpinned by an open-source, VO-compliant query infrastructure and will offer services that can be hosted on web pages or in Jupyter notebooks. The effort also informs the design of a new, modern landing page that meets the expectations of accessibility and ease of use. The new query infrastructure is based on nexsciTAP, a component-based, DBMS-agnostic Python implementation of the IVOA Table Access Protocol, developed at NExScI and integrated into the NASA Exoplanet Archive and the NEID Archive, and into the PyKOA Python client. This infrastructure incorporates R-tree spatial indexing, built as memory-mapped files as part of Montage, a software toolkit used to create composite astronomical images. Although R-trees are used most often in geospatial analysis, here they enable searches of the entire KOA archive, an eclectic collection of 100 million records of imaging and spectroscopic data, in 2 seconds, and they speed up spatial searches by x20. The front end is built on the open-source Plotly-Dash framework, which allows users to build an interactive user interface based on a single Python file.

astro-ph.IM↗

arXiv2024

Montage and Radio Astronomy

The Montage Image Mosaic Engine, first released in 2002, has found applicability across the electromagnetic spectrum to support data processing and visualization. This broad applicability has come about through its design as an Open Source ANSI-C toolkit (and Python binary extensions), with independent components to perform each step in the creation of a mosaic and with support for all WCS extensions. This design enables easy integration into custom environments, workflows and pipelines, and is the principal reason for its long lifetime. Here we emphasize the growing use of Montage in radio astronomy (37 peer-reviewed papers since 2020), and will focus on three high-profile applications: (1) Analysis of observations made with SKA precursor experiments, such as MeerKAT and the Murchison Wide-field Array, (2) Faraday tomography of LOFAR Two-Metre Sky Survey data (LoTSS-DR2), which explores the structure of the local interstellar medium, and (3) Identification of fast radio bursts.

astro-ph.IM↗

arXiv2024

The Benefits of the Virtual Observatory to Underserved Communities

The Virtual Observatory (VO) is a global ecosystem of interoperating services that connect worldwide data archives. The VO is implemented in all major astronomy archives through common interfaces developed by the 22 members of the International Virtual Observatory Alliance (IVOA). It was founded in 2002, and the newest members, the SKA Observatory and the Kazakhstan Virtual Observatory, joined in 2022. The VO offers access to data on FAIR principles and from its inception has supported Open Science. The VO acts as a democratizing influence in astronomy: it provides equal access to worldwide public data sets to underserved communities as well as to large data centers, and it enables international participation in scientific research and education. Thus, astronomers from many different communities are positioned to participate in the big science questions emerging in astronomy in the 2020s, such as interpreting transient sources that will be measured in forthcoming missions such as Rubin. In addition, the IVOA has signed an MoU with the IAU Office of Astronomy for Development (OAD). Under this MoU, IVOA members participated in "Astronomy from Archival Data," which involved educational activities for undergraduate and post-graduate students organized by Dr. Priya Hasan. The IVOA plans to participate in future such educational events. The presentation describes how new communities may participate in Virtual Observatory science and educational activities, including practices for developing VO-compliant data centers and archives and education and training for developers and end users.

astro-ph.IM↗

arXiv2024

Observers' Data Access Portal: Realtime Streaming for Astronomical Data

The W. M. Keck Observatory Archive (KOA) has released the Observers Data Access Portal (ODAP), a web-application that delivers astronomical data from the W. M. Keck Observatory to the scheduled program's principal investigator and their collaborators anywhere in the world in near real-time. Data files and their associated metadata are streamed to a user's desktop machine moments after they are written to disk and archived in KOA. The ODAP User Interface is built in React and uses the WebSocket protocol to stream data between KOA and the user. This document describes the design of the tool, challenges encountered, shows how ODAP is integrated into the Keck observing model, and provides an analysis of usage metrics.

astro-ph.IM↗

arXiv2024

Integration of Data Reduction and Near Real-Time Archiving into the Keck Observing Model

The W. M. Keck Observatory is welcoming a new era where data reduction and archiving are tightly integrated into our observing model, under the auspices of the Observatory's Data Services Initiative (DSI) project. While previously the Keck Observatory Archive (KOA) archived minimally processed, raw science data the day after observing, Keck is transitioning to a model in which it archives both raw frames and reduced data in near real-time. These data will be made available to observers and collaborators immediately upon ingestion through a dedicated new interface that will support collaboration and sharing among teams, as well as stream data directly to personal computers without access to WMKO's internal networks. Both the raw and science-ready data products will be made publicly available upon the expiration of data protections.

astro-ph.IM↗

arXiv2024

Processing All-Sky Images At Scale On The Amazon Cloud: A HiPS Example

We report here on a project that has developed a practical approach to processing all-sky image collections on cloud platforms, using as an exemplar application the creation of three-color Hierarchical Progressive Survey (HiPS) maps of the 2MASS data set with the Montage Image Mosaic Engine on Amazon Web Services. We will emphasize issues that must be considered by scientists wishing to use cloud platforms to perform such parallel processing, so providing a guide for scientists wishing to exploit cloud platforms for similar large-scale processing. A HiPS map is based on the HEALPix sky-tiling scheme. Progressive zooming of a HiPS map reveals an image sampled at ever smaller or larger spatial scales that are defined by the HEALPix standard. Briefly, the approach used by Montage involves creating a base mosaic at the lowest required HEALPix level, usually chosen to match as closely as possible the spatial sampling of the input images, then cutting out the HiPS cells in PNG format from this mosaic. The process is repeated at successive HEALPix levels to create a nested collection of FITS files, from which PNG files are created that are shown in HiPS viewers. Stretching FITS files to produce PNGs is based on an image histogram. For composite regions (up and including the whole sky), the histograms for each tile can be combined to create a composite histogram for the region. Using this single histogram for each of the individual FITS files means all the PNGs are on the same brightness scale and displaying them side by side in a HiPS viewer produces a continuous uniform map across the entire sky.

astro-ph.IM↗

arXiv2022

Real-time Data Ingestion at the Keck Observatory Archive (KOA)

Since February of this year, KOA began to prepare, transfer, and ingest data as they were acquired in near-real time; in most cases data are available to observers through KOA within one minute of acquisition. Real-time ingestion will be complete for all active instruments by the end of Summer 2022. The observatory is supporting the development of modern Python data reduction pipelines, which when delivered, will automatically create science-ready data sets at the end of each night for ingestion into the archive. This presentation will describe the infrastructure developed to support real-time data ingestion, itself part of a larger initiative at the Observatory to modernize end-to-end operations. During telescope operations, the software at WMKO is executed automatically when a newly acquired file is recognized through monitoring a keyword-based observatory control system; this system is used at Keck to execute virtually all observatory functions. The monitor uses callbacks built into the control system to begin data preparation of files for transmission to the archive on an individual basis: scheduling scripts or file system related triggers are unnecessary. An HTTP-based system called from the Flask micro-framework enables file transfers between WMKO and NExScI and triggers data ingestion at NExScI. The ingestion system at NEXScI is a compact (4 KLOC), highly fault-tolerant, Python-based system. It uses a shared file system to transfer data from WMKO to NExScI. The ingestion code is instrument agnostic, with instrument parameters read from configuration files. It replaces an unwieldy (50 KLOC) C-based system that had been in use since 2004.

astro-ph.IM↗

arXiv2022

Best Practices for Data Publication in the Astronomical Literature

We present an overview of best practices for publishing data in astronomy and astrophysics journals. These recommendations are intended as a reference for authors to help prepare and publish data in a way that will better represent and support science results, enable better data sharing, improve reproducibility, and enhance the reusability of data. Observance of these guidelines will also help to streamline the extraction, preservation, integration and cross-linking of valuable data from astrophysics literature into major astronomical databases, and consequently facilitate new modes of science discovery that will better exploit the vast quantities of panchromatic and multi-dimensional data associated with the literature. We encourage authors, journal editors, referees, and publishers to implement the best practices reviewed here, as well as related recommendations from international astronomical organizations such as the International Astronomical Union (IAU) for publication of nomenclature, data, and metadata. A convenient Checklist of Recommendations for Publishing Data in the Literature is included for authors to consult before the submission of the final version of their journal articles and associated data files. We recommend that publishers of journals in astronomy and astrophysics incorporate a link to this document in their Instructions to Authors.

astro-ph.IM↗

arXiv2021

Astronomical Image Processing at Scale With Pegasus and Montage

Image processing at scale is a powerful tool for creating new data sets and integrating them with existing data sets and performing analysis and quality assurance investigations. Workflow managers offer advantages in this type of processing, which involves multiple data access and processing steps. Generally, they enable automation of the workflow by locating data and resources, recovery from failures, and monitoring of performance. In this focus demo we demonstrate how the Pegasus Workflow Manager Python API manages image processing to create mosaics with the Montage Image Mosaic engine. Since 2001, Pegasus has been developed and maintained at USC/ISI. Montage was in fact one of the first applications used to design Pegasus and optimize its performance. Pegasus has since found application in many areas of science. LIGO exploited it in making discoveries of black holes. The Vera C. Rubin Observatory used it to compare the cost and performance of processing images on cloud platforms. While these are examples of projects at large scale, small team investigations on local clusters of machines can benefit from Pegasus as well.

astro-ph.IM↗

arXiv2020

The International Virtual Observatory Alliance (IVOA) in 2020

The International Virtual Observatory Alliance (IVOA) develops the technical standards needed for seamless discovery of and access to astronomy data worldwide, according to the Findable, Accessible, Interoperable and Reusable (FAIR) principles, with the goal of realizing the Virtual Observatory (VO). There are 21 member organizations. The Netherlands VO applied for membership in 2020. Astronomical communities from other nations have shown interest in joining the IVOA. This paper describes the activities of the IVOA in 2020 and summarizes the May and November 2020 virtual "interoperability meetings." The May meeting was the first to be held online and the first to have over 200 registrants.

astro-ph.IM↗

arXiv2020

Cost Management on Commercial Cloud Platforms

Commercial cloud platforms are a powerful technology for astronomical research. Despite the benefits of cloud computing -- such as on-demand scalability and reduction of systems management overhead -- confusion over how to manage costs remains, for many, one of the biggest barriers to entry. This confusion is exacerbated by the rapid growth in services offered by commercial providers, by the growth in the number of these providers, and by storage, compute, and I/O metered at separate rates -- all of which can change without notice. As a rule, processing is very cheap, storage is more expensive, and downloading is very expensive. Thus, an application that produces large image data sets for download will be far more expensive than an application that performs extensive processing on a small data set. This Birds of a Feather (BoF) session aimed to quantify the above statement by presenting case studies of the costing of astronomy applications on commercial clouds that covered a range of processing scenarios; these presentations were the basis for discussion by the attendees.

astro-ph.IM↗

arXiv2020

Creating High Quality All-Sky Visualizations of Astronomy Image Data Sets: HiPS and Montage

We describe a case study to use the Montage image mosaic engine to create maps of the ALLWISE image data set in the Hierarchical Progressive Survey (HiPS) sky-tesselation scheme. Our approach demonstrates that Montage reveals the science content of infrared images in greater detail than has hitherto been possible in HiPS maps. The approach exploits two unique (to our knowledge) characteristics of the Montage image mosaic engine: background modeling to rectify the time variable image backgrounds to common levels; and an adaptive image stretch to present images for visualization. The creation of the maps is supported by the development of four new tools that when fully tested will become part of the Montage distribution. The compute intensive part of the processing lies in the reprojection of the images, and we show how we optimized the processing for efficient creation of mosaics that are used in turn to create maps in the HiPS tiling scheme. We plan to apply our methodology to infrared image data sets such a those delivered by Spitzer, 2MASS, IRAS and Planck.

astro-ph.IM↗

arXiv2019

The International Virtual Observatory Alliance in 2019

The International Virtual Observatory Alliance (IVOA) held its bi-annual Interoperability Meetings in May 2019, and in October 2019 following the ADASS XXIX conference. We provide a brief report on the status of the IVOA and the activities of the Interoperability Meetings.

astro-ph.IM↗

arXiv2019

Image Processing in Python With Montage

The Montage image mosaic engine has found wide applicability in astronomy research, integration into processing environments, and is an examplar application for the development of advanced cyber-infrastructure. It is written in C to provide performance and portability. Linking C/C++ libraries to the Python kernel at run time as binary extensions allows them to run under Python at compiled speeds and enables users to take advantage of all the functionality in Python. We have built Python binary extensions of the 59 ANSI-C modules that make up version 5 of the Montage toolkit. This has involved a turning the code into a C library, with driver code fully separated to reproduce the calling sequence of the command-line tools; and then adding Python and C linkage code with the Cython library, which acts as a bridge between general C libraries and the Python interface. We will demonstrate how to use these Python binary extensions to perform image processing, including reprojecting and resampling images, rectifying background emission to a common level, creation of image mosaics that preserve the calibration and astrometric fidelity of the input images, creating visualizations with an adaptive stretch algorithm, processing HEALPix images, and analyzing and managing image metadata.

astro-ph.IM↗

arXiv2019

Deep Learning for Multi-Messenger Astrophysics: A Gateway for Discovery in the Big Data Era

This report provides an overview of recent work that harnesses the Big Data Revolution and Large Scale Computing to address grand computational challenges in Multi-Messenger Astrophysics, with a particular emphasis on real-time discovery campaigns. Acknowledging the transdisciplinary nature of Multi-Messenger Astrophysics, this document has been prepared by members of the physics, astronomy, computer science, data science, software and cyberinfrastructure communities who attended the NSF-, DOE- and NVIDIA-funded "Deep Learning for Multi-Messenger Astrophysics: Real-time Discovery at Scale" workshop, hosted at the National Center for Supercomputing Applications, October 17-19, 2018. Highlights of this report include unanimous agreement that it is critical to accelerate the development and deployment of novel, signal-processing algorithms that use the synergy between artificial intelligence (AI) and high performance computing to maximize the potential for scientific discovery with Multi-Messenger Astrophysics. We discuss key aspects to realize this endeavor, namely (i) the design and exploitation of scalable and computationally efficient AI algorithms for Multi-Messenger Astrophysics; (ii) cyberinfrastructure requirements to numerically simulate astrophysical sources, and to process and interpret Multi-Messenger Astrophysics data; (iii) management of gravitational wave detections and triggers to enable electromagnetic and astro-particle follow-ups; (iv) a vision to harness future developments of machine and deep learning and cyberinfrastructure resources to cope with the scale of discovery in the Big Data Era; (v) and the need to build a community that brings domain experts together with data scientists on equal footing to maximize and accelerate discovery in the nascent field of Multi-Messenger Astrophysics.

astro-ph.IM↗

arXiv2018

The KELT Follow-Up Network and Transit False Positive Catalog: Pre-vetted False Positives for TESS

The Kilodegree Extremely Little Telescope (KELT) project has been conducting a photometric survey for transiting planets orbiting bright stars for over ten years. The KELT images have a pixel scale of ~23"/pixel---very similar to that of NASA's Transiting Exoplanet Survey Satellite (TESS)---as well as a large point spread function, and the KELT reduction pipeline uses a weighted photometric aperture with radius 3'. At this angular scale, multiple stars are typically blended in the photometric apertures. In order to identify false positives and confirm transiting exoplanets, we have assembled a follow-up network (KELT-FUN) to conduct imaging with higher spatial resolution, cadence, and photometric precision than the KELT telescopes, as well as spectroscopic observations of the candidate host stars. The KELT-FUN team has followed-up over 1,600 planet candidates since 2011, resulting in more than 20 planet discoveries. Excluding ~450 false alarms of non-astrophysical origin (i.e., instrumental noise or systematics), we present an all-sky catalog of the 1,128 bright stars (6<V<10) that show transit-like features in the KELT light curves, but which were subsequently determined to be astrophysical false positives (FPs) after photometric and/or spectroscopic follow-up observations. The KELT-FUN team continues to pursue KELT and other planet candidates and will eventually follow up certain classes of TESS candidates. The KELT FP catalog will help minimize the duplication of follow-up observations by current and future transit surveys such as TESS.

astro-ph.EP↗