arXiv subjects

Andrew Connolly

Publications and source records attributed to Andrew Connolly.

At least 19 recordsLinked to original sources

arXiv2026

LightCurveLynx: Forward Modeling of Time-Domain Surveys with Application to ZTF SN Ia DR2

We present LightCurveLynx, a flexible and extensible software framework for end-to-end forward modeling time-domain light curves. Given the growing need for realistic simulations in the time-domain astronomy community, LightCurveLynx is designed to support a wide range of applications, including the development and validation of analysis pipelines, the optimization of survey strategies, and simulation-based inference studies. Realistic simulations can be generated from real survey metadata, forecasted survey plans, or user-defined mock survey strategies. We demonstrate the functionality of LightCurveLynx by generating a realistic simulation of Type Ia supernovae that is representative of the ZTF SN Ia Data Release 2 dataset and perform extensive comparisons between the simulated and observed samples to validate the software. The simulation shows excellent agreement with the data in parameter distributions (with the Kullback-Leibler divergence values around 0.01-0.02) and in noise properties. The Hubble diagram generated from the simulation also indicates that the sample is complete up to redshift 0.06, which is consistent with previous studies. Our results confirm that LightCurveLynx is robust, accurate, and ready for community use and contribution.

astro-ph.IM↗

arXiv2025

The Future of Artificial Intelligence and the Mathematical and Physical Sciences (AI+MPS)

This community paper developed out of the NSF Workshop on the Future of Artificial Intelligence (AI) and the Mathematical and Physics Sciences (MPS), which was held in March 2025 with the goal of understanding how the MPS domains (Astronomy, Chemistry, Materials Research, Mathematical Sciences, and Physics) can best capitalize on, and contribute to, the future of AI. We present here a summary and snapshot of the MPS community's perspective, as of Spring/Summer 2025, in a rapidly developing field. The link between AI and MPS is becoming increasingly inextricable; now is a crucial moment to strengthen the link between AI and Science by pursuing a strategy that proactively and thoughtfully leverages the potential of AI for scientific discovery and optimizes opportunities to impact the development of AI by applying concepts from fundamental science. To achieve this, we propose activities and strategic priorities that: (1) enable AI+MPS research in both directions; (2) build up an interdisciplinary community of AI+MPS researchers; and (3) foster education and workforce development in AI for MPS researchers and students. We conclude with a summary of suggested priorities for funding agencies, educational institutions, and individual researchers to help position the MPS community to be a leader in, and take full advantage of, the transformative potential of AI+MPS.

cs.AI↗

arXiv2025

NSF-DOE Vera C. Rubin Observatory Observations of Interstellar Comet 3I/ATLAS (C/2025 N1)

We report on the observation and measurement of astrometry, photometry, morphology, and activityof the interstellar object 3I/ATLAS, also designated C/2025 N1 (ATLAS) with the NSF-DOE Vera C. Rubin Observatory. Comet 3I/ATLAS, the third known interstellar object, was discovered on UT 2025 July 1. Rubin Observatory had coincidentally collected images of the object's region of the sky during routine commissioning. Facilitated by Rubin's high resolution and large aperture, we successfully recovered object detections from Rubin observations spanning UT 2025 June 21 (10 days before discovery, when 3I/ATLAS was 4.5 au from the Sun) through the date of discovery, and we acquired additional images through UT 2025 July 20 as part of commissioning. We measure on-sky locations of 3I/ATLAS in Rubin ugrizy bands, with a typical precision of about 70 mas, and briefly describe the reason this is coarser than our measured static source astrometric precision of about 3 mas in Rubin images. We measure grizy magnitudes of 3I/ATLAS photometry at about 0.01 mag precision, detecting no short-term photometric variability above 0.01 mag. We derive an estimated near-nucleus dust-to-nucleus scattering cross-section ratio of eta >= 13 on UT 2025 July 2 based on Rubin photometry and an upper limit nucleus size computed from Hubble Space Telescope observations. We find Rubin colors of g - r = (0.657 +/- 0.013) mag, r - i = (0.235 +/- 0.018) mag, i - z = (0.147 +/- 0.042) mag, z - y = (0.047 +/- 0.052) mag. These data represent the earliest observations of this object by a large (>=8-meter class) telescope and illustrate the type of measurements (and discoveries) Rubin's Legacy Survey of Space and Time (LSST) will begin to provide after it begins in early 2026.

astro-ph.EP↗

arXiv2025

Variability-finding in Rubin Data Preview 1 with LSDB

The Vera C. Rubin Observatory recently released Data Preview 1 (DP1) in advance of the upcoming Legacy Survey of Space and Time (LSST), which will enable boundless discoveries in time-domain astronomy over the next ten years. DP1 provides an ideal sandbox for validating innovative data analysis approaches for the LSST mission, whose scale challenges established software infrastructure paradigms. This note presents a pair of such pipelines for variability-finding using powerful software infrastructure suited to LSST data, namely the HATS (Hierarchical Adaptive Tiling Scheme) format and the LSDB framework, developed by the LSST Interdisciplinary Network for Collaboration and Computing (LINCC) Frameworks team. This article presents a pair of variability-finding pipelines built on LSDB, the HATS catalog of DP1 data, and preliminary results of detected variable objects, two of which are novel discoveries.

astro-ph.IM↗

arXiv2025

Using LSDB to enable large-scale catalog distribution, cross-matching, and analytics

The Vera C. Rubin Observatory will generate an unprecedented volume of data, including approximately 60 petabytes of raw data and around 30 trillion observed sources, posing a significant challenge for large-scale and end-user scientific analysis. As part of the LINCC Frameworks Project we are addressing these challenges with the development of the HATS (Hierarchical Adaptive Tiling Scheme) format and analysis package LSDB. HATS partitions data adaptively using a hierarchical tiling system to balance the file sizes, enabling efficient parallel analysis. Recent updates include improved metadata consistency, support for incremental updates, and enhanced compatibility with evolving datasets. LSDB complements HATS by providing a scalable, user-friendly interface for large catalog analysis, integrating spatial queries, crossmatching, and time-series tools while utilizing Dask for parallelization. We have successfully demonstrated the use of these tools with datasets such as ZTF and Pan-STARRS data releases on both cluster and cloud environments. We are deeply involved in several ongoing collaborations to ensure alignment with community needs, with future plans for IVOA standardization and support for upcoming Rubin, Euclid and Roman data. We provide our code and materials at lsdb.io.

astro-ph.IM↗

arXiv2024

Maximizing the scientific return of Roman and Rubin with a joint wide-sky observing strategy

This work presents the case for a single-band LSST-matched depth Roman Community Survey over the footprint of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) Wide-Fast-Deep to enhance the key science programs of both missions. We propose to observe the ~18K sq deg LSST Wide-Fast-Deep footprint in the F146 filter to mAB~25; this will take approximately 5 months of Roman observing time. The combination of the multiwavelength nature of LSST and angular resolution of Roman would lead to enhanced scientific returns for both the Roman and LSST surveys. Galaxy deblending and crowded field photometry will be significantly improved. The extension of Rubin LSST six-band optical photometry to IR wavelengths would improve photometric redshift (photo-z) estimation, leading to improved cosmological parameter estimation, penetrate interstellar dust in the Galactic plane, improve differential chromatic refraction derived Spectral Energy Distributions, maximize galaxy-star separation and minimize crowding confusion through improved angular resolution. Conversely, the LSST survey will provide a time-domain extension of the Roman survey on the shared footprint and 6-band optical photometry with sensitivity extending all the way to ultraviolet wavelengths.

astro-ph.IM↗

arXiv2022

The Astronomy Commons Platform: A Deployable Cloud-Based Analysis Platform for Astronomy

We present a scalable, cloud-based science platform solution designed to enable next-to-the-data analyses of terabyte-scale astronomical tabular datasets. The presented platform is built on Amazon Web Services (over Kubernetes and S3 abstraction layers), utilizes Apache Spark and the Astronomy eXtensions for Spark for parallel data analysis and manipulation, and provides the familiar JupyterHub web-accessible front-end for user access. We outline the architecture of the analysis platform, provide implementation details, rationale for (and against) technology choices, verify scalability through strong and weak scaling tests, and demonstrate usability through an example science analysis of data from the Zwicky Transient Facility's 1Bn+ light-curve catalog. Furthermore, we show how this system enables an end-user to iteratively build analyses (in Python) that transparently scale processing with no need for end-user interaction. The system is designed to be deployable by astronomers with moderate cloud engineering knowledge, or (ideally) IT groups. Over the past three years, it has been utilized to build science platforms for the DiRAC Institute, the ZTF partnership, the LSST Solar System Science Collaboration, the LSST Interdisciplinary Network for Collaboration and Computing, as well as for numerous short-term events (with over 100 simultaneous users). A live demo instance, the deployment scripts, source code, and cost calculators are accessible at http://hub.astronomycommons.org/.

astro-ph.IM↗

arXiv2020

A Gateway to Astronomical Image Processing: Vera C. RubinObservatory LSST Science Pipelines on AWS

The Legacy Survey of Space and Time, operated by the Vera C. Rubin Observatory, is a 10-year astronomical survey due to start operations in 2022 that will image half the sky every three nights. LSST will produce ~20TB of raw data per night which will be calibrated and analyzed in almost real time. Given the volume of LSST data, the traditional subset-download-process paradigm of data reprocessing faces significant challenges. We describe here, the first steps towards a gateway for astronomical science that would enable astronomers to analyze images and catalogs at scale. In this first step we focus on executing the Rubin LSST Science Pipelines, a collection of image and catalog processing algorithms, on Amazon Web Services (AWS). We describe our initial impressions on the performance, scalability and cost of deploying such a system in the cloud.

astro-ph.IM↗

arXiv2020

Sampling for Deep Learning Model Diagnosis (Technical Report)

Deep learning (DL) models have achieved paradigm-changing performance in many fields with high dimensional data, such as images, audio, and text. However, the black-box nature of deep neural networks is a barrier not just to adoption in applications such as medical diagnosis, where interpretability is essential, but also impedes diagnosis of under performing models. The task of diagnosing or explaining DL models requires the computation of additional artifacts, such as activation values and gradients. These artifacts are large in volume, and their computation, storage, and querying raise significant data management challenges. In this paper, we articulate DL diagnosis as a data management problem, and we propose a general, yet representative, set of queries to evaluate systems that strive to support this new workload. We further develop a novel data sampling technique that produce approximate but accurate results for these model debugging queries. Our sampling technique utilizes the lower dimension representation learned by the DL model and focuses on model decision boundaries for the data in this lower dimensional space. We evaluate our techniques on one standard computer vision and one scientific data set and demonstrate that our sampling technique outperforms a variety of state-of-the-art alternatives in terms of query accuracy.

cs.LG↗

arXiv2019

Future Uses of the LSST Facility: Input from the LSST Project Science Team

In this white paper, we discuss future uses of the LSST facility after the planned 10-year survey is complete. We expect the LSST survey to profoundly affect the scientific landscape over the next ten years, and it is likely that unexpected discoveries may drive its future scientific program. We discuss various operations and instrument options that could be considered for an extended LSST mission beyond ten years.

astro-ph.IM↗

arXiv2019

Astro2020 APC White Paper: Astronomy should be in the clouds

Commodity cloud computing, as provided by commercial vendors such as Amazon, Google, and Microsoft, has revolutionized computing in many sectors. With the advent of a new class of big data, public access astronomical facility such as LSST, DKIST, and WFIRST, there exists a real opportunity to combine these missions with cloud computing platforms and fundamentally change the way astronomical data is collected, processed, archived, and curated. Making these changes in a cross-mission, coordinated way can provide unprecedented economies of scale in personnel, data collection and management, archiving, algorithm and software development and, most importantly, science.

astro-ph.IM↗

arXiv2019

Astro2020 APC White Paper: Elevating the Role of Software as a Product of the Research Enterprise

Software is a critical part of modern research, and yet there are insufficient mechanisms in the scholarly ecosystem to acknowledge, cite, and measure the impact of research software. The majority of academic fields rely on a one-dimensional credit model whereby academic articles (and their associated citations) are the dominant factor in the success of a researcher's career. In the petabyte era of astronomical science, citing software and measuring its impact enables academia to retain and reward researchers that make significant software contributions. These highly skilled researchers must be retained to maximize the scientific return from petabyte-scale datasets. Evolving beyond the one-dimensional credit model requires overcoming several key challenges, including the current scholarly ecosystem and scientific culture issues. This white paper will present these challenges and suggest practical solutions for elevating the role of software as a product of the research enterprise.

astro-ph.IM↗

arXiv2019

Subband Image Reconstruction using Differential Chromatic Refraction

Refraction by the atmosphere causes the positions of sources to depend on the airmass through which an observation was taken. This shift is dependent on the underlying spectral energy of the source and the filter or bandpass through which it is observed. Wavelength-dependent refraction within a single passband is often referred to as differential chromatic refraction (DCR). With a new generation of astronomical surveys undertaking repeated observations of the same part of the sky over a range of different airmasses and parallactic angles, DCR should be a detectable and measurable astrometric signal. In this paper we introduce a novel procedure that takes this astrometric signal and uses it to infer the underlying spectral energy distribution of a source; we solve for multiple latent images at specific wavelengths via a generalized deconvolution procedure built on robust statistics. We demonstrate the utility of such an approach for estimating a partially deconvolved image, at higher spectral resolution than the input images, for surveys such as the Large Synoptic Survey Telescope (LSST).

astro-ph.IM↗

arXiv2019

The Zwicky Transient Facility: System Overview, Performance, and First Results

The Zwicky Transient Facility (ZTF) is a new optical time-domain survey that uses the Palomar 48-inch Schmidt telescope. A custom-built wide-field camera provides a 47 deg$^2$ field of view and 8 second readout time, yielding more than an order of magnitude improvement in survey speed relative to its predecessor survey, the Palomar Transient Factory (PTF). We describe the design and implementation of the camera and observing system. The ZTF data system at the Infrared Processing and Analysis Center provides near-real-time reduction to identify moving and varying objects. We outline the analysis pipelines, data products, and associated archive. Finally, we present on-sky performance analysis and first scientific results from commissioning and the early survey. ZTF's public alert stream will serve as a useful precursor for that of the Large Synoptic Survey Telescope.

astro-ph.IM↗

arXiv2019

Machine Learning for the Zwicky Transient Facility

The Zwicky Transient Facility is a large optical survey in multiple filters producing hundreds of thousands of transient alerts per night. We describe here various machine learning (ML) implementations and plans to make the maximal use of the large data set by taking advantage of the temporal nature of the data, and further combining it with other data sets. We start with the initial steps of separating bogus candidates from real ones, separating stars and galaxies, and go on to the classification of real objects into various classes. Besides the usual methods (e.g., based on features extracted from light curves) we also describe early plans for alternate methods including the use of domain adaptation, and deep learning. In a similar fashion we describe efforts to detect fast moving asteroids. We also describe the use of the Zooniverse platform for helping with classifications through the creation of training samples, and active learning. Finally we mention the synergistic aspects of ZTF and LSST from the ML perspective.

astro-ph.IM↗

arXiv2019

The Zwicky Transient Facility: Science Objectives

The Zwicky Transient Facility (ZTF), a public-private enterprise, is a new time domain survey employing a dedicated camera on the Palomar 48-inch Schmidt telescope with a 47 deg$^2$ field of view and 8 second readout time. It is well positioned in the development of time domain astronomy, offering operations at 10% of the scale and style of the Large Synoptic Survey Telescope (LSST) with a single 1-m class survey telescope. The public surveys will cover the observable northern sky every three nights in g and r filters and the visible Galactic plane every night in g and r. Alerts generated by these surveys are sent in real time to brokers. A consortium of universities which provided funding ("partnership") are undertaking several boutique surveys. The combination of these surveys producing one million alerts per night allows for exploration of transient and variable astrophysical phenomena brighter than r $\sim$ 20.5 on timescales of minutes to years. We describe the primary science objectives driving ZTF including the physics of supernovae and relativistic explosions, multi-messenger astrophysics, supernova cosmology, active galactic nuclei and tidal disruption events, stellar variability, and Solar System objects.

astro-ph.IM↗

arXiv2018

Multi-Messenger Astrophysics: Harnessing the Data Revolution

The past year has witnessed discovery of the first identified counterparts to a gravitational wave transient (GW 170817A) and a very high-energy neutrino (IceCube-170922A). These source identifications, and ensuing detailed studies, have realized longstanding dreams of astronomers and physicists to routinely carry out observations of cosmic sources by other than electromagnetic means, and inaugurated the era of "multi-messenger" astronomy. While this new era promises extraordinary physical insights into the universe, it brings with it new challenges, including: highly heterogeneous, high-volume, high-velocity datasets; globe-spanning cross-disciplinary teams of researchers, regularly brought together into transient collaborations; an extraordinary breadth and depth of domain-specific knowledge and computing resources required to anticipate, model, and interpret observations; and the routine need for adaptive, distributed, rapid-response observing campaigns to fully exploit the scientific potential of each source. We argue, therefore, that the time is ripe for the community to conceive and propose an Institute for Multi-Messenger Astrophysics that would coordinate its resources in a sustained and strategic fashion to efficiently address these challenges, while simultaneously serving as a center for education and key supporting activities. In this fashion, we can prepare now to realize the bright future that we see, beyond, through these newly opened windows onto the universe.

astro-ph.IM↗

arXiv2016

Comparative Evaluation of Big-Data Systems on Scientific Image Analytics Workloads

Scientific discoveries are increasingly driven by analyzing large volumes of image data. Many new libraries and specialized database management systems (DBMSs) have emerged to support such tasks. It is unclear, however, how well these systems support real-world image analysis use cases, and how performant are the image analytics tasks implemented on top of such systems. In this paper, we present the first comprehensive evaluation of large-scale image analysis systems using two real-world scientific image data processing use cases. We evaluate five representative systems (SciDB, Myria, Spark, Dask, and TensorFlow) and find that each of them has shortcomings that complicate implementation or hurt performance. Such shortcomings lead to new research opportunities in making large-scale image analysis both efficient and easy to use.

cs.DB↗