arXiv subjects

Michael A. C. Johnson

Publications and source records attributed to Michael A. C. Johnson.

6 recordsLinked to original sources

arXiv2024

Evaluation of Provenance Serialisations for Astronomical Provenance

Provenance data from astronomical pipelines are instrumental in establishing trust and reproducibility in the data processing and products. In addition, astronomers can query their provenance to answer questions routed in areas such as anomaly detection, recommendation, and prediction. The next generation of astronomical survey telescopes such as the Vera Rubin Observatory or Square Kilometre Array, are capable of producing peta to exabyte scale data, thereby amplifying the importance of even small improvements to the efficiency of provenance storage or querying. In order to determine how astronomers should store and query their provenance data, this paper reports on a comparison between the turtle and JSON provenance serialisations. The triple store Apache Jena Fuseki and the graph database system Neo4j were selected as representative database management systems (DBMS) for turtle and JSON, respectively. Simulated provenance data was uploaded to and queried over each DBMS and the metrics measured for comparison were the accuracy and timing of the queries as well as the data upload times. It was found that both serialisations are competent for this purpose, and both have similar query accuracy. The turtle provenance was found to be more efficient at storing and uploading the data. Regarding queries, for small datasets ($<$5MB) and simple information retrieval queries, the turtle serialisation was also found to be more efficient. However, queries for JSON serialised provenance were found to be more efficient for more complex queries which involved matching patterns across the DBMS, this effect scaled with the size of the queried provenance.

astro-ph.IM↗

arXiv2024

Pipeline Provenance for Analysis, Evaluation, Trust or Reproducibility

Data volumes and rates of research infrastructures will continue to increase in the upcoming years and impact how we interact with their final data products. Little of the processed data can be directly investigated and most of it will be automatically processed with as little user interaction as possible. Capturing all necessary information of such processing ensures reproducibility of the final results and generates trust in the entire process. We present PRAETOR, a software suite that enables automated generation, modelling, and analysis of provenance information of Python pipelines. Furthermore, the evaluation of the pipeline performance, based upon a user defined quality matrix in the provenance, enables the first step of machine learning processes, where such information can be fed into dedicated optimisation procedures.

astro-ph.IM↗

arXiv2021

Astronomical Pipeline Provenance: A Use Case Evaluation

In this decade astronomy is undergoing a paradigm shift to handle data from next generation observatories such as the Square Kilometre Array (SKA) or the Vera C. Rubin Observatory (LSST). Producing real time data streams of up to 10 TB/s and data products of the order of 600 Pbytes/year, the SKA will be the biggest civil data producing machine of the world that demands novel solutions on how these data volumes can be stored and analysed. Through the use of complex, automated pipelines the provenance of this real time data processing is key to establish confidence within the system, its final data products, and ultimately its scientific results. The intention of this paper is to lay the foundation for making an automated provenance generation tool for astronomical/data-processing pipelines. We therefore present a use case analysis, specific to the astronomical needs which addresses the issues of trust and reproducibility as well as other ulterior use cases which are of interest to astronomers. This analysis is subsequently used as the basis to discuss the requirements, challenges, and opportunities involved in designing both the tool and the associated provenance model.

astro-ph.IM↗

arXiv2019

Gaia DR2 Distances and Peculiar Velocities for Galactic Black Hole Transients

We report on a first census of Galactic black hole X-ray binary (BHXRB) properties with the second data release (DR2) of {\em Gaia}, focusing on dynamically confirmed and strong candidate black hole transients. DR2 provides five-parameter astrometric solutions including position, parallax and proper motion for 11 of a sample of 24 systems. Distance estimates are tested with parallax inversion as well as Bayesian inference. We derive an empirically motivated characteristic scale length of $L$=2.17$\pm$0.12 kpc for this BHXRB population to infer distances based upon an exponentially decreasing space density prior. Geometric DR2 parallaxes provide new, independent distance estimates, but the faintness of this population in quiescence results in relatively large fractional distance uncertainties. Despite this, DR2 estimates generally agree with literature distances. The most discrepant case is BW Cir, for which detailed studies of the donor star have suggested a distant location at >~25 kpc. A large DR2 measured parallax and relatively high proper motion instead prefer significantly smaller distances, suggesting that the source may instead be amongst the nearest of XRBs. However, both distances create problems for interpretation of the source, and follow-up data are required to resolve its true nature. DR2 also provides a first distance estimate to one source, MAXI J1820+070, and novel proper motion estimates for 7 sources. Peculiar velocities relative to Galactic rotation exceed $\sim$ 50 km s$^{-1}$ for the bulk of the sample, with a median system kinetic energy of peculiar motion of $\sim$ 5 $\times$ 10$^{47}$ erg. BW Cir could be a new high-velocity BHXRB if its astrometry is confirmed. A putative anti-correlation between peculiar velocity and black hole mass is found, as expected in mass-dependent BH kick formation channels, but this trend remains weak in the DR2 data.

astro-ph.HE↗

arXiv2019

Prospecting Period Measurements with LSST - Low Mass X-ray Binaries as a Test Case

The Large Synoptic Survey Telescope (LSST) will provide for unbiased sampling of variability properties of objects with $r$ mag $<$ 24. This should allow for those objects whose variations reveal their orbital periods ($P_{orb}$), such as low mass X-ray binaries (LMXBs) and related objects, to be examined in much greater detail and with uniform systematic sampling. However, the baseline LSST observing strategy has temporal sampling that is not optimised for such work in the Galaxy. Here we assess four candidate observing strategies for measurement of $P_{orb}$ in the range 10 minutes to 50 days. We simulate multi-filter quiescent LMXB lightcurves including ellipsoidal modulation and stochastic flaring, and then sample these using LSST's operations simulator (OpSim) over the (mag, $P_{orb}$) parameter space, and over five sightlines sampling a range of possible reddening values. The percentage of simulated parameter space with correctly returned periods ranges from $\sim$23 %, for the current baseline strategy, to $\sim$70 % for the two simulated specialist strategies. Convolving these results with a $P_{orb}$ distribution, a modelled Galactic spatial distribution and reddening maps, we conservatively estimate that the most recent version of the LSST baseline strategy will allow $P_{orb}$ determination for $\sim$18 % of the Milky Way's LMXB population, whereas strategies that do not reduce observations of the Galactic Plane can improve this dramatically to $\sim$32 %. This increase would allow characterisation of the full binary population by breaking degeneracies between suggested $P_{orb}$ distributions in the literature. Our results can be used in the ongoing assessment of the effectiveness of various potential cadencing strategies.

astro-ph.IM↗

arXiv2018

The Plane's The Thing: The Case for Wide-Fast-Deep Coverage of the Galactic Plane and Bulge

We argue that the exclusion of the Galactic Plane and Bulge from the uniform wide-fast-deep (WFD) LSST survey cadence is fundamentally inconsistent with two of the main science drivers of LSST: Mapping the Milky Way and Exploring the Transient Optical Sky. We outline the philosophical basis for this claim and then describe a number of important science goals that can only be addressed by WFD-like coverage of the Plane and Bulge.

astro-ph.GA↗