Africanus II. QuartiCal: calibrating radio interferometer data at scale using Numba and Dask

Abstract

Calibration of radio interferometer data ought to be a solved problem; it has been an integral part of data reduction for some time. However, as larger, more sensitive radio interferometers are conceived and built, the calibration problem grows in both size and difficulty. The increasing size can be attributed to the fact that the data volume scales quadratically with the number of antennas in an array. Additionally, new instruments may have up to two orders of magnitude more channels than their predecessors. Simultaneously, increasing sensitivity is making calibration more challenging: low-level RFI and calibration artefacts (in the resulting images) which would previously have been subsumed by the noise may now limit dynamic range and, ultimately, the derived science. It is against this backdrop that we introduce QuartiCal: a new Python package implementing radio interferometric calibration routines. QuartiCal improves upon its predecessor, CubiCal, in terms of both flexibility and performance. Whilst the same mathematical framework - complex optimization using Wirtinger derivatives - is in use, the approach has been refined to support arbitrary length chains of parameterized gain terms. QuartiCal utilizes Dask, a library for parallel computing in Python, to express calibration as an embarrassingly parallel task graph. These task graphs can (with some constraints) be mapped onto a number of different hardware configurations, allowing QuartiCal to scale from running locally on consumer hardware to a distributed, cloud-based cluster. QuartiCal's qualitative behaviour is demonstrated using MeerKAT observations of PSR J2009-2026. These qualitative results are followed by an analysis of QuartiCal's performance in terms of wall time and memory footprint for a number of calibration scenarios and hardware configurations.