arXiv subjects
Publications and source records attributed to Tim Jenness.
The General Single-Dish Data format (GSDD) was developed in the mid-1980s as a data model to support centimeter, millimeter and submillimeter instrumentation at NRAO, JCMT, the University of Arizona and IRAM. We provide an overview of the GSDD requirements and associated data model, discuss the implementation of the resultant file formats, describe its usage in the observatories and provide a retrospective on the format.
The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard is, however, showing its age. Developed in the late 1970s the FITS authors made a number of implementation choices for the format that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not appreciate the challenges which we would be facing today in astronomical computing. Difficulties we now face include, but are not limited to, having to address the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets and the need to capture significantly more complex metadata and data relationships. There are members of the community today who find some (or all) of these limitations unworkable, and have decided to move ahead with storing data in other formats. This reaction should be taken as a wakeup call to the FITS community to make changes in the FITS standard, or to see its usage fall. In this paper we detail some selected important problems which exist within the FITS standard today. It is not our intention to prescribe specific remedies to these issues; rather, we hope to call attention of the FITS and greater astronomical computing communities to these issues in the hopes that it will spur action to address them.
The Starlink Hierarchical Data System has been a very successful niche astronomy file format and library for over 30 years. Development of the library was frozen ten years ago when funding for Starlink was stopped and almost no-one remains who understands the implementation details. To ensure the long-term sustainability of the Starlink application software and to make the extensible N-Dimensional Data Format accessible to a broader range of users, we propose to re-implement the HDS library application interface as a layer on top of the Hierarchical Data Format version 5. We present an overview of the new implementation of version 5 of the HDS file format and describe differences between the expectations of the HDS and HDF5 library interfaces. We finish by comparing the old and new HDS implementations by looking at a comparison of file sizes and by comparing performance benchmarks.
The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships. There are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them.
We present, as a case study, a description of the partnership between an observatory (JCMT) and a data centre (CADC) that led to the development of the JCMT Science Archive (JSA). The JSA is a successful example of a service designed to use Virtual Observatory (VO) technologies from the start. We describe the motivation, process and lessons learned from this approach.
The future of astronomy is inextricably entwined with the care and feeding of astronomical data products. Community standards such as FITS and NDF have been instrumental in the success of numerous astronomy projects. Their very success challenges us to entertain pragmatic strategies to adapt and evolve the standards to meet the aggressive data-handling requirements of facilities now being designed and built. We discuss characteristics that have made standards successful in the past, as well as desirable features for the future, and an open discussion follows.
CCAT will be a large sub-millimeter telescope to be built near the ALMA site in northern Chile. The telescope must support a varied set of instrumentation including large format KID cameras, a large heterodyne array and a KID-based direct detection multi-object spectrometer. We are developing a new raw data model based on HDF5 that can cope with the expected data rates of order Gbit/s and is flexible enough to hold raw data from all planned instruments.
CCAT is a large submillimetre telescope to be built near the ALMA site in northern Chile. A large-format KID camera, with up to 48,000 detectors at a single waveband sampled at about 1 kHz, will have a data rate about 50 times larger than SCUBA-2, the largest existing submillimetre camera. Creating a map from this volume of data will be a challenge, both in terms of memory and processing time required. We investigate how to extend SMURF, the iterative map-maker used for reducing SCUBA-2 observations, to a distributed-node parallel system, and estimate how the processing time scales with the number of nodes in the system.
ORAC-DR is a general purpose data reduction pipeline system designed to be instrument and observatory agnostic. The pipeline works with instruments as varied as infrared integral field units, imaging arrays and spectrographs, and sub-millimeter heterodyne arrays & continuum cameras. This paper describes the architecture of the pipeline system and the implementation of the core infrastructure. We finish by discussing the lessons learned since the initial deployment of the pipeline system in the late 1990s.
The extensible N-Dimensional Data Format (NDF) was designed and developed in the late 1980s to provide a data model suitable for use in a variety of astronomy data processing applications supported by the UK Starlink Project. Starlink applications were used extensively, primarily in the UK astronomical community, and form the basis of a number of advanced data reduction pipelines today. This paper provides an overview of the historical drivers for the development of NDF and the lessons learned from using a defined hierarchical data model for many years in data reduction software, data pipelines and in data acquisition systems.
The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES) is a joint project of the National Optical Astronomy Observatory and the Department of Computer Science at the University of Arizona. The goal is to build the software infrastructure necessary to process and filter alerts produced by time-domain surveys, with the ultimate source of such alerts being the Large Synoptic Survey Telescope (LSST). The ANTARES broker will add value to alerts by annotating them with information from external sources such as previous surveys from across the electromagnetic spectrum. In addition, the temporal history of annotated alerts will provide further annotation for analysis. These alerts will go through a cascade of filters to select interesting candidates. For the prototype, `interesting' is defined as the rarest or most unusual alert, but future systems will accommodate multiple filtering goals. The system is designed to be flexible, allowing users to access the stream at multiple points throughout the process, and to insert custom filters where necessary. We describe the basic architecture of ANTARES and the principles that will guide development and implementation.
SPLAT-VO is a powerful graphical tool for displaying, comparing, modifying and analyzing astronomical spectra, as well as searching and retrieving spectra from services around the world using Virtual Observatory (VO) protocols and services. The development of SPLAT-VO started in 1999, as part of the Starlink StarJava initiative, sometime before that of the VO, so initial support for the VO was necessarily added once VO standards and services became available. Further developments were supported by the Joint Astronomy Centre, Hawaii until 2009. Since end of 2011 development of SPLAT-VO has been continued by the German Astrophysical Virtual Observatory, and the Astronomical Institute of the Academy of Sciences of the Czech Republic. From this time several new features have been added, including support for the latest VO protocols, along with new visualization and spectra storing capabilities. This paper presents the history of SPLAT-VO, it's capabilities, recent additions and future plans, as well as a discussion on the motivations and lessons learned up to now.
CCAT will be a 25m diameter sub-millimeter telescope capable of operating in the 0.2 to 2.1mm wavelength range. It will be located at an altitude of 5600m on Cerro Chajnantor in northern Chile near the ALMA site. The anticipated first generation instruments include large format (60,000 pixel) kinetic inductance detector (KID) cameras, a large format heterodyne array and a direct detection multi-object spectrometer. The paper describes the architecture of the CCAT software and the development strategy.
Current ground-based submillimeter instruments (e.g. SCUBA-2, SHARC-2 and LABOCA) have hundreds to thousands of detectors, sampled at tens to hundreds of hertz, generating up to hundreds of gigabytes per night. Since noise is correlated between detectors and in time, due to atmospheric signals and temperature oscillations, naive map-making is not applicable. In addition, the size of the data sets makes direct likelihood based inversion techniques intractable. As a result, the data reduction approach for most current submm cameras is to adopt iterative methods in order to separate noise from sky signal, and hence effectively produce astronomical images. We investigate how today's map-makers scale to the next generation of instruments, which will have tens of thousands of detectors sampled at thousands of hertz, leading to data sets of challenging size. We propose strategies for reducing such large data sets.
CCAT is a sub-millimeter telescope to be built on Cerro Chajnantor in Chile near the ALMA site. The remote location means that all observing will be done by remote observers with the future goal of fully autonomous observing using a dynamic scheduler. The fully autonomous observing mode provides a natural means for accepting transient alert notifications for immediate follow up.
CCAT will be a 25-meter telescope for sub millimeter astronomy located at 5600m altitude on Cerro Chajnantor in northern Chile. CCAT will combine high sensitivity, a wide field of view, and a broad wavelength range (0.35 to 2.1mm) to provide an unprecedented capability for deep, large-area multicolor submillimeter surveys. It is planned to have a suite of instruments including large format KID cameras, a large heterodyne array and a KID-based direct detection multi-object spectrometer. The remote location drives a desire for fully autonomous observing coupled with data reduction pipelines and fast feedback to principal investigators.
We mapped five massive star forming regions with the SCUBA-2 camera on the James Clerk Maxwell Telescope (JCMT). Temperature and column density maps are obtained from the SCUBA-2 450 and 850 $μ$m images. Most of the dense clumps we find have central temperatures below 20 K with some as cold as 8 K, suggesting that they have no internal heating due to the presence of embedded protostars. This is surprising, because at the high densities inferred from these images and at these low temperatures such clumps should be unstable, collapsing to form stars and generating internal heating. The column densities at the clump centres exceed 10$^{23}$ cm$^{-2}$, and the derived peak visual extinction values are from 25-500 mag for $β$ = 1.5-2.5, indicating highly opaque centres. The observed cloud gas masses range from $\sim$ 10 to 10$^{3}$ M$_{\odot}$. The outer regions of the clumps follow an $r^{-2.36\pm0.35}$ density distribution and this power-law structure is observed outside of typically 10$^{4}$ AU. All these findings suggest that these clumps are high-mass starless clumps and most likely contain high-mass starless cores.
MS$\,$0451.6$-$0305 is a rich galaxy cluster whose strong lensing is particularly prominent at submm wavelengths. We combine new SCUBA-2 data with imaging from Herschel SPIRE and PACS and HST in order to try to understand the nature of the sources being lensed. In the region of the "giant submm arc," we uncover seven multiply imaged galaxies (up from the previously known three), of which six are found to be at a redshift of $z\sim2.9$, and possibly constitute an interacting system. Using a novel forward-modelling approach, we are able to simultaneously deblend and fit SEDs to the individual galaxies that contribute to the giant submm arc, constraining their dust temperatures, far infrared luminosities and star formation rates. The submm arc first identified by SCUBA can now be seen to be composed of at least five distinct sources, four of these within the galaxy group at $z\sim2.9$. The total unlensed luminosity for this galaxy group is $(3.1\pm0.3) \times 10^{12}\,\mathrm{L}_\odot$, which gives an unlensed star formation rate of $(450\pm50)$ M$_\odot$ yr$^{-1}$. From the properties of this system, we see no evidence of evolution towards lower temperatures in the dust temperature versus far-infrared luminosity relation for high redshift galaxies.