arXiv subjects
Publications and source records attributed to Hugo Buddelmeijer.
We extend the data-driven approach for astronomical pipelines to the METIS instrument and integrate the ESO Data Processing System (EDPS) by adapting it to support backward chaining.
We present the final data release of the Kilo-Degree Survey (KiDS-DR5), a public European Southern Observatory (ESO) wide-field imaging survey optimised for weak gravitational lensing studies. We combined matched-depth multi-wavelength observations from the VLT Survey Telescope and the VISTA Kilo-degree INfrared Galaxy (VIKING) survey to create a nine-band optical-to-near-infrared survey spanning $1347$ deg$^2$. The median $r$-band $5\sigma$ limiting magnitude is 24.8 with median seeing $0.7^{\prime\prime}$. The main survey footprint includes $4$ deg$^2$ of overlap with existing deep spectroscopic surveys. We complemented these data in DR5 with a targeted campaign to secure an additional $23$ deg$^2$ of KiDS- and VIKING-like imaging over a range of additional deep spectroscopic survey fields. From these fields, we extracted a catalogue of $126\,085$ sources with both spectroscopic and photometric redshift information, which enables the robust calibration of photometric redshifts across the full survey footprint. In comparison to previous releases, DR5 represents a $34\%$ areal extension and includes an $i$-band re-observation of the full footprint, thereby increasing the effective $i$-band depth by $0.4$ magnitudes and enabling multi-epoch science. Our processed nine-band imaging, single- and multi-band catalogues with masks, and homogenised photometry and photometric redshifts can be accessed through the ESO Archive Science Portal.
ScopeSim is a flexible multipurpose instrument data simulation framework built in Python. It enables both raw and reduced observation data to be simulated for a wide range of telescopes and instruments quickly and efficiently on a personal computer. The software is currently being used to generate simulated raw input data for developing the data reduction pipelines for the MICADO and METIS instruments at the ELT. The ScopeSim environment consists of three main packages which are responsible for providing on-sky target templates (ScopeSim_templates), the data to build the optical models of various telescopes and instruments (instrument reference database), and the simulation engine (ScopeSim). This strict division of responsibilities allows ScopeSim to be used to simulate observation data for many different instrument and telescope configurations for both imaging and spectroscopic instruments. ScopeSim has been built to avoid redundant calculations wherever possible. As such it is able to deliver simulated observations on time scales of seconds to minutes. All the code and data is open source and hosted on Github. The community is also most welcome, and indeed encouraged to contribute to code ideas, target templates, and instrument packages.
We design the imaging data calibration and reduction software for MICADO, the First Light near-IR instrument on the Extremely Large Telescope. In this process we have hit the limit of what can be achieved with a detailed software design that is primarily captured in pdf/word documents. Trade-offs between hardware and calibration software are required to meet stringent science requirements. To support such trade-offs, more software needs to be developed in the early phases of the project: simulators, archives, prototype recipes and pipelines. This requires continuous and efficient exchange of evolving designs between the software and hardware groups, which is hard to achieve with manually maintained documents. This, and maintaining the consistency between the design documents and various software components is possible with a machine readable version of the design. We construct a detailed design that is readable by both software and humans. From this the design documentation, prototype pipelines and data archives are generated automatically. We present the implementation of such an approach for the calibration software detailed design for the ELT MICADO imager which is based on expertise and lessons learned in earlier projects (e.g. OmegaCAM, MUSE, Euclid).
We present the curation and verification of a new combined optical and near infrared dataset for cosmology and astrophysics, derived from the combination of $ugri$-band imaging from the Kilo Degree Survey (KiDS) and $ZY\!J\!H\!K_{\rm s}$-band imaging from the VISTA Kilo degree Infrared Galaxy (VIKING) survey. This dataset is unrivaled in cosmological imaging surveys due to its combination of area ($458$ deg$^2$ before masking), depth ($r\le25$), and wavelength coverage ($ugriZY\!J\!H\!K_{\rm s}$). The combination of survey depth, area, and (most importantly) wavelength coverage allows significant reductions in systematic uncertainties (i.e. reductions of between 10 and 60\% in bias, outlier rate, and scatter) in photometric-to-spectroscopic redshift comparisons, compared to the optical-only case at photo-$z$ above $0.7$. The complementarity between our optical and NIR surveys means that over $80\%$ of our sources, across all photo-$z$, have significant detections (i.e. not upper limits) in our $8$ reddest bands. We derive photometry, photo-$z$, and stellar masses for all sources in the survey, and verify these data products against existing spectroscopic galaxy samples. We demonstrate the fidelity of our higher-level data products by constructing the survey stellar mass functions in 8 volume-complete redshift bins. We find that these photometrically derived mass functions provide excellent agreement with previous mass evolution studies derived using spectroscopic surveys. The primary data products presented in this paper are publicly available at http://kids.strw.leidenuniv.nl/.
Despite the high accuracy of photometric redshifts (zphot) derived using Machine Learning (ML) methods, the quantification of errors through reliable and accurate Probability Density Functions (PDFs) is still an open problem. First, because it is difficult to accurately assess the contribution from different sources of errors, namely internal to the method itself and from the photometric features defining the available parameter space. Second, because the problem of defining a robust statistical method, always able to quantify and qualify the PDF estimation validity, is still an open issue. We present a comparison among PDFs obtained using three different methods on the same data set: two ML techniques, METAPHOR (Machine-learning Estimation Tool for Accurate PHOtometric Redshifts) and ANNz2, plus the spectral energy distribution template fitting method, BPZ. The photometric data were extracted from the KiDS (Kilo Degree Survey) ESO Data Release 3, while the spectroscopy was obtained from the GAMA (Galaxy and Mass Assembly) Data Release 2. The statistical evaluation of both individual and stacked PDFs was done through quantitative and qualitative estimators, including a dummy PDF, useful to verify whether different statistical estimators can correctly assess PDF quality. We conclude that, in order to quantify the reliability and accuracy of any zphot PDF method, a combined set of statistical estimators is required.
Verlinde (2016) proposed that the observed excess gravity in galaxies and clusters is the consequence of Emergent Gravity (EG). In this theory the standard gravitational laws are modified on galactic and larger scales due to the displacement of dark energy by baryonic matter. EG gives an estimate of the excess gravity (described as an apparent dark matter density) in terms of the baryonic mass distribution and the Hubble parameter. In this work we present the first test of EG using weak gravitational lensing, within the regime of validity of the current model. Although there is no direct description of lensing and cosmology in EG yet, we can make a reasonable estimate of the expected lensing signal of low redshift galaxies by assuming a background LambdaCDM cosmology. We measure the (apparent) average surface mass density profiles of 33,613 isolated central galaxies, and compare them to those predicted by EG based on the galaxies' baryonic masses. To this end we employ the ~180 square degrees overlap of the Kilo-Degree Survey (KiDS) with the spectroscopic Galaxy And Mass Assembly (GAMA) survey. We find that the prediction from EG, despite requiring no free parameters, is in good agreement with the observed galaxy-galaxy lensing profiles in four different stellar mass bins. Although this performance is remarkable, this study is only a first step. Further advancements on both the theoretical framework and observational tests of EG are needed before it can be considered a fully developed and solidly tested theory.
The Kilo-Degree Survey (KiDS) is an optical wide-field imaging survey carried out with the VLT Survey Telescope and the OmegaCAM camera. KiDS will image 1500 square degrees in four filters (ugri), and together with its near-infrared counterpart VIKING will produce deep photometry in nine bands. Designed for weak lensing shape and photometric redshift measurements, the core science driver of the survey is mapping the large-scale matter distribution in the Universe back to a redshift of ~0.5. Secondary science cases are manifold, covering topics such as galaxy evolution, Milky Way structure, and the detection of high-redshift clusters and quasars. KiDS is an ESO Public Survey and dedicated to serving the astronomical community with high-quality data products derived from the survey data, as well as with calibration data. Public data releases will be made on a yearly basis, the first two of which are presented here. For a total of 148 survey tiles (~160 sq.deg.) astrometrically and photometrically calibrated, coadded ugri images have been released, accompanied by weight maps, masks, source lists, and a multi-band source catalog. A dedicated pipeline and data management system based on the Astro-WISE software system, combined with newly developed masking and source classification software, is used for the data production of the data products described here. The achieved data quality and early science projects based on the data products in the first two data releases are reviewed in order to validate the survey data. Early scientific results include the detection of nine high-z QSOs, fifteen candidate strong gravitational lenses, high-quality photometric redshifts and galaxy structural parameters for hundreds of thousands of galaxies. (Abridged)
A novel method to infer logical relationships between sets is presented. These sets can be any collection of elements, for example astronomical catalogs of celestial objects. The method does not require the contents of the sets to be known explicitly. It combines incomplete knowledge about the relationships between sets to infer a priori unknown relationships. Relationships between sets are represented by sets of Boolean hypercubes. This leads to deductive reasoning by application of logical operators to these sets of hypercubes. A pseudocode for an efficient implementation is described. The method is used in the Astro-WISE information system to infer relationships between catalogs of astronomical objects. These catalogs can be very large and, more importantly, their contents do not have to be available at all times. Science products are stored in Astro-WISE with references to other science products from which they are derived, or their dependencies. This creates full data lineage that links every science product all the way back to the raw data. Catalogs are created in a way that maximizes knowledge about their relationship with their dependencies. The presented algorithm is used to determine which objects a catalog represents by leveraging this information.
Astro-WISE is the Astronomical Wide-field Imaging System for Europe. It is a scientific information system which consists of hardware and software federated over about a dozen institutes throughout Europe. It has been developed to exploit the ever increasing avalanche of data produced by astronomical surveys and data intensive scientific experiments in general. The demo explains the architecture of the Astro-WISE information system and shows the use of Astro-WISE interfaces. Wide-field astronomical images are derived from the raw image to the final catalog according to the user's request. The demo is based on the standard Astro-WISE guided tour, which can be accessed from the Astro-WISE website. The typical Astro-WISE data processing chain is shown, which can be used for data handling for a variety of different instruments, currently 14, including OmegaCAM, MegaCam, WFI, WFC, ACS/HST, etc.
The request driven way of deriving data in Astro-WISE is extended to a query driven way of visualization. This allows scientists to focus on the science they want to perform, because all administration of their data is automated. This can be done over an abstraction layer that enhances control and flexibility for the scientist.
We present the design of a novel way of handling astronomical catalogs in Astro-WISE in order to achieve the scalability required for the data produced by large scale surveys. A high level of automation and abstraction is achieved in order to facilitate interoperation with visualization software for interactive exploration. At the same time flexibility in processing is enhanced and data is shared implicitly between scientists. This is accomplished by using a data model that primarily stores how catalogs are derived; the contents of the catalogs are only created when necessary and stored only when beneficial for performance. Discovery of existing catalogs and creation of new catalogs is done through the same process by directly requesting the final set of sources (astronomical objects) and attributes (physical properties) that is required, for example from within visualization software. New catalogs are automatically created to provide attributes of sources for which no suitable existing catalogs can be found. These catalogs are defined to contain the new attributes on the largest set of sources the calculation of the attributes is applicable to, facilitating reuse for future data requests. Subsequently, only those parts of the catalogs that are required for the requested end product are actually processed, ensuring scalability. The presented mechanisms primarily determine which catalogs are created and what data has to be processed and stored: the actual processing and storage itself is left to existing functionality of the underlying information system.
Interactive visualization of astronomical catalogs requires novel techniques due to the huge volumes and complex structure of the data produced by existing and upcoming astronomical surveys. The creation as well as the disclosure of the catalogs can be handled by data pulling mechanisms. These prevent unnecessary processing and facilitate data sharing by having users request the desired end products. In this work we present query driven visualization as a logical continuation of data pulling. Scientists can request catalogs in a declarative way and set process parameters directly from within the visualization. This results in profound interoperation between software with a high level of abstraction. New messages for the Simple Application Messaging Protocol are proposed to achieve this abstraction. Support for these messages are implemented in the Astro-WISE information system and in a set of demonstrational applications.