Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · astro-ph/0508145

Optimized Data Loading for a Multi-Terabyte Sky Survey Repository

Y. Dora Cai·Ruth Aydt·Robert J. Brunner
View original source ↗Read source document ↗Export metadata ↓

Abstract

Advanced instruments in a variety of scientific domains are collecting massive amounts of data that must be post-processed and organized to support scientific research activities. Astronomers have been pioneers in the use of databases to host highly structured repositories of sky survey data. As more powerful telescopes come online, the increased volume and complexity of the data collected poses enormous challenges to state-of-the-art database systems and data-loading techniques. When the data source is an instrument taking ongoing samples, the database loading must, at a minimum, keep up with the data-acquisition rate. These challenges are being faced not only by the astronomy community, but also by other scientific disciplines interested in building scalable databases to house multi-terabyte archives of complex structured data. In this paper we present SkyLoader, our novel framework for fast and scalable data loading that is being used to populate a multi-table, multi-terabyte database repository for the Palomar-Quest sky survey. Our framework consists of an efficient algorithm for bulk loading, an effective data structure to support data integrity and proper error handling during the loading process, support for optimized parallelism that matches the number of concurrent loaders with the database host capabilities, and guidelines for database and system tuning. Performance studies showing the positive effects of the adopted strategies are also presented. Our parallel bulk loading with array buffering technique has made fast population of a multi-terabyte repository a reality, reducing the loading time for a 40-gigabyte data set from more than 20 hours to less than 3 hours. We believe our framework offers a promising approach for loading other large and complex scientific databases.

Explore related subjects

astro-ph

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Y. Dora Cai, Ruth Aydt, Robert J. Brunner. 2005-08-04. Optimized Data Loading for a Multi-Terabyte Sky Survey Repository. https://arxiv.org/abs/astro-ph/0508145

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2020

Cosmic Variance and Its Effect on the Luminosity Function Determination in Deep High z Surveys

We study cosmic variance in deep high redshift surveys and its influence on the determination of the luminosity function for high redshift galaxies. For several survey geometries relevant for HST and JWST instruments, we characterize the distribution of the galaxy number counts. This is obtained by means of analytic estimates via the two point correlation function in extended Press-Schechter theory as well as by using synthetic catalogs extracted from N-body cosmological simulations of structure formation. We adopt a simple luminosity - dark halo mass relation to investigate the environment effects on the fitting of the luminosity function. We show that in addition to variations of the normalization of the luminosity function, a steepening of its slope is also expected in underdense fields, similarly to what is observed within voids in the local universe. Therefore, to avoid introducing artificial biases, caution must be taken when attempting to correct for field underdensity, such as in the case of HST UDF i-dropout sample, which exhibits a deficit of bright counts with respect to the average counts in GOODS. A public version of the cosmic variance calculator based on the two point correlation function integration is made available on the web.

astro-ph↗
arXiv2020

Receiver control for the Submillimeter Array

Efficient operation of a submillimeter interferometer requires remote (preferably automated) control of mechanically tuned local oscillators, phase-lock loops, mixers, optics, calibration vanes and cryostats. The present control system for these aspects of the Submillimeter Array (SMA) will be described. Distributed processing forms the underlying architecture and the software is split between hardware platforms in a leader/follower arrangement. In each antenna cabin, a serial network of up to ten independent 80C196 microcontroller boards attaches to the real-time PowerPC computer (running LynxOS). A multi-threaded, gcc-compiled leader program on the PowerPC accepts top-level requests via remote procedure calls (RPC), subsequently dispatches tuning commands to the relevant follower microcontrollers, and regularly reports the system status to optical-fiber-based reflective memory for common access by the telescope monitor and error reporting system. All serial communication occurs asynchronously via encoded, variable-length packets. The microcontrollers respond to the requested commands and queries by accessing non-volatile, rewriteable lookup-tables (when appropriate) and executing embedded software that operates additional electronic devices (DACs, ADCs, etc.). Since various receiver hardware components require linear or rotary motion, each microcontroller also implements a position servo via a one-millisecond interrupt service routine which drives a DC-motor/encoder combination that remains standard across each subsystem.

astro-ph↗
arXiv2019

Classical analysis of the rotational dynamic of spiral galaxies: Quo Vadis Dark Matter?

In this paper we study a stellar dynamic model for the stars' rotational-dynamics, with a distribution of its own mass, rotating around its center with a higher density, like spiral galaxies happen, by means of a classical calculus of the rotation velocities of a particle around its rotational axis, inside a smoothed distribution of matter. The stars are supposed to be particles and their distribution in the galaxy is modelled as a matter distribution inversely proportional to its distance from its center. Two kinds of matter distribution are supposed: one with constant density, and other with radial distribution. Two types of galaxy symmetry are also considered: spherical and oblate ellipsoidal. Using only classical mechanics arguments it is shown that the calculated velocity distribution inside the galaxy is similar to that obtained from astronomical observations, without the necessity of suppose the existence of dark matter or other phenomena.

astro-ph↗