A pseudo-parallel Python environment for database curation

Eckhard Sutorius; Johann Bryant; Ross Collins; Nicholas Cross; Nigel Hambly; Mike Read

Home/Archive/arXiv

arXiv · 0711.2042

A pseudo-parallel Python environment for database curation

Eckhard Sutorius·Johann Bryant·Ross Collins·Nicholas Cross·Nigel Hambly·Mike Read

View original source ↗Read source document ↗Export metadata ↓

Abstract

One of the major challenges providing large databases like the WFCAM Science Archive (WSA) is to minimize ingest times for pixel/image metadata and catalogue data. In this article we describe how the pipeline processed data are ingested into the database as the first stage in building a release database which will be succeeded by advanced processing (source merging, seaming, detection quality flagging etc.). To accomplish the ingestion procedure as fast as possible we use a mixed Python/C++ environment and run the required tasks in a simple parallel modus operandi where the data are split into daily chunks and then processed on different computers. The created data files can be ingested into the database immediately as they are available. This flexible way of handling the data allows the most usage of the available CPUs as the comparison with sequential processing shows.

Explore related subjects

astro-ph

Keep this discovery

BibTeX RIS

Eckhard Sutorius, Johann Bryant, Ross Collins, Nicholas Cross, Nigel Hambly, Mike Read. 2007-11-13. A pseudo-parallel Python environment for database curation. https://arxiv.org/abs/0711.2042

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2010

HX-POL - A Balloon-Borne Hard X-Ray Polarimeter

We report on the design and estimated performance of a balloon-borne hard X-ray polarimeter called HX-POL. The experiment uses a combination of Si and Cadmium Zinc Telluride detectors to measure the polarization of 50 keV-400 keV X-rays from cosmic sources through the dependence of the angular distribution of Compton scattered photons on the polarization direction. On a one-day balloon flight, HX-POL would allow us to measure the polarization of bright Crab-like sources for polarization degrees well below 10%. On a longer (15-30 day) flight from Australia or Antarctica, HX-POL would be be able to measure the polarization of bright galactic X-ray sources down to polarization degrees of a few percent. Hard X-ray polarization measurements provide unique venues for the study of particle acceleration processes by compact objects and relativistic outflows. In this paper, we discuss the overall instrument design and performance. Furthermore, we present results from laboratory tests of the Si and CZT detectors.

astro-ph↗

arXiv2010

Quintessence models with an oscillating equation of state and their potentials

In this paper, we investigate the quintessence models with an oscillating equation of state (EoS) and their potentials. From the constructed potentials, which have the EoS of $ω_ϕ=ω_0+ω_1\sin z$, we find they are all the oscillating functions of the field $ϕ$, and the oscillating amplitudes are decreasing (or increasing) with $ϕ$. From the evolutive equation of the field $ϕ$, we find this is caused by the expansion of the universe. This also makes that it is very difficult to build a model whose EoS oscillates forever. However one can build a model with EoS oscillating for a period. Then we discuss three quintessence models, which are the combinations of the invert power law functions and the oscillating functions of the field $ϕ$. We find they all follow the oscillating EoS.

astro-ph↗

arXiv2010

Is "Spike" a Reliable Feature in Porb Distribution of AM HER Stars?

Orbital periods in AM Her stars (polars) are synchronized with spin periods of white dwarf by its high magnetic field. Since the last study of Porb distribution of these systems, the number of known objects of such type has more than doubled. This challenged us to compile a new updated catalogue of cataclysmic variables with highly magnetic white dwarfs (polars) and to study their Porb distribution. In this paper we also discus if "spike" is reliable feature in the distribution. ("Spike" is a concentration of polars in the distribution of their orbital periods near Porb = 114 min and was previously discussed by Ritter & Kolb (1992) and Shahbaz & Wood (1996).)

astro-ph↗