SearcharxivSearch

arXiv subjects

Andrew Hanushevsky

Publications and source records attributed to Andrew Hanushevsky.

5 recordsLinked to original sources

Data Movement Model for the Vera C. Rubin Observatory

The sky images captured nightly by the camera on the Vera C. Rubin Observatory's telescope will be processed across facilities on three continents. Data acquisition will occur at the observatory's location on Cerro Pach\'{o}n in the Andes mountains of Chile. A first copy of the raw image data set is stored at the summit and immediately transmitted via dedicated network links to the archive center within the US Data Facility at SLAC National Accelerator Laboratory in California, USA and from there to two European facilities for processing and archiving purposes. Data products resulting from periodic processing campaigns of the entire set of images collected since the beginning of the survey are made available to the scientific community in the form of data releases. In this paper we present an overall view of how we leverage the tools selected for managing the movement of data among the Rubin processing and serving facilities, including Rucio and FTS. We also present the tools we developed to integrate Rucio's data model and Rubin's Data Butler, the software abstraction layer that mediates all access to storage by pipeline tasks that implement science algorithms.

astro-ph.IM

Creating a content delivery network for general science on the internet backbone using XCaches

A general problem faced by computing on the grid for opportunistic users is that delivering cycles is simpler than delivering data to those cycles. In this project we show how we integrated XRootD caches placed on the internet backbone to implement a content delivery network for general science workflows. We will show that for some workflows on different science domains like high energy physics, gravitational waves, and others the combination of data reuse from the workflows together with the use of caches increases CPU efficiency while decreasing network bandwidth use.

cs.DC

HEP Software Foundation Community White Paper Working Group -- Data Organization, Management and Access (DOMA)

Without significant changes to data organization, management, and access (DOMA), HEP experiments will find scientific output limited by how fast data can be accessed and digested by computational resources. In this white paper we discuss challenges in DOMA that HEP experiments, such as the HL-LHC, will face as well as potential ways to address them. A research and development timeline to assess these changes is also proposed.

physics.comp-ph

Designing a Multi-petabyte Database for LSST

The 3.2 giga-pixel LSST camera will produce approximately half a petabyte of archive images every month. These data need to be reduced in under a minute to produce real-time transient alerts, and then added to the cumulative catalog for further analysis. The catalog is expected to grow about three hundred terabytes per year. The data volume, the real-time transient alerting requirements of the LSST, and its spatio-temporal aspects require innovative techniques to build an efficient data access system at reasonable cost. As currently envisioned, the system will rely on a database for catalogs and metadata. Several database systems are being evaluated to understand how they perform at these data rates, data volumes, and access patterns. This paper describes the LSST requirements, the challenges they impose, the data access philosophy, results to date from evaluating available database technologies against LSST requirements, and the proposed database architecture to meet the data challenges.

cs.DB

On the Verge of One Petabyte - the Story Behind the BaBar Database System

The BaBar database has pioneered the use of a commercial ODBMS within the HEP community. The unique object-oriented architecture of Objectivity/DB has made it possible to manage over 700 terabytes of production data generated since May'99, making the BaBar database the world's largest known database. The ongoing development includes new features, addressing the ever-increasing luminosity of the detector as well as other changing physics requirements. Significant efforts are focused on reducing space requirements and operational costs. The paper discusses our experience with developing a large scale database system, emphasizing universal aspects which may be applied to any large scale system, independently of underlying technology used.

cs.DB