Searcharxiv⌕ Search

arXiv subjects

Jim Gray

Publications and source records attributed to Jim Gray.

At least 55 records · Page 3Linked to original sources
arXiv2004

The Sloan Digital Sky Survey Science Archive: Migrating a Multi-Terabyte Astronomical Archive from Object to Relational DBMS

The Sloan Digital Sky Survey Science Archive is the first in a series of multi-Terabyte digital archives in Astronomy and other data-intensive sciences. To facilitate data mining in the SDSS archive, we adapted a commercial database engine and built specialized tools on top of it. Originally we chose an object-oriented database management system due to its data organization capabilities, platform independence, query performance and conceptual fit to the data. However, after using the object database for the first couple of years of the project, it soon began to fall short in terms of its query support and data mining performance. This was as much due to the inability of the database vendor to respond our demands for features and bug fixes as it was due to their failure to keep up with the rapid improvements in hardware performance, particularly faster RAID disk systems. In the end, we were forced to abandon the object database and migrate our data to a relational database. We describe below the technical issues that we faced with the object database and how and why we migrated to relational technology.

cs.DB↗
arXiv2004

A Quick Look at SATA Disk Performance

We have been investigating the use of low-cost, commodity components for multi-terabyte SQL Server databases. Dubbed storage bricks, these servers are white box PCs containing the largest ATA drives, value-priced AMD or Intel processors, and inexpensive ECC memory. One issue has been the wiring mess, air flow problems, length restrictions, and connector failures created by seven or more parallel ATA (PATA) ribbon cables and drives in]a tower or 3U rack-mount chassis. Large capacity Serial ATA (SATA) drives have recently become widely available for the PC environment at a reasonable price. In addition to being faster, the SATA connectors seem more reliable, have a more reasonable length restriction (1m) and allow better airflow. We tested two drive brands along with two RAID controllers to evaluate SATA drive performance and reliablility. This paper documents our results so far.

cs.DB↗
arXiv2004

Cosmological Parameters from Eigenmode Analysis of Sloan Digital Sky Survey Galaxy Redshifts

We present estimates of cosmological parameters from the application of the Karhunen-Loeve transform to the analysis of the 3D power spectrum of density fluctuations using Sloan Digital Sky Survey galaxy redshifts. We use Omega_m*h and f_b = Omega_b/Omega_m to describe the shape of the power spectrum, sigma8 for the (linearly extrapolated) normalization, and beta to parametrize linear theory redshift space distortions. On scales k < 0.16 h/Mpc, our maximum likelihood values are Omega_m*h = 0.264 +/-0.043, f_b = 0.286 +/- 0.065, sigma8 = 0.966 +/- 0.048, and beta = 0.45 +/- 0.12. When we take a prior on Omega_b from WMAP, we find Omega_m*h = 0.207 +/- 0.030, which is in excellent agreement with WMAP and 2dF. This indicates that we have reasonably measured the gross shape of the power spectrum but we have difficulty breaking the degeneracy between Omega_m*h and f_b because the baryon oscillations are not resolved in the current spectroscopic survey window function.

astro-ph↗
arXiv2003

The Lowell Database Research Self Assessment

A group of senior database researchers gathers every few years to assess the state of database research and to point out problem areas that deserve additional focus. This report summarizes the discussion and conclusions of the sixth ad-hoc meeting held May 4-6, 2003 in Lowell, Mass. It observes that information management continues to be a critical component of most complex software systems. It recommends that database researchers increase focus on: integration of text, data, code, and streams; fusion of information from heterogeneous data sources; reasoning about uncertain data; unsupervised data mining for interesting correlations; information privacy; and self-adaptation and repair.

cs.DB↗
arXiv2003

The Sloan Digital Sky Survey Quasar Catalog II. First Data Release

We present the second edition of the Sloan Digital Sky Survey (SDSS) Quasar Catalog. The catalog consists of 16713 objects in the SDSS First Data Release (DR1) that have luminosities larger than M_i=-22 (H_0=70 km/s, Omega_M=0.3, Omega_Lambda=0.7), have at least one emission line with FWHM larger than 1000 km/s, and have highly reliable redshifts. The area covered is ~1360 deg^2 and the redshifts range from 0.08 to 5.41, with a median value of 1.43. Each object has positions accurate to better than 0.2" rms per coordinate, five-band (ugriz) CCD-based photometry with typical accuracy of 0.03 mag, and information on the morphology and selection method, in addition to radio, near-infrared, and X-ray emission properties of the quasars, when available, from other large area surveys. Calibrated digital spectra covering 3800-9200 Angstroms at a spectral resolution 1800-2100, are available. This publication supersedes the first SDSS Quasar Catalog, which was based on material from the SDSS Early Data Release. A summary of corrections to current quasar databases is provided. The majority of the objects were found in SDSS commissioning data using a multicolor selection technique. Since the quasar selection algorithm was undergoing testing during the entire DR1 observational period, care must be taken when assembling samples for use in statistical studies. A total of 15786 objects (94%) in the catalog were discovered by the SDSS; 12,173 of which are reported here for the first time, including five quasars brighter than i=16.0 and 17 quasars with redshifts larger than 4.5.

astro-ph↗
arXiv2002

TerraService.NET: An Introduction to Web Services

This article explores the design and construction of a geo-spatial Internet web service application from the host web site perspective and from the perspective of an application using the web service. The TerraService.NET web service was added to the popular TerraServer database and web site with no major structural changes to the database. The article discusses web service design, implementation, and deployment concepts and design guidelines. Web services enable applications that aggregate and interact with information and resources from Internet-scale distributed servers. The article presents the design of two USDA applications that interoperate with database and web service resources in Fort Collins Colorado and the TerraService web service located in Tukwila Washington.

cs.DL↗
arXiv2002

TeraScale SneakerNet: Using Inexpensive Disks for Backup, Archiving, and Data Exchange

Large datasets are most economically trnsmitted via parcel post given the current economics of wide-area networking. This article describes how the Sloan Digital Sky Survey ships terabyte scale datasets both within the US and to Europe and Asia. We 3GT storage bricks (Ghz processor, GB ram, GbpsEthernet, TB disk) for about 2k$ each. These bricks act as database servers on the LAN. They are loaded at one site and read at the second site. The paper describes the bricks, their economics, and some software issues that they raise.

cs.NI↗
arXiv2002

Petabyte Scale Data Mining: Dream or Reality?

Science is becoming very data intensive1. Today's astronomy datasets with tens of millions of galaxies already present substantial challenges for data mining. In less than 10 years the catalogs are expected to grow to billions of objects, and image archives will reach Petabytes. Imagine having a 100GB database in 1996, when disk scanning speeds were 30MB/s, and database tools were immature. Such a task today is trivial, almost manageable with a laptop. We think that the issue of a PB database will be very similar in six years. In this paper we scale our current experiments in data archiving and analysis on the Sloan Digital Sky Survey2,3 data six years into the future. We analyze these projections and look at the requirements of performing data mining on such data sets. We conclude that the task scales rather well: we could do the job today, although it would be expensive. There do not seem to be any show-stoppers that would prevent us from storing and using a Petabyte dataset six years from today.

cs.DB↗
arXiv2002

Online Scientific Data Curation, Publication, and Archiving

Science projects are data publishers. The scale and complexity of current and future science data changes the nature of the publication process. Publication is becoming a major project component. At a minimum, a project must preserve the ephemeral data it gathers. Derived data can be reconstructed from metadata, but metadata is ephemeral. Longer term, a project should expect some archive to preserve the data. We observe that pub-lished scientific data needs to be available forever ? this gives rise to the data pyramid of versions and to data inflation where the derived data volumes explode. As an example, this article describes the Sloan Digital Sky Survey (SDSS) strategies for data publication, data access, curation, and preservation.

cs.DL↗
arXiv2002

Web Services for the Virtual Observatory

Web Services form a new, emerging paradigm to handle distributed access to resources over the Internet. There are platform independent standards (SOAP, WSDL), which make the developers? task considerably easier. This article discusses how web services could be used in the context of the Virtual Observatory. We envisage a multi-layer architecture, with interoperating services. A well-designed lower layer consisting of simple, standard services implemented by most data providers will go a long way towards establishing a modular architecture. More complex applications can be built upon this core layer. We present two prototype applications, the SdssCutout and the SkyQuery as examples of this layered architecture.

cs.DC↗
arXiv2002

Spatial Clustering of Galaxies in Large Datasets

Datasets with tens of millions of galaxies present new challenges for the analysis of spatial clustering. We have built a framework that integrates a database of object catalogs, tools for creating masks of bad regions, and a fast (NlogN) correlation code. This system has enabled unprecedented efficiency in carrying out the analysis of galaxy clustering in the SDSS catalog. A similar approach is used to compute the three-dimensional spatial clustering of galaxies on very large scales. We describe our strategy to estimate the effect of photometric errors using a database. We discuss our efforts as an early example of data-intensive science. While it would have been possible to get these results without the framework we describe, it will be infeasible to perform these computations on the future huge datasets without using this framework.

cs.DB↗
arXiv2002

The SDSS SkyServer: Public Access to the Sloan Digital Sky Server Data

The SkyServer provides Internet access to the public Sloan Digi-tal Sky Survey (SDSS) data for both astronomers and for science education. This paper describes the SkyServer goals and archi-tecture. It also describes our experience operating the SkyServer on the Internet. The SDSS data is public and well-documented so it makes a good test platform for research on database algorithms and performance.

cs.DL↗
arXiv2002

Data Mining the SDSS SkyServer Database

An earlier paper (Szalay et. al. "Designing and Mining MultiTerabyte Astronomy Archives: The Sloan Digital Sky Survey," ACM SIGMOD 2000) described the Sloan Digital Sky Survey's (SDSS) data management needs by defining twenty database queries and twelve data visualization tasks that a good data management system should support. We built a database and interfaces to support both the query load and also a website for ad-hoc access. This paper reports on the database design, describes the data loading pipeline, and reports on the query implementation and performance. The queries typically translated to a single SQL statement. Most queries run in less than 20 seconds, allowing scientists to interactively explore the database. This paper is an in-depth tour of those queries. Readers should first have studied the companion overview paper Szalay et. al. "The SDSS SkyServer, Public Access to the Sloan Digital Sky Server Data" ACM SIGMOND 2002.

cs.DB↗
arXiv2001

The SDSS SkyServer, Public Access to the Sloan Digital Sky Server Data

The SkyServer provides Internet access to the public Sloan Digital Sky Survey (SDSS) data for both astronomers and for science education. This paper describes the SkyServer goals and architecture. It also describes our experience operating the SkyServer on the Internet. The SDSS data is public and well-documented so it makes a good test platform for research on database algorithms and performance.

cs.DL↗
arXiv1999

The Sloan Digital Sky Survey and its Archive

The next-generation astronomy archives will cover most of the universe at fine resolution in many wavelengths. One of the first of these projects, the Sloan Digital Sky Survey (SDSS) will create a 5-wavelength catalog over 10,000 square degrees of the sky. The 200 million objects in the multi-terabyte database will have mostly numerical attributes, defining a space of 100+ dimensions. Points in this space have highly correlated distributions. The archive will enable astronomers to explore the data interactively. Data access will be aided by multidimensional spatial indices. The data will be partitioned in many ways. Small tag objects consisting of the most popular attributes speed up frequent searches. Splitting the data among multiple servers enables parallel, scalable I/O. Hashing techniques allow efficient clustering and pairwise comparison algorithms. Randomly sampled subsets allow debugging otherwise large queries at the desktop. Central servers will operate a data pump that supports sweeping searches that touch most of the data.

astro-ph↗
arXiv1999

What Next? A Dozen Information-Technology Research Goals

Charles Babbage's vision of computing has largely been realized. We are on the verge of realizing Vannevar Bush's Memex. But, we are some distance from passing the Turing Test. These three visions and their associated problems have provided long-range research goals for many of us. For example, the scalability problem has motivated me for several decades. This talk defines a set of fundamental research problems that broaden the Babbage, Bush, and Turing visions. They extend Babbage's computational goal to include highly-secure, highly-available, self-programming, self-managing, and self-replicating systems. They extend Bush's Memex vision to include a system that automatically organizes, indexes, digests, evaluates, and summarizes information (as well as a human might). Another group of problems extends Turing's vision of intelligent machines to include prosthetic vision, speech, hearing, and other senses. Each problem is simply stated and each is orthogonal from the others, though they share some common core technologies

cs.GL↗
arXiv1999

Designing and Mining Multi-Terabyte Astronomy Archives: The Sloan Digital Sky Survey

The next-generation astronomy digital archives will cover most of the universe at fine resolution in many wave-lengths, from X-rays to ultraviolet, optical, and infrared. The archives will be stored at diverse geographical locations. One of the first of these projects, the Sloan Digital Sky Survey (SDSS) will create a 5-wavelength catalog over 10,000 square degrees of the sky (see http://www.sdss.org/). The 200 million objects in the multi-terabyte database will have mostly numerical attributes, defining a space of 100+ dimensions. Points in this space have highly correlated distributions. The archive will enable astronomers to explore the data interactively. Data access will be aided by a multidimensional spatial index and other indices. The data will be partitioned in many ways. Small tag objects consisting of the most popular attributes speed up frequent searches. Splitting the data among multiple servers enables parallel, scalable I/O and applies parallel processing to the data. Hashing techniques allow efficient clustering and pair-wise comparison algorithms that parallelize nicely. Randomly sampled subsets allow debugging otherwise large queries at the desktop. Central servers will operate a data pump that supports sweeping searches that touch most of the data. The anticipated queries require special operators related to angular distances and complex similarity tests of object properties, like shapes, colors, velocity vectors, or temporal behaviors. These issues pose interesting data management challenges.

cs.DB↗