arXiv subjects

Aniruddha R. Thakar

Publications and source records attributed to Aniruddha R. Thakar.

11 recordsLinked to original sources

arXiv2020

SciServer: a Science Platform for Astronomy and Beyond

We present SciServer, a science platform built and supported by the Institute for Data Intensive Engineering and Science at the Johns Hopkins University. SciServer builds upon and extends the SkyServer system of server-side tools that introduced the astronomical community to SQL (Structured Query Language) and has been serving the Sloan Digital Sky Survey catalog data to the public. SciServer uses a Docker/VM based architecture to provide interactive and batch mode server-side analysis with scripting languages like Python and R in various environments including Jupyter (notebooks), RStudio and command-line in addition to traditional SQL-based data analysis. Users have access to private file storage as well as personal SQL database space. A flexible resource access control system allows users to share their resources with collaborators, a feature that has also been very useful in classroom environments. All these services, wrapped in a layer of REST APIs, constitute a scalable collaborative data-driven science platform that is attractive to science disciplines beyond astronomy.

astro-ph.IM↗

arXiv2012

Running a distributed virtual observatory: US Virtual Astronomical Observatory operations

Operation of the US Virtual Astronomical Observatory shares some issues with modern physical observatories, e.g., intimidating data volumes and rapid technological change, and must also address unique concerns like the lack of direct control of the underlying and scattered data resources, and the distributed nature of the observatory itself. In this paper we discuss how the VAO has addressed these challenges to provide the astronomical community with a coherent set of science-enabling tools and services. The distributed nature of our virtual observatory-with data and personnel spanning geographic, institutional and regime boundaries-is simultaneously a major operational headache and the primary science motivation for the VAO. Most astronomy today uses data from many resources. Facilitation of matching heterogeneous datasets is a fundamental reason for the virtual observatory. Key aspects of our approach include continuous monitoring and validation of VAO and VO services and the datasets provided by the community, monitoring of user requests to optimize access, caching for large datasets, and providing distributed storage services that allow user to collect results near large data repositories. Some elements are now fully implemented, while others are planned for subsequent years. The distributed nature of the VAO requires careful attention to what can be a straightforward operation at a conventional observatory, e.g., the organization of the web site or the collection and combined analysis of logs. Many of these strategies use and extend protocols developed by the international virtual observatory community.

astro-ph.IM↗

arXiv2007

Large-Scale Query and XMatch, Entering the Parallel Zone

Current and future astronomical surveys are producing catalogs with millions and billions of objects. On-line access to such big datasets for data mining and cross-correlation is usually as highly desired as unfeasible. Providing these capabilities is becoming critical for the Virtual Observatory framework. In this paper we present various performance tests that show how using Relational Database Management Systems (RDBMS) and a Zoning algorithm to partition and parallelize the computation, we can facilitate large-scale query and cross-match.

cs.DB↗

arXiv2006

Constraining the Projected Radial Distribution of Galactic Satellites with the Sloan Digital Sky Survey

We use the Sloan Digital Sky Survey (SDSS) spectroscopic sample to constrain the projected radial distribution of satellites around isolated ~ L* galaxies. We employ mock galaxy catalogs derived from high-resolution cosmological simulations to investigate the effects of interloper contamination and show that interlopers significantly bias the estimated slope of the projected radial distribution of satellites. We also show that the distribution of interlopers around galaxies is expected to be non-uniform in velocity space because galaxies are clustered and reside in crowded environments. Successful methods of interloper contamination correction should therefore take into account environments of the host galaxies. Two such new methods are presented and the most reliable of them is used to correct for interloper contamination in analyses of the SDSS galaxy sample. The best fit power-law slope of the interloper-corrected surface density distribution of satellites, Sigma(R) ~ R^alpha, in the volume-limited SDSS sample is alpha ~ -1.7 +/- 0.1, independent of the galaxy and satellite luminosities. Comparison with Lambda-CDM simulations shows that the radial distribution of the SDSS satellites is more concentrated than that of subhalos around galaxy-sized halos, especially at R < 100 kpc/h. The predicted dark matter radial distribution is somewhat more concentrated than the profile of the SDSS satellites, but the difference is not statistically significant for our sample.

astro-ph↗

arXiv2005

When Database Systems Meet the Grid

We illustrate the benefits of combining database systems and Grid technologies for data-intensive applications. Using a cluster of SQL servers, we reimplemented an existing Grid application that finds galaxy clusters in a large astronomical database. The SQL implementation runs an order of magnitude faster than the earlier Tcl-C-file-based implementation. We discuss why and how Grid applications can take advantage of database systems.

cs.DC↗

arXiv2004

There Goes the Neighborhood: Relational Algebra for Spatial Data Search

We explored ways of doing spatial search within a relational database: (1) hierarchical triangular mesh (a tessellation of the sphere), (2) a zoned bucketing system, and (3) representing areas as disjunctive-normal form constraints. Each of these approaches has merits. They all allow efficient point-in-region queries. A relational representation for regions allows Boolean operations among them and allows quick tests for point-in-region, regions-containing-point, and region-overlap. The speed of these algorithms is much improved by a zone and multi-scale zone-pyramid scheme. The approach has the virtue that the zone mechanism works well on B-Trees native to all SQL systems and integrates naturally with current query optimizers - rather than requiring a new spatial access method and concomitant query optimizer extensions. Over the last 5 years, we have used these techniques extensively in our work on SkyServer.sdss.org, and SkyQuery.net.

cs.DB↗

arXiv2004

Extending the SDSS Batch Query System to the National Virtual Observatory Grid

The Sloan Digital Sky Survey science database is approaching 2TB. While the vast majority of queries normally execute in seconds or minutes, this interactive execution time can be disproportionately increased by a small fraction of queries that take hours or days to run; either because they require non-index scans of the largest tables or because they request very large result sets. In response to this, we added a multi-queue job submission and tracking system. The transfer of very large result sets from queries over the network is another serious problem. Statistics suggested that much of this data transfer is unnecessary; users would prefer to store results locally in order to allow further cross matching and filtering. To allow local analysis, we implemented a system that gives users their own personal database (MyDB) at the portal site. Users may transfer data to their MyDB, and then perform further analysis before extracting it to their own machine. We intend to extend the MyDB and asynchronous query ideas to multiple NVO nodes. This implies development, in a distributed manner, of several features, which have been demonstrated for a single node in the SDSS Batch Query System (CasJobs). The generalization of asynchronous queries necessitates some form of MyDB storage as well as workflow tracking services on each node and coordination strategies among nodes.

cs.DB↗

arXiv2004

The Sloan Digital Sky Survey Science Archive: Migrating a Multi-Terabyte Astronomical Archive from Object to Relational DBMS

The Sloan Digital Sky Survey Science Archive is the first in a series of multi-Terabyte digital archives in Astronomy and other data-intensive sciences. To facilitate data mining in the SDSS archive, we adapted a commercial database engine and built specialized tools on top of it. Originally we chose an object-oriented database management system due to its data organization capabilities, platform independence, query performance and conceptual fit to the data. However, after using the object database for the first couple of years of the project, it soon began to fall short in terms of its query support and data mining performance. This was as much due to the inability of the database vendor to respond our demands for features and bug fixes as it was due to their failure to keep up with the rapid improvements in hardware performance, particularly faster RAID disk systems. In the end, we were forced to abandon the object database and migrate our data to a relational database. We describe below the technical issues that we faced with the object database and how and why we migrated to relational technology.

cs.DB↗

arXiv2003

Angular Clustering with Photometric Redshifts in the Sloan Digital Sky Survey: Bimodality in the Clustering Properties of Galaxies

Understanding the clustering of galaxies has long been a goal of modern observational cosmology. Utilizing our photometric redshift technique a volume limited sample containing more than 2 million galaxies is constructed from the SDSS galaxy catalog. In the largest such analysis to date, we study the angular clustering as a function of luminosity and spectral type. Using Limber's equation we calculate the clustering length for the full data set as r0=5.77+/-0.10 Mpc/h. We find that r0 increases with luminosity by a factor of 1.6 over the sampled luminosity range, in agreement with previous redshift surveys. We also find that both the clustering length and the slope of the correlation function depend on the galaxy type. In particular, by splitting the galaxies in four groups by their rest-frame type we find a bimodal behavior in their clustering properties. Galaxies with spectral types similar to elliptical galaxies have a correlation length of 6.59 +/- 0.17 Mpc/h and a slope of the angular correlation function of 0.96 +/- 0.05 while blue galaxies have a clustering length of 4.51 +/- 0.19 Mpc/h and a slope of 0.68 +/- 0.09. The two intermediate color groups behave like their more extreme 'siblings', rather than showing a gradual transition in slope. We discuss these correlations in the context of current cosmological models for structure formation.

astro-ph↗

arXiv1996

NGC 4138 - A Case Study in Counterrotating Disk Formation

The Sa(r) galaxy NGC 4138 has been recently found to contain an extensive counterrotating disk which appears to be still forming. Up to a third of the stars in the disk system may be on retrograde orbits. A counterrotating ring of H II regions, along with extended counterrotating H I gas, suggests that the retrograde material has been recently acquired in the gas phase and is still trickling in. Using numerical simulations, we have attempted to model the process by which the counterrotating mass has been accreted by this galaxy. We investigate two possibilities: continuous retrograde infall of gas, and a retrograde merger with a gas-rich dwarf galaxy. Both processes are successful in producing a counterrotating disk of the observed mass and dimensions without heating up the primary significantly. Contrary to our experience with a fiducial cold, thin primary disk, the gas-rich merger works well for the massive, compact primary disk of NGC 4138 even though the mass of the dwarf galaxy is a significant fraction of the mass of the primary disk. Although we have restricted ourselves mainly to coplanar infall and mergers, we report on one inclined infall simulation as well. We also explore the possibility that the H-alpha ring seen in the inner half of the disk is a consequence of counterrotating gas clouds colliding with corotating gas already present in the disk and forming stars in the process.

astro-ph↗

arXiv1995

Formation of Massive Counterrotating Disks in Spiral Galaxies

We present results of numerical simulations of the formation of a massive counterrotating gas disk in a spiral galaxy. Using a hierarchical tree gravity solver combined with a sticky-particle gas dissipation scheme for our simulations, we have investigated three mechanisms: episodic and continuous gas infall, and a merger with a gas-rich dwarf galaxy. We find that both episodic and continuous gas infall work reasonably well and are able to produce a substantial gas counterrotating disk without upsetting the stability of the existing disk drastically, but it is very important for the gas to be well-dispersed in phase-space and not form concentrated clumps prior to its absorption by the disk galaxy. The initial angular momentum of the gas also plays a crucial role in determining the scale length of the counterrotating disk formed and the time it takes to form. The rate of infall, i.e. the mass of gas falling in per unit time, has to be small enough to preclude excessive heating of the preexisting disk. It is much easier in general to produce a smaller counterrotating disk than it is to produce an extensive disk whose scale length is similar to that of the original prograde disk. A gas-rich dwarf merger does not appear to be a viable mechanism to produce a massive counterrotating disk, because only a very small dwarf galaxy can produce a counterrotating disk without increasing the thickness of the existing disk by an order of magnitude, and the time-scale for this process is prohibitively long because it makes it very unlikely that several such mergers can accumulate a massive counterrotating disk over a Hubble time.

astro-ph↗