SearcharxivSearch

arXiv subjects

Jesús Salgado

Publications and source records attributed to Jesús Salgado.

10 recordsLinked to original sources

Bringing Computation to the data: Interoperable serverless function execution for astrophysical data analysis in the SRCNet

Serverless computing is a paradigm in which the underlying infrastructure is fully managed by the provider, enabling applications and services to be executed with elastic resource provisioning and minimal operational overhead. A core model within this paradigm is Function-as-a-Service (FaaS), where lightweight functions are deployed and triggered on demand, scaling seamlessly with workload. FaaS offers flexibility, cost-effectiveness, and fine-grained scalability, qualities particularly relevant for large-scale scientific infrastructures where data volumes are too large to centralise and computation must increasingly occur close to the data. The Square Kilometre Array Observatory (SKAO) exemplifies this challenge. Once operational, it will generate about 700~PB of data products annually, distributed across the SKA Regional Centre Network (SRCNet), a federation of international centres providing storage, computing, and analysis services. In such a context, FaaS offers a mechanism to bring computation to the data. We studied the principles of serverless and FaaS computing and explored their application to radio astronomy workflows. Representative functions for astrophysical data analysis were developed and deployed, including micro-functions derived from existing libraries and wrappers around domain-specific applications. In particular, a Gaussian convolution function was implemented and integrated within the SRCNet ecosystem. The use case demonstrates that FaaS can be embedded into the existing SRCNet ecosystem of services, allowing functions to run directly at sites where data replicas are stored. This reduces latency, minimises transfers, and improves efficiency, aligning with federated, data-proximate computation. The results show that serverless models provide a scalable and efficient pathway to address the data volumes of the SKA era.

cs.DC

Integration of storage endpoints into a Rucio data lake, as an activity to prototype a SKA Regional Centres Network

The Square Kilometre Array (SKA) infrastructure will consist of two radio telescopes that will be the most sensitive telescopes on Earth. The SKA community will have to process and manage near exascale data, which will be a technical challenge for the coming years. In this respect, the SKA Global Network of Regional Centres plays a key role in data distribution and management. The SRCNet will provide distributed computing and data storage capacity, as well as other important services for the network. Within the SRCNet, several teams have been set up for the research, design and development of 5 prototypes. One of these prototypes is related to data management and distribution, where a data lake has been deployed using Rucio. In this paper we focus on the tasks performed by several of the teams to deploy new storage endpoints within the SKAO data lake. In particular, we will describe the steps and deployment instructions for the services required to provide the Rucio data lake with a new Rucio Storage Element based on StoRM and WebDAV within the Spanish SRC prototype.

astro-ph.IM

An approach to provide serverless scientific pipelines within the context of SKA

Function-as-a-Service (FaaS) is a type of serverless computing that allows developers to write and deploy code as individual functions, which can be triggered by specific events or requests. FaaS platforms automatically manage the underlying infrastructure, scaling it up or down as needed, being highly scalable, cost-effective and offering a high level of abstraction. Prototypes being developed within the SKA Regional Center Network (SRCNet) are exploring models for data distribution, software delivery and distributed computing with the goal of moving and executing computation to where the data is. Since SKA will be the largest data producer on the planet, it will be necessary to distribute this massive volume of data to the SRCNet nodes that will serve as a hub for computing and analysis operations on the closest data. Within this context, in this work we want to validate the feasibility of designing and deploying functions and applications commonly used in radio interferometry workflows within a FaaS platform to demonstrate the value of this computing model as an alternative to explore for data processing in the distributed nodes of the SRCNet. We have analyzed several FaaS platforms and successfully deployed one of them, where we have imported several functions using two different methods: microfunctions from the CASA framework, which are written in Python code, and highly specific native applications like wsclean. Therefore, we have designed a simple catalogue that can be easily scaled to provide all the key features of FaaS in highly distributed environments using orchestrators, as well as having the ability to integrate them with workflows or APIs. This paper contributes to the ongoing discussion of the potential of FaaS models for scientific data processing, particularly in the context of large-scale, distributed projects such as SKA.

cs.DC

The Astropy Project: Sustaining and Growing a Community-oriented Open-source Project and the Latest Major Release (v5.0) of the Core Package

The Astropy Project supports and fosters the development of open-source and openly-developed Python packages that provide commonly needed functionality to the astronomical community. A key element of the Astropy Project is the core package $\texttt{astropy}$, which serves as the foundation for more specialized projects and packages. In this article, we summarize key features in the core package as of the recent major release, version 5.0, and provide major updates for the Project. We then discuss supporting a broader ecosystem of interoperable packages, including connections with several astronomical observatories and missions. We also revisit the future outlook of the Astropy Project and the current status of Learn Astropy. We conclude by raising and discussing the current and future challenges facing the Project.

astro-ph.IM

ESASky v.2.0: all the skies in your browser

With the goal of simplifying the access to science data to scientists and citizens, ESA recently released ESASky (http://sky.esa.int), a new open-science easy-to-use portal with the science-ready Astronomy data from ESA and other major data providers. In this presentation, we announced version 2.0 of the application, which includes access to all science-ready images, catalogues and spectra, a feature to help planning of future JWST observations, the possibility to search for data of all (targeted and serendipitously observed) Solar System Objects in Astronomy images, a first support to mobile devices and several other smaller usability features. We also discussed the future evolution of the portal and the lessons learnt from the 1+ year of operations from the point of view of access, visualization and manipulation of big datasets (all sky maps, also called HiPS) and large catalogues (like e.g. the Gaia DR1 catalogues or the Hubble Source Catalogue) and the design and validation principles for the development of friendly GUIs for thin layer web clients aimed at scientists.

astro-ph.IM

The ESA Gaia Archive: Data Release 1

ESA Gaia mission is producing the more accurate source catalogue in astronomy up to now. That represents a challenge on the archiving area to make accessible this information to the astronomers in an efficient way. Also, new astronomical missions have reinforced the change on the development of archives. Archives, as simple applications to access the data are being evolving into complex data center structures where computing power services are available for users and data mining tools are integrated into the server side. In the case of astronomy science that involves the use of big catalogues, as in Gaia (or Euclid to come), the common ways to work on the data need to be changed to a new paradigm "move code close to the data", what implies that data mining functionalities are becoming a must to allow the science exploitation. To enable these capabilities, a TAP+ interface, crossmatch capabilities, full catalogue histograms, serialisation of intermediate results in cloud resources like VOSpace, etc have been implemented for the Gaia DR1, to enable the exploitation of these science resources by the community without the bottlenecks on the connection bandwidth. We present the architecture, infrastructure and tools already available in the Gaia Archive Data Release 1 (http://archives.esac.esa.int/gaia/) and we describe capabilities and infrastructure.

astro-ph.IM

The Gaia Archive

The Archive is the main Gaia data distribution hub. The contents of DR1 are briefly reviewed and the data structures discussed. The system architecture, based on Virtual Observatory standards, is also presented, together with the extensions that allow e.g. authenticated access, persistent uploads and table sharing. Finally some usage examples are provided.

astro-ph.IM

Visualisation of Multi-mission Astronomical Data with ESASky

ESASky is a science-driven discovery portal to explore the multi-wavelength sky and visualise and access multiple astronomical archive holdings. The tool is a web application that requires no prior knowledge of any of the missions involved and gives users world-wide simplified access to the highest-level science data products from multiple astronomical space-based astronomy missions plus a number of ESA source catalogues. The first public release of ESASky features interfaces for the visualisation of the sky in multiple wavelengths, the visualisation of query results summaries, and the visualisation of observations and catalogue sources for single and multiple targets. This paper describes these features within ESASky, developed to address use cases from the scientific community. The decisions regarding the visualisation of large amounts of data and the technologies used were made in order to maximise the responsiveness of the application and to keep the tool as useful and intuitive as possible.

astro-ph.IM

ESASky: The whole of space Astronomy at your fingertips

ESASky is a new science-driven discovery portal for all ESA astronomical missions that gives users worldwide a simplified access to high-level science-ready products from ESA and other data providers. The tool features a sky exploration interface and a single/multiple target interface, and it requires no prior knowledge of specific details of each mission. Users can explore the sky in multiple wavelengths, quickly see the data available for their targets, and retrieve the relevant products, with just a few clicks. The first version of the tool, released in May 2016, provides access to imaging data and a number of catalogues. Future releases will enable retrieval of spectroscopic data and will incorporate futures to allow time-domain exploration and the study of Solar System objects.

astro-ph.IM

ESA Sky: a new Astronomy Multi-Mission Interface

We present a science-driven discovery portal for all the ESA Astronomy Missions called ESA Sky that allow users to explore the multi-wavelength sky and to seamlessly retrieve science-ready data in all ESA Astronomy mission archives from a web application without prior-knowledge of any of the missions. The first public beta of the service has been released, currently featuring an interface for exploration of the multi-wavelength sky and for single and/or multiple target searches of science-ready imaging data and catalogues. Future releases will enable retrieval of spectra and will have special time-domain exploration features. From a technical point of view, the system offers progressive multi-resolution all-sky projections of full mission datasets using a new generation of HEALPix projections called HiPS, developed at the CDS; detailed geometrical footprints to connect the all-sky mosaics to individual observations; and direct access to science-ready data at the underlying mission-specific science archives.

astro-ph.IM