Searcharxiv⌕ Search

arXiv subjects

Deborah Bard

Publications and source records attributed to Deborah Bard.

14 recordsLinked to original sources
arXiv2024

Workflows Community Summit 2024: Future Trends and Challenges in Scientific Workflows

The Workflows Community Summit gathered 111 participants from 18 countries to discuss emerging trends and challenges in scientific workflows, focusing on six key areas: time-sensitive workflows, AI-HPC convergence, multi-facility workflows, heterogeneous HPC environments, user experience, and FAIR computational workflows. The integration of AI and exascale computing has revolutionized scientific workflows, enabling higher-fidelity models and complex, time-sensitive processes, while introducing challenges in managing heterogeneous environments and multi-facility data dependencies. The rise of large language models is driving computational demands to zettaflop scales, necessitating modular, adaptable systems and cloud-service models to optimize resource utilization and ensure reproducibility. Multi-facility workflows present challenges in data movement, curation, and overcoming institutional silos, while diverse hardware architectures require integrating workflow considerations into early system design and developing standardized resource management tools. The summit emphasized improving user experience in workflow systems and ensuring FAIR workflows to enhance collaboration and accelerate scientific discovery. Key recommendations include developing standardized metrics for time-sensitive workflows, creating frameworks for cloud-HPC integration, implementing distributed-by-design workflow modeling, establishing multi-facility authentication protocols, and accelerating AI integration in HPC workflow management. The summit also called for comprehensive workflow benchmarks, workflow-specific UX principles, and a FAIR workflow maturity model, highlighting the need for continued collaboration in addressing the complex challenges posed by the convergence of AI, HPC, and multi-facility research environments.

cs.DC↗
arXiv2022

The LBNL Superfacility Project Report

The Superfacility model is designed to leverage HPC for experimental science. It is more than simply a model of connected experiment, network, and HPC facilities; it encompasses the full ecosystem of infrastructure, software, tools, and expertise needed to make connected facilities easy to use. The three-year Lawrence Berkeley National Laboratory (LBNL) Superfacility project was initiated in 2019 to coordinate work being performed at LBNL to support this model, and to provide a coherent and comprehensive set of science requirements to drive existing and new work. A key component of the project was the in-depth engagements with eight science teams that represent challenging use cases across the DOE Office of Science. By the close of the project, we met our project goal by enabling our science application engagements to demonstrate automated pipelines that analyze data from remote facilities at large scale, without routine human intervention. In several cases, we have gone beyond demonstrations and now provide production-level services. To achieve this goal, the Superfacility team developed tools, infrastructure, and policies for near-real-time computing support, dynamic high-performance networking, data management and movement tools, API-driven automation, HPC-scale notebooks via Jupyter, authentication using Federated Identity and container-based edge services supported. The lessons we learned during this project provide a valuable model for future large, complex, cross-disciplinary collaborations. There is a pressing need for a coherent computing infrastructure across national facilities, and LBNL's Superfacility project is a unique model for success in tackling the challenges that will be faced in hardware, software, policies, and services across multiple science domains.

cs.DC↗
arXiv2022

Climate of the Field: Snowmass 2021

How are formal policies put in place to create an inclusive, equitable, safe environment? How do these differ between different communities of practice (institutions, labs, collaborations, working groups)? What policies towards a more equitable community are working? For those that aren't working, what external support is needed in order to make them more effective? We present a discussion of the current climate of the field in high energy particle physics and astrophysics (HEPA), as well as current efforts toward making the community a more diverse, inclusive, and equitable environment. We also present issues facing both institutions and HEPA collaborations, with a set of interviews with a selection of HEPA collaboration DEI leaders. We encourage the HEPA community and the institutions & agencies that support it to think critically about the prioritization of people in HEPA over the coming decade, and what resources and policies need to be in place in order to protect and elevate minoritized populations within the HEPA community.

physics.soc-ph↗
arXiv2022

Software and Computing for Small HEP Experiments

This white paper briefly summarized key conclusions of the recent US Community Study on the Future of Particle Physics (Snowmass 2021) workshop on Software and Computing for Small High Energy Physics Experiments.

hep-ex↗
arXiv2022

Data Preservation for Cosmology

We describe the needs and opportunities for preserving cosmology datasets and simulations, and facilitating their joint analysis beyond the lifetime of individual projects. We recommend that DOE fund a new cosmology data archive center to coordinate this work across the multiple DOE computing facilities.

astro-ph.CO↗
arXiv2021

Real-Time XFEL Data Analysis at SLAC and NERSC: a Trial Run of Nascent Exascale Experimental Data Analysis

X-ray scattering experiments using Free Electron Lasers (XFELs) are a powerful tool to determine the molecular structure and function of unknown samples (such as COVID-19 viral proteins). XFEL experiments are a challenge to computing in two ways: i) due to the high cost of running XFELs, a fast turnaround time from data acquisition to data analysis is essential to make informed decisions on experimental protocols; ii) data collection rates are growing exponentially, requiring new scalable algorithms. Here we report our experiences analyzing data from two experiments at the Linac Coherent Light Source (LCLS) during September 2020. Raw data were analyzed on NERSC's Cori XC40 system, using the Superfacility paradigm: our workflow automatically moves raw data between LCLS and NERSC, where it is analyzed using the software package CCTBX. We achieved real time data analysis with a turnaround time from data acquisition to full molecular reconstruction in as little as 10 min -- sufficient time for the experiment's operators to make informed decisions. By hosting the data analysis on Cori, and by automating LCLS-NERSC interoperability, we achieved a data analysis rate which matches the data acquisition rate. Completing data analysis with 10 mins is a first for XFEL experiments and an important milestone if we are to keep up with data collection trends.

cs.DC↗
arXiv2019

CosmoGAN: creating high-fidelity weak lensing convergence maps using Generative Adversarial Networks

Inferring model parameters from experimental data is a grand challenge in many sciences, including cosmology. This often relies critically on high fidelity numerical simulations, which are prohibitively computationally expensive. The application of deep learning techniques to generative modeling is renewing interest in using high dimensional density estimators as computationally inexpensive emulators of fully-fledged simulations. These generative models have the potential to make a dramatic shift in the field of scientific simulations, but for that shift to happen we need to study the performance of such generators in the precision regime needed for science applications. To this end, in this work we apply Generative Adversarial Networks to the problem of generating weak lensing convergence maps. We show that our generator network produces maps that are described by, with high statistical confidence, the same summary statistics as the fully simulated maps.

astro-ph.IM↗
arXiv2018

CosmoFlow: Using Deep Learning to Learn the Universe at Scale

Deep learning is a promising tool to determine the physical model that describes our universe. To handle the considerable computational cost of this problem, we present CosmoFlow: a highly scalable deep learning application built on top of the TensorFlow framework. CosmoFlow uses efficient implementations of 3D convolution and pooling primitives, together with improvements in threading for many element-wise operations, to improve training performance on Intel(C) Xeon Phi(TM) processors. We also utilize the Cray PE Machine Learning Plugin for efficient scaling to multiple nodes. We demonstrate fully synchronous data-parallel training on 8192 nodes of Cori with 77% parallel efficiency, achieving 3.5 Pflop/s sustained performance. To our knowledge, this is the first large-scale science application of the TensorFlow framework at supercomputer scale with fully-synchronous training. These enhancements enable us to process large 3D dark matter distribution and predict the cosmological parameters $Ω_M$, $σ_8$ and n$_s$ with unprecedented accuracy.

astro-ph.CO↗
arXiv2017

Galactos: Computing the Anisotropic 3-Point Correlation Function for 2 Billion Galaxies

The nature of dark energy and the complete theory of gravity are two central questions currently facing cosmology. A vital tool for addressing them is the 3-point correlation function (3PCF), which probes deviations from a spatially random distribution of galaxies. However, the 3PCF's formidable computational expense has prevented its application to astronomical surveys comprising millions to billions of galaxies. We present Galactos, a high-performance implementation of a novel, O(N^2) algorithm that uses a load-balanced k-d tree and spherical harmonic expansions to compute the anisotropic 3PCF. Our implementation is optimized for the Intel Xeon Phi architecture, exploiting SIMD parallelism, instruction and thread concurrency, and significant L1 and L2 cache reuse, reaching 39% of peak performance on a single node. Galactos scales to the full Cori system, achieving 9.8PF (peak) and 5.06PF (sustained) across 9636 nodes, making the 3PCF easily computable for all galaxies in the observable universe.

astro-ph.CO↗
arXiv2014

GREAT3 results I: systematic errors in shear estimation and the impact of real galaxy morphology

We present first results from the third GRavitational lEnsing Accuracy Testing (GREAT3) challenge, the third in a sequence of challenges for testing methods of inferring weak gravitational lensing shear distortions from simulated galaxy images. GREAT3 was divided into experiments to test three specific questions, and included simulated space- and ground-based data with constant or cosmologically-varying shear fields. The simplest (control) experiment included parametric galaxies with a realistic distribution of signal-to-noise, size, and ellipticity, and a complex point spread function (PSF). The other experiments tested the additional impact of realistic galaxy morphology, multiple exposure imaging, and the uncertainty about a spatially-varying PSF; the last two questions will be explored in Paper II. The 24 participating teams competed to estimate lensing shears to within systematic error tolerances for upcoming Stage-IV dark energy surveys, making 1525 submissions overall. GREAT3 saw considerable variety and innovation in the types of methods applied. Several teams now meet or exceed the targets in many of the tests conducted (to within the statistical errors). We conclude that the presence of realistic galaxy morphology in simulations changes shear calibration biases by $\sim 1$ per cent for a wide range of methods. Other effects such as truncation biases due to finite galaxy postage stamps, and the impact of galaxy type as measured by the S\'{e}rsic index, are quantified for the first time. Our results generalize previous studies regarding sensitivities to galaxy size and signal-to-noise, and to PSF properties such as seeing and defocus. Almost all methods' results support the simple model in which additive shear biases depend linearly on PSF ellipticity.

astro-ph.CO↗
arXiv2014

Masked areas in shear peak statistics: a forward modeling approach

The statistics of shear peaks have been shown to provide valuable cosmological information beyond the power spectrum, and will be an important constraint of models of cosmology with the large survey areas provided by forthcoming astronomical surveys. Surveys include masked areas due to bright stars, bad pixels etc, which must be accounted for in producing constraints on cosmology from shear maps. We advocate a forward-modeling approach, where the impact of masking (and other survey artifacts) are accounted for in the theoretical prediction of cosmological parameters, rather than removed from survey data. We use masks based on the Deep Lens Survey, and explore the impact of up to 37% of the survey area being masked on LSST and DES-scale surveys. By reconstructing maps of aperture mass, the masking effect is smoothed out, resulting in up to 14% smaller statistical uncertainties compared to simply reducing the survey area by the masked area. We show that, even in the presence of large survey masks, the bias in cosmological parameter estimation produced in the forward-modeling process is ~1%, dominated by bias caused by limited simulation volume. We also explore how this potential bias scales with survey area and find that small survey areas are more significantly impacted by the differences in cosmological structure in the data and simulated volumes, due to cosmic variance.

astro-ph.CO↗
arXiv2013

Facilities for Dark Energy Investigations

The discovery of cosmic acceleration has inspired ambitious experimental and observational efforts to understand its origin. Many of these take the form of large astronomical surveys, sometimes using new, special-purpose instrumentation, and in some cases entirely new facilities. This Report summarizes some of the major ongoing and planned dark energy experiments, focusing on those in which the U.S. community has a leading or significant supporting role. It provides background information for other Reports from the "Dark Energy and CMB" working group of the APS Division of Particles and Fields long-term planning exercise. We provide 1-2 page summaries of the SDSS-III Baryon Oscillation Spectroscopic Survey (BOSS) and its SDSS-IV successor eBOSS, the Dark Energy Survey (DES), the Subaru Hyper-Suprime Camera (HSC) and Prime Focus Spectrograph (PFS), the Hobby-Eberley Telescope Dark Energy Experiment (HEDTEX), the Dark Energy Spectroscopic Instrument (DESI), the Large Synpotic Survey Telescope (LSST), and the Euclid and AFTA-WFIRST (Wide Field Infrared Survey Telescope) space missions. Over the next one to two decades, these experiments seek to improve the precision of measurements of the cosmic expansion history and cosmic structure growth by factors of 10-100, while simultaneously increasing their redshift range and tightening control and cross-checks of sytematic effects. These measurements will dramatically improve the empirical constraints on theoretical explanations of cosmic acceleration and could lead to breakthroughs in our understanding of the nature of gravity and spacetime or the forms of energy present in the universe.

astro-ph.CO↗
arXiv2012

Cosmological Calculations on the GPU

Cosmological measurements require the calculation of nontrivial quantities over large datasets. The next generation of survey telescopes (such as DES, PanSTARRS, and LSST) will yield measurements of billions of galaxies. The scale of these datasets, and the nature of the calculations involved, make cosmological calculations ideal models for implementation on graphics processing units (GPUs). We consider two cosmological calculations, the two-point angular correlation function and the aperture mass statistic, and aim to improve the calculation time by constructing code for calculating them on the GPU. Using CUDA, we implement the two algorithms on the GPU and compare the calculation speeds to comparable code run on the CPU. We obtain a code speed-up of between 10 - 180x faster, compared to performing the same calculation on the CPU. The code has been made publicly available.

astro-ph.IM↗