arXiv subjects

Michael Bader

Publications and source records attributed to Michael Bader.

At least 19 recordsLinked to original sources

arXiv2025

Mixed-Precision in High-Order Methods: the Impact of Floating-Point Precision on the ADER-DG Algorithm

We present a mixed-precision implementation of the high-order discontinuous Galerkin method with ADER time stepping (ADER-DG) for solving hyperbolic systems of partial differential equations (PDEs) in the hyperbolic PDE engine ExaHyPE. The implementation provides a simple API extension for specifying the numerical precision for individual kernels, and thus allows for testing the effect of low and mixed precision on the accuracy of the solution. To showcase this, we study the impact of precision on the overall convergence order and actual accuracy of the method as achieved for four common hyperbolic PDE systems and five relevant scenarios that feature an analytic solution. For all scenarios, we also assess how sensitive each kernel of the ADER-DG algorithm is to using double, single or even half precision. This addresses the question where thoughtful adoption of mixed precision can mitigate hurtful effects of low precision on the overall simulation.

math.NA↗

arXiv2022

Numerical simulations of seismo-acoustic nuisance patterns from an induced M1.8 earthquake in the Helsinki, southern Finland, metropolitan area

Seismic waves can couple with the atmosphere and generate sound waves. The influence of faulting mechanisms on earthquake sound patterns provides opportunities for earthquake source characterization. Sound radiated from earthquakes can be perceived as disturbing which can negatively impact the social acceptance of geoengineering applications. Motivated by reports of felt and heard disturbances associated with the week-long stimulation of a 6 km-deep geothermal system in 2018 below the Otaniemi district of Espoo, Helsinki, we conduct fully coupled 3D numerical simulations of wave propagation in the solid Earth and the atmosphere. We assess the sensitivity of the ground shaking and audible noise distributions to the source geometry of the induced earthquakes. Utilizing recent computational advances and the open-source software SeisSol, we model seismo-acoustic frequencies up to 25 Hz, thereby reaching the lower limit of the audible sound frequency range. We present synthetic distributions of shaking and audible and discuss implications for better understanding seismic nuisances in metropolitan regions. In five 3D coupled elastic-acoustic scenario, we analyze the ground velocity and pressure levels of earthquake-generated seismic and acoustic waves. We show that S waves generate the strongest sound disturbance with sound pressure levels of up to 0.04 Pa. We use statistical analysis to compare our noise distributions with commonly used empirical relationships. We find that our 3D synthetic amplitudes are generally smaller than the empirical predictions and that the interaction of the source-specific radiation pattern and topography can lead to significant non-linear effects. Our study highlights the complexity and information content of spatially variable audible effects associated with small induced earthquakes on local scales.

physics.geo-ph↗

arXiv2021

Doubt and Redundancy Kill Soft Errors -- Towards Detection and Correction of Silent Data Corruption in Task-based Numerical Software

Resilient algorithms in high-performance computing are subject to rigorous non-functional constraints. Resiliency must not increase the runtime, memory footprint or I/O demands too significantly. We propose a task-based soft error detection scheme that relies on error criteria per task outcome. They formalise how ``dubious'' an outcome is, i.e. how likely it contains an error. Our whole simulation is replicated once, forming two teams of MPI ranks that share their task results. Thus, ideally each team handles only around half of the workload. If a task yields large error criteria values, i.e.~is dubious, we compute the task redundantly and compare the outcomes. Whenever they disagree, the task result with a lower error likeliness is accepted. We obtain a self-healing, resilient algorithm which can compensate silent floating-point errors without a significant performance, I/O or memory footprint penalty. Case studies however suggest that a careful, domain-specific tailoring of the error criteria remains essential.

cs.SE↗

arXiv2021

An Efficient ADER-DG Local Time Stepping Scheme for 3D HPC Simulation of Seismic Waves in Poroelastic Media

Many applications from geosciences require simulations of seismic waves in porous media. Biot's theory of poroelasticity describes the coupling between solid and fluid phases and introduces a stiff source term, thereby increasing computational cost and motivating efficient methods utilising High-Performance Computing. We present a novel realisation of the discontinuous Galerkin scheme with Arbitrary DERivative time stepping (ADER-DG) that copes with stiff source terms. To integrate this source term with a reasonable time step size, we use an element-local space-time predictor, which needs to solve medium-sized linear systems - with 1000 to 10000 unknowns - in each element update (i.e., billions of times). We present a novel block-wise back-substitution algorithm for solving these systems efficiently. In comparison to LU decomposition, we reduce the number of floating-point operations by a factor of up to 25. The block-wise back-substitution is mapped to a sequence of small matrix-matrix multiplications, for which code generators are available to generate highly optimised code. We verify the new solver thoroughly in problems of increasing complexity. We demonstrate high-order convergence for 3D problems. We verify the correct treatment of point sources, material interfaces and traction-free boundary conditions. In addition, we compare against a finite difference code for a newly defined layer over half-space problem. We find that extremely high accuracy is required to resolve the slow P-wave at a free surface, while solid particle velocities are not affected by coarser resolutions. By using a clustered local time stepping scheme, we reduce time to solution by a factor of 6 to 10 compared to global time stepping. We conclude our study with a scaling and performance analysis, demonstrating our implementation's efficiency and its potential for extreme-scale simulations.

cs.DC↗

arXiv2021

High Performance Uncertainty Quantification with Parallelized Multilevel Markov Chain Monte Carlo

Numerical models of complex real-world phenomena often necessitate High Performance Computing (HPC). Uncertainties increase problem dimensionality further and pose even greater challenges. We present a parallelization strategy for multilevel Markov chain Monte Carlo, a state-of-the-art, algorithmically scalable Uncertainty Quantification (UQ) algorithm for Bayesian inverse problems, and a new software framework allowing for large-scale parallelism across forward model evaluations and the UQ algorithms themselves. The main scalability challenge presents itself in the form of strong data dependencies introduced by the MLMCMC method, prohibiting trivial parallelization. Our software is released as part of the modular and open-source MIT UQ Library (MUQ), and can easily be coupled with arbitrary user codes. We demonstrate it using the DUNE and the ExaHyPE Engine. The latter provides a realistic, large-scale tsunami model in which identify the source of a tsunami from buoy-elevation data.

cs.MS↗

arXiv2021

3D Acoustic-Elastic Coupling with Gravity: The Dynamics of the 2018 Palu, Sulawesi Earthquake and Tsunami

We present a highly scalable 3D fully-coupled Earth & ocean model of earthquake rupture and tsunami generation. We model seismic, acoustic and surface gravity wave propagation in elastic (Earth) and acoustic (ocean) materials sourced by physics-based non-linear earthquake dynamic rupture. Complicated geometries, including high-resolution bathymetry, coastlines and segmented earthquake faults are discretized by adaptive unstructured tetrahedral meshes. A Discontinuous Galerkin discretization with ADER local time-stepping (ADER-DG) yields petascale computational efficiency and high-order accuracy in time and space. We compare the 3D fully-coupled approach to a benchmark problem for 3D-2D linked models that use 2D shallow-water modeling. We present a large-scale fully-coupled model of the 2018 Sulawesi events that links the dynamics from supershear earthquake faulting to elastic and acoustic waves in Earth and ocean to tsunami gravity wave propagation in the narrow Palu Bay. And we demonstrate scalability and performance of the MPI+OpenMP parallelization on three petascale supercomputers.

physics.comp-ph↗

arXiv2020

TeaMPI -- Replication-based Resilience without the (Performance) Pain

In an era where we can not afford to checkpoint frequently, replication is a generic way forward to construct numerical simulations that can continue to run even if hardware parts fail. Yet, replication often is not employed on larger scales, as na\"ively mirroring a computation once effectively halves the machine size, and as keeping replicated simulations consistent with each other is not trivial. We demonstrate for the ExaHyPE engine -- a task-based solver for hyperbolic equation systems -- that it is possible to realise resiliency without major code changes on the user side, while we introduce a novel algorithmic idea where replication reduces the time-to-solution. The redundant CPU cycles are not burned "for nothing". Our work employs a weakly consistent data model where replicas run independently yet inform each other through heartbeat messages whether they are still up and running. Our key performance idea is to let the tasks of the replicated simulations share some of their outcomes, while we shuffle the actual task execution order per replica. This way, replicated ranks can skip some local computations and automatically start to synchronise with each other. Our experiments with a production-level seismic wave-equation solver provide evidence that this novel concept has the potential to make replication affordable for large-scale simulations in high-performance computing.

cs.DC↗

arXiv2020

An Environment for Sustainable Research Software in Germany and Beyond: Current State, Open Challenges, and Call for Action

Research software has become a central asset in academic research. It optimizes existing and enables new research methods, implements and embeds research knowledge, and constitutes an essential research product in itself. Research software must be sustainable in order to understand, replicate, reproduce, and build upon existing research or conduct new research effectively. In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future. Research software therefore requires an environment that supports sustainability. Hence, a change is needed in the way research software development and maintenance are currently motivated, incentivized, funded, structurally and infrastructurally supported, and legally treated. Failing to do so will threaten the quality and validity of research. In this paper, we identify challenges for research software sustainability in Germany and beyond, in terms of motivation, selection, research software engineering personnel, funding, infrastructure, and legal aspects. Besides researchers, we specifically address political and academic decision-makers to increase awareness of the importance and needs of sustainable research software practices. In particular, we recommend strategies and measures to create an environment for sustainable research software, with the ultimate goal to ensure that software-driven research is valid, reproducible and sustainable, and that software is recognized as a first class citizen in research. This paper is the outcome of two workshops run in Germany in 2019, at deRSE19 - the first International Conference of Research Software Engineers in Germany - and a dedicated DFG-supported follow-up workshop in Berlin.

cs.GL↗

arXiv2020

Vectorization and Minimization of Memory Footprint for Linear High-Order Discontinuous Galerkin Schemes

We present a sequence of optimizations to the performance-critical compute kernels of the high-order discontinuous Galerkin solver of the hyperbolic PDE engine ExaHyPE -- successively tackling bottlenecks due to SIMD operations, cache hierarchies and restrictions in the software design. Starting from a generic scalar implementation of the numerical scheme, our first optimized variant applies state-of-the-art optimization techniques by vectorizing loops, improving the data layout and using Loop-over-GEMM to perform tensor contractions via highly optimized matrix multiplication functions provided by the LIBXSMM library. We show that memory stalls due to a memory footprint exceeding our L2 cache size hindered the vectorization gains. We therefore introduce a new kernel that applies a sum factorization approach to reduce the kernel's memory footprint and improve its cache locality. With the L2 cache bottleneck removed, we were able to exploit additional vectorization opportunities, by introducing a hybrid Array-of-Structure-of-Array data layout that solves the data layout conflict between matrix multiplications kernels and the point-wise functions to implement PDE-specific terms. With this last kernel, evaluated in a benchmark simulation at high polynomial order, only 2\% of the floating point operations are still performed using scalar instructions and 22.5\% of the available performance is achieved.

cs.MS↗

arXiv2019

Role-Oriented Code Generation in an Engine for Solving Hyperbolic PDE Systems

The development of a high performance PDE solver requires the combined expertise of interdisciplinary teams with respect to application domain, numerical scheme and low-level optimization. In this paper, we present how the ExaHyPE engine facilitates the collaboration of such teams by isolating three roles: application, algorithms, and optimization expert. We thus support team members in letting them focus on their own area of expertise while integrating their contributions into an HPC production code. Inspired by web application development practices, ExaHyPE relies on two custom code generation modules, the Toolkit and the Kernel Generator, which follow a Model-View-Controller architectural pattern on top of the Jinja2 template engine library. Using Jinja2's templates to abstract the critical components of the engine and generated glue code, we isolate the application development from the engine. The template language also allows us to define and use custom template macros that isolate low-level optimizations from the numerical scheme described in the templates. We present three use cases, each focusing on one of our user roles, showcasing how the design of the code generation modules allows to easily expand the solver schemes to support novel demands from applications, to add optimized algorithmic schemes (with reduced memory footprint, e.g.), or provide improved low-level SIMD vectorization support.

cs.MS↗

arXiv2019

A stable discontinuous Galerkin method for the perfectly matched layer for elastodynamics in first order form

We present a stable discontinuous Galerkin (DG) method with a perfectly matched layer (PML) for three and two space dimensional linear elastodynamics, in velocity-stress formulation, subject to well-posed linear boundary conditions. First, we consider the elastodynamics equation, in a cuboidal domain, and derive an unsplit PML truncating the domain using complex coordinate stretching. Leveraging the hyperbolic structure of the underlying system, we construct continuous energy estimates, in the time domain for the elastic wave equation, and in the Laplace space for a sequence of PML model problems, with variations in one, two and three space dimensions, respectively. They correspond to PMLs normal to boundary faces, along edges and in corners. Second, we develop a DG numerical method for the linear elastodynamics equation using physically motivated numerical flux and penalty parameters, which are compatible with all well-posed, internal and external, boundary conditions. When the PML damping vanishes, by construction, our choice of penalty parameters yield an upwind scheme and a discrete energy estimate analogous to the continuous energy estimate. Third, to ensure numerical stability of the discretization when PML damping is present, it is necessary to extend the numerical DG fluxes, and the numerical inter-element and boundary procedures, to the PML auxiliary differential equations. This is crucial for deriving discrete energy estimates analogous to the continuous energy estimates. By combining the DG spatial approximation with the high order ADER time stepping scheme and the accuracy of the PML we obtain an arbitrarily accurate wave propagation solver in the time domain. Numerical experiments are presented in two and three space dimensions corroborating the theoretical results.

math.NA↗

arXiv2019

Lightweight Task Offloading Exploiting MPI Wait Times for Parallel Adaptive Mesh Refinement

Balancing the workload of sophisticated simulations is inherently difficult, since we have to balance both computational workload and memory footprint over meshes that can change any time or yield unpredictable cost per mesh entity, while modern supercomputers and their interconnects start to exhibit fluctuating performance. We propose a novel lightweight balancing technique for MPI+X to accompany traditional, prediction-based load balancing. It is a reactive diffusion approach that uses online measurements of MPI idle time to migrate tasks temporarily from overloaded to underemployed ranks. Tasks are deployed to ranks which otherwise would wait, processed with high priority, and made available to the overloaded ranks again. This migration is non-persistent. Our approach hijacks idle time to do meaningful work and is totally non-blocking, asynchronous and distributed without a global data view. Tests with a seismic simulation code developed in the ExaHyPE engine uncover the method's potential. We found speed-ups of up to 2-3 for ill-balanced scenarios without logical modifications of the code base and show that the strategy is capable to react quickly to temporarily changing workload or node performance.

cs.DC↗

arXiv2019

A stable discontinuous Galerkin method for linear elastodynamics in 3D geometrically complex media using physics based numerical fluxes

High order accurate and explicit time-stable solvers are well suited for hyperbolic wave propagation problems. As a result of the complexities of real geometries, internal interfaces and nonlinear boundary and interface conditions, discontinuities and sharp wave fronts may become fundamental features of the solution. Thus, geometrically flexible and adaptive numerical algorithms are critical for high fidelity and efficient simulations of wave phenomena in many applications. Adaptive curvilinear meshes hold promise to minimise the effort to represent complicated geometries or heterogeneous material data avoiding the bottleneck of feature-preserving meshing. To enable the design of stable DG methods on three space dimensional (3D) curvilinear elements we construct a structure preserving anti-symmetric coordinate transformation motivated by the underlying physics. Using a physics-based numerical penalty-flux, we develop a 3D provably energy-stable discontinuous Galerkin finite element approximation of the elastic wave equation in geometrically complex and heterogenous media. By construction, our numerical flux is upwind and yields a discrete energy estimate analogous to the continuous energy estimate. The ability to treat conforming and non-conforming curvilinear elements allows for flexible adaptive mesh refinement strategies. The numerical scheme has been implemented in ExaHyPE, a simulation engine for parallel dynamically adaptive simulations of wave problems on adaptive Cartesian meshes. We present 3D numerical experiments of wave propagation in heterogeneous isotropic and anisotropic elastic solids demonstrating stability and high order accuracy. We demonstrate the potential of our approach for computational seismology in a regional wave propagation scenario in a geologically constrained 3D model including the geometrically complex free-surface topography of Mount Zugspitze, Germany.

math.NA↗

arXiv2019

ExaHyPE: An Engine for Parallel Dynamically Adaptive Simulations of Wave Problems

ExaHyPE ("An Exascale Hyperbolic PDE Engine") is a software engine for solving systems of first-order hyperbolic partial differential equations (PDEs). Hyperbolic PDEs are typically derived from the conservation laws of physics and are useful in a wide range of application areas. Applications powered by ExaHyPE can be run on a student's laptop, but are also able to exploit thousands of processor cores on state-of-the-art supercomputers. The engine is able to dynamically increase the accuracy of the simulation using adaptive mesh refinement where required. Due to the robustness and shock capturing abilities of ExaHyPE's numerical methods, users of the engine can simulate linear and non-linear hyperbolic PDEs with very high accuracy. Users can tailor the engine to their particular PDE by specifying evolved quantities, fluxes, and source terms. A complete simulation code for a new hyperbolic PDE can often be realised within a few hours - a task that, traditionally, can take weeks, months, often years for researchers starting from scratch. In this paper, we showcase ExaHyPE's workflow and capabilities through real-world scenarios from our two main application areas: seismology and astrophysics.

cs.MS↗

arXiv2019

A High-Order Discontinuous Galerkin Solver with Dynamic Adaptive Mesh Refinement to Simulate Cloud Formation Processes

We present a high-order discontinuous Galerkin (DG) solver of the compressible Navier-Stokes equations for cloud formation processes. The scheme exploits an underlying parallelized implementation of the ADER-DG method with dynamic adaptive mesh refinement. We improve our method by a PDE-independent general refinement criterion, based on the local total variation of the numerical solution. While established methods use numerics tailored towards the specific simulation, our scheme works scenario independent. Our generic scheme shows competitive results for both classical CFD and stratified scenarios. We focus on two dimensional simulations of two bubble convection scenarios over a background atmosphere. The largest simulation here uses order 6 and 6561 cells which were reduced to 1953 cells by our refinement criterion.

physics.comp-ph↗

arXiv2019

Yet Another Tensor Toolbox for discontinuous Galerkin methods and other applications

The numerical solution of partial differential equations is at the heart of many grand challenges in supercomputing. Solvers based on high-order discontinuous Galerkin (DG) discretisation have been shown to scale on large supercomputers with excellent performance and efficiency, if the implementation exploits all levels of parallelism and is tailored to the specific architecture. However, every year new supercomputers emerge and the list of hardware-specific considerations grows, simultaneously with the list of desired features in a DG code. Thus we believe that a sustainable DG code needs an abstraction layer to implement the numerical scheme in a suitable language. We explore the possibility to abstract the numerical scheme as small tensor operations, describe them in a domain-specific language (DSL) resembling the Einstein notation, and to map them to existing code generators which generate small matrix matrix multiplication routines. The compiler for our DSL implements classic optimisations that are used for large tensor contractions, and we present novel optimisation techniques such as equivalent sparsity patterns and optimal index permutations for temporary tensors. Our application examples, which include the earthquake simulation software SeisSol, show that the generated kernels achieve over 50 % peak performance while the DSL considerably simplifies the implementation.

cs.MS↗

arXiv2018

Exploiting the Space Filling Curve Ordering of Particles in the Neighbour Search of Gadget3

Gadget3 is nowadays one of the most frequently used high performing parallel codes for cosmological hydrodynamical simulations. Recent analyses have shown t\ hat the Neighbour Search process of Gadget3 is one of the most time-consuming parts. Thus, a considerable speedup can be expected from improvements of the u\ nderlying algorithms. In this work we propose a novel approach for speeding up the Neighbour Search which takes advantage of the space-filling-curve particle ordering. Instead of performing Neighbour Search for all particles individually, nearby active particles can be grouped and one single Neighbour Search can be performed to obta\ in a common superset of neighbours. Thus, with this approach we reduce the number of searches. On the other hand, tree walks are performed within a larger searching radius. There is an optimal size of grouping that maximize the speedup, which we found by numerical experiments. We tested the algorithm within the boxes of the Magneticum project. As a result we obtained a speedup of $1.65$ in the Density and of $1.30$ in the Hydrodynamics computation, respectively, and a total speedup of $1.34.$

astro-ph.IM↗

arXiv2018

Influence of A-Posteriori Subcell Limiting on Fault Frequency in Higher-Order DG Schemes

Soft error rates are increasing as modern architectures require increasingly small features at low voltages. Due to the large number of components used in HPC architectures, these are particularly vulnerable to soft errors. Hence, when designing applications that run for long time periods on large machines, algorithmic resilience must be taken into account. In this paper we analyse the inherent resiliency of a-posteriori limiting procedures in the context of the explicit ADER DG hyperbolic PDE solver ExaHyPE. The a-posteriori limiter checks element-local high-order DG solutions for physical admissibility, and can thus be expected to also detect hardware-induced errors. Algorithmically, it can be interpreted as element-local checkpointing and restarting of the solver with a more robust finite volume scheme on a fine subgrid. We show that the limiter indeed increases the resilience of the DG algorithm, detecting and correcting particularly those faults which would otherwise lead to a fatal failure.

physics.comp-ph↗