arXiv subjects

Tobias Weinzierl

Publications and source records attributed to Tobias Weinzierl.

35 records · Page 2Linked to original sources

arXiv2021

The maximum discrete surface-to-volume ratio of space-filling curve partitions

Space-filling curves (SFCs) are used in high performance computing to distribute a computational domain or its mesh, respectively, amongst different compute units, i.e.~cores or nodes or accelerators. The part of the domain allocated to each compute unit is called a partition. Besides the balancing of the work, the communication cost to exchange data between units determines the quality of a chosen partition. This cost can be approximated by the surface-to-volume ratio of partitions: the volume represents the amount of local work, while the surface represents the amount of data to be transmitted. Empirical evidence suggests that space-filling curves yield advantageous surface-to-volume ratios. Formal proofs are available only for regular grids. We investigate the surface-to-volume ratio of space-filling curve partitions for adaptive grids and derive the maximum surface-to-volume ratio as a function of the number of cells in the partition. In order to prove our main theorem, we construct a new framework for the study of adaptive grids, notably introducing the concepts of a shape and of classified partitions. The new methodological framework yields insight about the SFC-induced partition character even if the grids refine rather aggressively in localised areas: it quantifies the obtained surface-to-volume ratio. This framework thus has the potential to guide the design of better load balancing algorithms on the long term.

cs.CE↗

arXiv2020

Stabilised Asynchronous Fast Adaptive Composite Multigrid using Additive Damping

Multigrid solvers face multiple challenges on parallel computers. Two fundamental ones read as follows: Multiplicative solvers issue coarse grid solves which exhibit low concurrency and many multigrid implementations suffer from an expensive coarse grid identification phase plus adaptive mesh refinement overhead. We propose a new additive multigrid variant for spacetrees, i.e. meshes as they are constructed from octrees and quadtrees: It is an additive scheme, i.e. all multigrid resolution levels are updated concurrently. This ensures a high concurrency level, while the transfer operators between the mesh levels can still be constructed algebraically. The novel flavour of the additive scheme is an augmentation of the solver with an additive, auxiliary damping parameter per grid level per vertex that is in turn constructed through the next coarser level---an idea which utilises smoothed aggregation principles or the motivation behind AFACx: Per level, we solve an additional equation whose purpose is to damp too aggressive solution updates per vertex which would otherwise, in combination with all the other levels, yield an overcorrection and, eventually, oscillations. This additional equation is constructed additively as well, i.e. is once more solved concurrently to all other equations. This yields improved stability, closer to what is seen with multiplicative schemes, while pipelining techniques help us to write down the additive solver with single-touch semantics for dynamically adaptive meshes.

math.NA↗

arXiv2020

TeaMPI -- Replication-based Resilience without the (Performance) Pain

In an era where we can not afford to checkpoint frequently, replication is a generic way forward to construct numerical simulations that can continue to run even if hardware parts fail. Yet, replication often is not employed on larger scales, as naïvely mirroring a computation once effectively halves the machine size, and as keeping replicated simulations consistent with each other is not trivial. We demonstrate for the ExaHyPE engine -- a task-based solver for hyperbolic equation systems -- that it is possible to realise resiliency without major code changes on the user side, while we introduce a novel algorithmic idea where replication reduces the time-to-solution. The redundant CPU cycles are not burned "for nothing". Our work employs a weakly consistent data model where replicas run independently yet inform each other through heartbeat messages whether they are still up and running. Our key performance idea is to let the tasks of the replicated simulations share some of their outcomes, while we shuffle the actual task execution order per replica. This way, replicated ranks can skip some local computations and automatically start to synchronise with each other. Our experiments with a production-level seismic wave-equation solver provide evidence that this novel concept has the potential to make replication affordable for large-scale simulations in high-performance computing.

cs.DC↗

arXiv2020

ExaHyPE: An Engine for Parallel Dynamically Adaptive Simulations of Wave Problems

ExaHyPE ("An Exascale Hyperbolic PDE Engine") is a software engine for solving systems of first-order hyperbolic partial differential equations (PDEs). Hyperbolic PDEs are typically derived from the conservation laws of physics and are useful in a wide range of application areas. Applications powered by ExaHyPE can be run on a student's laptop, but are also able to exploit thousands of processor cores on state-of-the-art supercomputers. The engine is able to dynamically increase the accuracy of the simulation using adaptive mesh refinement where required. Due to the robustness and shock capturing abilities of ExaHyPE's numerical methods, users of the engine can simulate linear and non-linear hyperbolic PDEs with very high accuracy. Users can tailor the engine to their particular PDE by specifying evolved quantities, fluxes, and source terms. A complete simulation code for a new hyperbolic PDE can often be realised within a few hours - a task that, traditionally, can take weeks, months, often years for researchers starting from scratch. In this paper, we showcase ExaHyPE's workflow and capabilities through real-world scenarios from our two main application areas: seismology and astrophysics.

cs.MS↗

arXiv2020

Delayed approximate matrix assembly in multigrid with dynamic precisions

The accurate assembly of the system matrix is an important step in any code that solves partial differential equations on a mesh. We either explicitly set up a matrix, or we work in a matrix-free environment where we have to be able to quickly return matrix entries upon demand. Either way, the construction can become costly due to non-trivial material parameters entering the equations, multigrid codes requiring cascades of matrices that depend upon each other, or dynamic adaptive mesh refinement that necessitates the recomputation of matrix entries or the whole equation system throughout the solve. We propose that these constructions can be performed concurrently with the multigrid cycles. Initial geometric matrices and low accuracy integrations kickstart the multigrid, while improved assembly data is fed to the solver as and when it becomes available. The time to solution is improved as we eliminate an expensive preparation phase traditionally delaying the actual computation. We eliminate algorithmic latency. Furthermore, we desynchronise the assembly from the solution process. This anarchic increase of the concurrency level improves the scalability. Assembly routines are notoriously memory- and bandwidth-demanding. As we work with iteratively improving operator accuracies, we finally propose the use of a hierarchical, lossy compression scheme such that the memory footprint is brought down aggressively where the system matrix entries carry little information or are not yet available with high accuracy.

cs.MS↗

arXiv2020

Lightweight Task Offloading Exploiting MPI Wait Times for Parallel Adaptive Mesh Refinement

Balancing the workload of sophisticated simulations is inherently difficult, since we have to balance both computational workload and memory footprint over meshes that can change any time or yield unpredictable cost per mesh entity, while modern supercomputers and their interconnects start to exhibit fluctuating performance. We propose a novel lightweight balancing technique for MPI+X to accompany traditional, prediction-based load balancing. It is a reactive diffusion approach that uses online measurements of MPI idle time to migrate tasks temporarily from overloaded to underemployed ranks. Tasks are deployed to ranks which otherwise would wait, processed with high priority, and made available to the overloaded ranks again. This migration is non-persistent. Our approach hijacks idle time to do meaningful work and is totally non-blocking, asynchronous and distributed without a global data view. Tests with a seismic simulation code developed in the ExaHyPE engine uncover the method's potential. We found speed-ups of up to 2-3 for ill-balanced scenarios without logical modifications of the code base and show that the strategy is capable to react quickly to temporarily changing workload or node performance.

cs.DC↗

arXiv2020

Enclave Tasking for Discontinuous Galerkin Methods on Dynamically Adaptive Meshes

High-order Discontinuous Galerkin (DG) methods promise to be an excellent discretisation paradigm for partial differential equation solvers by combining high arithmetic intensity with localised data access. They also facilitate dynamic adaptivity without the need for conformal meshes. A parallel evaluation of DG's weak formulation within a mesh traversal is non-trivial, as dependency graphs over dynamically adaptive meshes change, as causal constraints along resolution transitions have to be preserved, and as data sends along MPI domain boundaries have to be triggered in the correct order. We propose to process mesh elements subject to constraints with high priority or, where needed, serially throughout a traversal. The remaining cells form enclaves and are spawned into a task system. This introduces concurrency, mixes memory-intensive DG integrations with compute-bound Riemann solves, and overlaps computation and communication. We discuss implications on MPI and show that MPI parallelisation improves by a factor of three through enclave tasking, while we obtain an additional factor of two from shared memory if grids are dynamically adaptive.

cs.MS↗

arXiv2019

A high-level characterisation and generalisation of communication-avoiding programming techniques

Today's hardware's explosion of concurrency plus the explosion of data we build upon in both machine learning and scientific simulations have multifaceted impact on how we write our codes. They have changed our notion of performance and, hence, of what a good code is: Good code has, first of all, to be able to exploit the unprecedented levels of parallelism. To do so, it has to manage to move the compute data into the compute facilities on time. As communication and memory bandwidth cannot keep pace with the growth in compute capabilities and as latency increases---at least relative to what the hardware could do---communication-avoiding techniques gain importance. We characterise and classify the field of communication-avoiding algorithms. A review of some examples of communication-avoiding programming by means of our new terminology shows that we are well-advised to broaden our notion of "communication-avoiding" and to look beyond numerical linear algebra. An abstraction, generalisation and weakening of the term enriches our toolset of how to tackle the data movement challenges. Through this, we eventually gain access to a richer set of tools that we can use to deliver proper code for current and upcoming hardware generations.

cs.DC↗

arXiv2019

Studies on the energy and deep memory behaviour of a cache-oblivious, task-based hyperbolic PDE solver

We study the performance behaviour of a seismic simulation using the ExaHyPE engine with a specific focus on memory characteristics and energy needs. ExaHyPE combines dynamically adaptive mesh refinement (AMR) with ADER-DG. It is parallelized using tasks, and it is cache efficient. AMR plus ADER-DG yields a task graph which is highly dynamic in nature and comprises both arithmetically expensive tasks and tasks which challenge the memory's latency. The expensive tasks and thus the whole code benefit from AVX vectorization, though we suffer from memory access bursts. A frequency reduction of the chip improves the code's energy-to-solution. Yet, it does not mitigate burst effects. The bursts' latency penalty becomes worse once we add Intel Optane technology, increase the core count significantly, or make individual, computationally heavy tasks fall out of close caches. Thread overbooking to hide away these latency penalties contra-productive with non-inclusive caches as it destroys the cache and vectorization character. In cases where memory-intense and computationally expensive tasks overlap, ExaHyPE's cache-oblivious implementation can exploit deep, non-inclusive, heterogeneous memory effectively, as main memory misses arise infrequently and slow down only few cores. We thus propose that upcoming supercomputing simulation codes with dynamic, inhomogeneous task graphs are actively supported by thread runtimes in intermixing tasks of different compute character, and we propose that future hardware actively allows codes to downclock the cores running particular task types.

cs.MS↗

arXiv2018

The Peano software - parallel, automaton-based, dynamically adaptive grid traversals

We discuss the design decisions, design alternatives and rationale behind the third generation of Peano, a framework for dynamically adaptive Cartesian meshes derived from spacetrees. Peano ties the mesh traversal to the mesh storage and supports only one element-wise traversal order resulting from space-filling curves. The user is not free to choose a traversal order herself. The traversal can exploit regular grid subregions and shared memory as well as distributed memory systems with almost no modifications to a serial application code. We formalize the software design by means of two interacting automata---one automaton for the multiscale grid traversal and one for the application-specific algorithmic steps. This yields a callback-based programming paradigm. We further sketch the supported application types and the two data storage schemes realized, before we detail high-performance computing aspects and lessons learned. Special emphasis is put on observations regarding the used programming idioms and algorithmic concepts. This transforms our report from a "one way to implement things" code description into a generic discussion and summary of some alternatives, rationale and design decisions to be made for any tree-based adaptive mesh refinement software.

cs.MS↗

arXiv2018

Coloured and task-based stencil codes

Simple stencil codes are and remain an important building block in scientific computing. On shared memory nodes, they are traditionally parallelised through colouring or (recursive) tiling. New OpenMP versions alternatively allow users to specify data dependencies explicitly and to outsource the decision how to distribute the work to the runtime system. We evaluate traditional multithreading strategies on both Broadwell and KNL, study the arising assignment of tasks to threads and, from there, derive two efficient ways to parallelise stencil codes on regular Cartesian grids that fuse colouring and task-based approaches.

cs.MS↗

arXiv2018

Efficient implementation of ADER discontinuous Galerkin schemes for a scalable hyperbolic PDE engine

In this paper we discuss a new and very efficient implementation of high order accurate ADER discontinuous Galerkin (ADER-DG) finite element schemes on modern massively parallel supercomputers. The numerical methods apply to a very broad class of nonlinear systems of hyperbolic partial differential equations. ADER-DG schemes are by construction communication avoiding and cache blocking and are furthermore very well-suited for vectorization, so that they appear to be a good candidate for the future generation of exascale supercomputers. We introduce the numerical algorithm and show some applications to a set of hyperbolic equations with increasing level of complexity, ranging from the compressible Euler equations over the equations of linear elasticity and the unified Godunov-Peshkov-Romenski (GPR) model of continuum mechanics to general relativistic magnetohydrodynamics (GRMHD) and the Einstein field equations of general relativity. We present strong scaling results of the new ADER-DG schemes up to 180,000 CPU cores. To our knowledge, these are the largest runs ever carried out with high order ADER-DG schemes for nonlinear hyperbolic PDE systems. We also provide a detailed performance comparison with traditional Runge-Kutta DG schemes.

math.NA↗

arXiv2018

A simple diffuse interface approach on adaptive Cartesian grids for the linear elastic wave equations with complex topography

In most classical approaches of computational geophysics for seismic wave propagation problems, complex surface topography is either accounted for by boundary-fitted unstructured meshes, or, where possible, by mapping the complex computational domain from physical space to a topologically simple domain in a reference coordinate system. In this paper we propose a completely different strategy. We address the problem of geometrically complex free surface boundary conditions with a novel diffuse interface method on adaptive Cartesian meshes that consists in the introduction of a characteristic function $ 0\leqα\leq 1$ which identifies the location of the solid medium and the surrounding air and thus implicitly defines the location of the free surface boundary. Our new approach completely avoids the problem of mesh generation, since all that is needed for the definition of the complex surface topography is to set a scalar color function to unity inside the regions covered by the solid and to zero outside. An analysis of the eigenvalues of the PDE system shows that the complexity of the geometry has no influence on the admissible time step size due to the CFL condition. The model reduces to the classical linear elasticity equations inside the solid medium where the gradients of $α$ are zero, while in the diffuse interface zone at the free surface boundary the governing PDE system becomes nonlinear. We can prove that the solution of the Riemann problem with arbitrary data and a jump in $α$ from unity to zero yields a Godunov-state at the interface that satisfies the free-surface boundary condition exactly. In order to reduce numerical dissipation, we use high order DG finite element schemes on adaptive AMR grids together with a high resolution shock capturing subcell finite volume (FV) limiter in the diffuse interface region.

math.NA↗

arXiv2018

Stop talking to me -- a communication-avoiding ADER-DG realisation

We present a communication- and data-sensitive formulation of ADER-DG for hyperbolic differential equation systems. Sensitive here has multiple flavours: First, the formulation reduces the persistent memory footprint. This reduces pressure on the memory subsystem. Second, the formulation realises the underlying predictor-corrector scheme with single-touch semantics, i.e., each degree of freedom is read on average only once per time step from the main memory. This reduces communication through the memory controllers. Third, the formulation breaks up the tight coupling of the explicit time stepping's algorithmic steps to mesh traversals. This averages out data access peaks. Different operations and algorithmic steps are ran on different grid entities. Finally, the formulation hides distributed memory data transfer behind the computation aligned with the mesh traversal. This reduces pressure on the machine interconnects. All techniques applied by our formulation are elaborated by means of a rigorous task formalism. They break up ADER-DG's tight causal coupling of compute steps and can be generalised to other predictor-corrector schemes.

cs.MS↗

arXiv2017

Quasi-matrix-free hybrid multigrid on dynamically adaptive Cartesian grids

We present a family of spacetree-based multigrid realizations using the tree's multiscale nature to derive coarse grids. They align with matrix-free geometric multigrid solvers as they never assemble the system matrices which is cumbersome for dynamically adaptive grids and full multigrid. The most sophisticated realizations use BoxMG to construct operator-dependent prolongation and restriction in combination with Galerkin/Petrov-Galerkin coarse-grid operators. This yields robust solvers for nontrivial elliptic problems. We embed the algebraic, problem- and grid-dependent multigrid operators as stencils into the grid and evaluate all matrix-vector products in-situ throughout the grid traversals. While such an approach is not literally matrix-free---the grid carries the matrix---we propose to switch to a hierarchical representation of all operators. Only differences of algebraic operators to their geometric counterparts are held. These hierarchical differences can be stored and exchanged with small memory footprint. Our realizations support arbitrary dynamically adaptive grids while they vertically integrate the multilevel operations through spacetree linearization. This yields good memory access characteristics, while standard colouring of mesh entities with domain decomposition allows us to use parallel manycore clusters. All realization ingredients are detailed such that they can be used by other codes.

math.NA↗

arXiv2016

Complex additive geometric multilevel solvers for Helmholtz equations on spacetrees

We introduce a family of implementations of low order, additive, geometric multilevel solvers for systems of Helmholtz equations. Both grid spacing and arithmetics may comprise complex numbers and we thus can apply complex scaling techniques to the indefinite Helmholtz operator. Our implementations are based upon the notion of a spacetree and work exclusively with a finite number of precomputed local element matrices. They are globally matrix-free. Combining various relaxation factors with two grid transfer operators allows us to switch from pure additive multigrid over a hierarchical basis method into BPX with several multiscale smoothing variants within one code base. Pipelining allows us to realise a full approximation storage (FAS) scheme within the additive environment where, amortised, each grid vertex carrying degrees of freedom is read/written only once per iteration. The codes thus realise a single-touch policy. Among the features facilitated by matrix-free FAS is arbitrary dynamic mesh refinement (AMR) for all solver variants. AMR as enabler for full multigrid (FMG) cycling---the grid unfolds throughout the computation---allows us to reduce the cost per unknown per order of accuracy. The present paper primary contributes towards software realisation and design questions. Our experiments show that the consolidation of single-touch FAS, dynamic AMR and vectorisation-friendly, complex scaled, matrix-free FMG cycles delivers a mature implementation blueprint for solvers for a non-trivial class of problems such as Helmholtz equations. Besides this validation, we put particular emphasis on a strict implementation formalism as well as some implementation correctness proofs.

cs.MS↗

arXiv2016

Form Follows Function -- Do algorithms and applications challenge or drag behind the hardware evolution?

We summarise some of the key statements made at the workshop Form Follows Function at ISC High Performance 2016. The summary highlights what type of co-design the presented projects experience; often in the absence of an explicit co-design agenda. Their software development picks up hardware trends but it also influences the hardware development. Observations illustrate that this cycle not always is optimal for both sides as it is not proactively steered. Key statements characterise ideas how it might be possible to integrate both hardware and software creation closer to the best of both worlds---again even without classic co-design in mind where new pieces of hardware are created. The workshop finally identified three development idioms that might help to improve software and system design with respect to emerging hardware.

cs.MS↗