arXiv subjects
Publications and source records attributed to Carsten Burstedde.
We describe the application of a scalable algorithm for interpolating solution data in the overlapping mesh region of two solvers. This feature is essential to obtain a globally consistent solution for in-situ coupled atmospheric wave simulation. We provide timings and discuss a real-world application run.
We introduce an algorithm that performs a one-directional mesh overset of a parallel forest of octrees with another distributed mesh of unrelated partition. The forest mesh consists of several adaptively refined octrees. Individual smooth mappings for every tree allow to represent a broad range of geometric domains. The other mesh is generic and defines a distributed set of query points, e.g. stemming from a quadrature rule applied to each cell. We face the problem of finding data for all queries in the remote forest. The forest is partitioned according to its natural Morton ordering. Thus, the partition boundaries can be encoded globally with one Morton index per process, which allows for precise, communication-free searching of the queries in the partition geometry. This is necessary to organize non-blocking communication of the queries to the relevant processes only. In a subsequent local search of the forest, we process the incoming queries and return the data of interest to each query's origin. The algorithm can be generalized, for example to load balancing of the overset, or adaptive refinement of the meshes around their intersection area. In 2D and 3D example scenarios we demonstrate the algorithm's performance and scalability to 12,288 processes.
The forest-of-refinement-trees approach allows for dynamic adaptive mesh refinement (AMR) at negligible cost. While originally developed for quadrilateral and hexahedral elements, previous work established the theory and algorithms for unstructured meshes of simplicial and prismatic elements. To harness the full potential of tree-based AMR for three-dimensional mixed-element meshes, this paper introduces the pyramid as a new functional element type; its primary purpose is to connect tetrahedral and hexahedral elements without hanging edges. We present a well-defined space-filling curve (SFC) for the pyramid and detail how the unique challenges on the element and forest level associated with the pyramidal refinement are resolved. We propose the necessary functional design and generalize the fundamental global parallel algorithms for refinement, coarsening, partitioning, and face ghost exchange to fully support this new element. Our demonstrations confirm the efficiency and scalability of this complete, hybrid-element dynamic AMR framework.
In this work we extend the Dune solver library with another grid interface to the open-source p4est software. While Dune already supports about a dozen different mesh implementations through its mesh interface Dune-Grid, we undertake this new coupling effort in order to inherit p4est's practically unlimited MPI scalability as well as its relatively thin data structures, and its native support for multi-block (forest) mesh topologies in both 2D and 3D. The presented implementation is compared to an existing implementation based on Dune-ALUGrid for a variety of challenging test examples in a parallel environment. The numerical experiments show that the implementation presented here is outperforming Dune-ALUGrid in terms of scalability. In addition, an alternative balancing strategy is presented to ensure 2:1 balancing across element faces showing improved performance compared to the existing p4est balance strategy in the numerical examples considered in this work.
We describe a fast, direct solver for elliptic partial differential equations on a two-dimensional hierarchy of adaptively refined, Cartesian meshes. Our solver, inspired by the Hierarchical Poincaré-Steklov (HPS) method introduced by Gillman and Martinsson (SIAM J. Sci. Comput., 2014) uses fast solvers on locally uniform Cartesian patches stored in the leaves of a quadtree and is the first such solver that works directly with the adaptive quadtree mesh managed using the grid management library \pforest (C. Burstedde, L. Wilcox, O. Ghattas, SIAM J. Sci. Comput. 2011). Within each Cartesian patch, stored in leaves of the quadtree, we use a second order finite volume discretization on cell-centered meshes. Key contributions of our algorithm include 4-to-1 merge and split implementations for the HPS build stage and solve stage, respectively. We demonstrate our solver on Poisson and Helmholtz problems with a mesh adapted to the high local curvature of the right-hand side.
We present a technical enhancement within the p4est software for parallel adaptive mesh refinement. In p4est primitives are stored as octants in three and quadrants in two dimensions. While, classically, they are encoded by the native approach using its spatial and refinement level, any other mathematically equivalent encoding might be used instead. Recognizing this, we add two alternative representations to the classical, explicit version, based on a long monotonic index and 128-bit AVX quad integers, respectively. The first one requires changes in logic for low-level quadrant manipulating algorithms, while the other exploits data level parallelism and requires algorithms to be adapted to SIMD instructions. The resultant algorithms and data structures lead to higher performance and lesser memory usage in comparison with the standard baseline. We benchmark selected algorithms on a cluster with two Intel(R) Xeon(R) Gold 6130 Skylake family CPUs per node, which provides support for AVX2 extensions, 192 GB RAM per node, and up to 512 computational cores in total.
We specify a file-oriented data format suitable for parallel, partition-independent disk I/O. Here, a partition refers to a disjoint and ordered distribution of the data elements between one or more processes. The format is designed such that the file contents are invariant under linear (i. e., unpermuted), parallel repartition of the data prior to writing. The file contents are indistinguishable from writing in serial. In the same vein, the file can be read on any number of processes that agree on any partition of the number of elements stored. In addition to the format specification we propose an optional convention to implement transparent per-element data compression. The compressed data and metadata is layered inside ordinary format elements. Overall, we pay special attention to both human and machine readability. If pure ASCII data is written, or compressed data is reencoded to ASCII, the entire file including its header and sectioning metadata remains entirely in ASCII. If binary data is written, the metadata stays easy on the human eye. We refer to this format as scda. Conceptually, it lies one layer below and is oblivious to the definition of variables, the binary representation of numbers, considerations of endianness, and self-describing headers, which may all be specified on top of scda. The main purpose of the format is to abstract any parallelism and provide sufficient structure as a foundation for a generic and flexible archival and checkpoint/restart. A documented reference implementation is available as part of the general-purpose libsc free software library.
We discuss parallel algorithms to gather topological information about off-process mesh neighbor elements. This information is commonly called the ghost layer, whose creation is a fundamental, necessary task in executing most parallel, element-based computer simulations. Approaches differ in that the ghost layer may either be inherently part of the mesh data structure that is maintained and modified, or kept separate and constructed/deleted as needed. In this work, we present an updated design following the latter approach, which we favor for its modularity of algorithms and data structures. We target arbitrary adaptive, non-conforming forest-of-(oc)trees meshes of mixed element shapes, such as cubes, prisms, and tetrahedra, and restrict ourselves to face-ghosts. Our algorithm has low complexity and redundancy since we reduce it to generic codimension-1 subalgorithms that can be flexibly combined. We cover several existing solutions as special cases and optimize further using recursive, amortized tree searches and traversals.
We introduce several parallel algorithms operating on a distributed forest of adaptive quadtrees/octrees. They are targeted at large-scale applications relying on data layouts that are more complex than required for standard finite elements, such as hp-adaptive Galerkin methods, particle tracking and semi-Lagrangian schemes, and in-situ post-processing and visualization. Specifically, we design algorithms to derive an adapted worker forest based on sparse data, to identify owner processes in a top-down search of remote objects, and to allow for variable process counts and per-element data sizes in partitioning and parallel file I/O. We demonstrate the algorithms' usability and performance in the context of a particle tracking example that we scale to 21e9 particles and 64Ki MPI processes on the Juqueen supercomputer, and we describe the construction of a parallel assembly of variably sized spheres in space creating up to 768e9 elements on the Juwels supercomputer.
We investigate various block preconditioners for a low-order Raviart-Thomas discretization of the mixed Poisson problem on adaptive quadrilateral meshes. In addition to standard diagonal and Schur complement preconditioners, we present a dedicated AMG solver for saddle point problems (SPAMG). A key element is a stabilized prolongation operator that couples the flux and scalar components. Our numerical experiments in 2D and 3D show that the SPAMG preconditioner displays nearly mesh-independent iteration counts for adaptive meshes and heterogeneous coefficients.
We present an MPI-parallel algorithm for the in-situ visualization of computational data that is built around a distributed linear forest-of-octrees data structure. Such octrees are frequently used in element-based numerical simulations; they store the leaves of the tree that are local in the curent parallel partition. We proceed in three stages. First, we prune all elements whose bounding box is not visible by a parallel top-down traversal, and repartition the remaining ones for load-balancing. Second, we intersect each element with every ray passing its box to derive color and opacity values for the ray segment. To reduce data, we aggregate the segments up the octree in a strictly distributed fashion in cycles of coarsening and repartition. Third, we composite all remaining ray segments to a tiled partition of the image and write it to disk using parallel I/O. The scalability of the method derives from three concepts: By exploiting the space filling curve encoding of the octrees and by relying on recently developed tree algorithms for top-down partition traversal, we are able to determine sender/receiver pairs without handshaking and/or collective communication. Furthermore, by partnering the linear traversal of tree leaves with the group action of the attenuation/emission ODE along each segment, we avoid back-to-front sorting of elements throughout. Lastly, the method is problem adaptive with respect to the refinement and partition of the elements and to the accuracy of ODE integration.
Regional hydrology studies are often supported by high resolution simulations of subsurface flow that require expensive and extensive computations. Efficient usage of the latest high performance parallel computing systems becomes a necessity. The simulation software ParFlow has been demonstrated to meet this requirement and shown to have excellent solver scalability for up to 16,384 processes. In the present work we show that the code requires further enhancements in order to fully take advantage of current petascale machines. We identify ParFlow's way of parallelization of the computational mesh as a central bottleneck. We propose to reorganize this subsystem using fast mesh partition algorithms provided by the parallel adaptive mesh refinement library p4est. We realize this in a minimally invasive manner by modifying selected parts of the code to reinterpret the existing mesh data structures. We evaluate the scaling performance of the modified version of ParFlow, demonstrating good weak and strong scaling up to 458k cores of the Juqueen supercomputer, and test an example application at large scale.
We introduce a space-filling curve for triangular and tetrahedral red-refinement that can be computed using bitwise interleaving operations similar to the well-known Z-order or Morton curve for cubical meshes. To store sufficient information for random access, we define a low-memory encoding using 10 bytes per triangle and 14 bytes per tetrahedron. We present algorithms that compute the parent, children, and face-neighbors of a mesh element in constant time, as well as the next and previous element in the space-filling curve and whether a given element is on the boundary of the root simplex or not. Our presentation concludes with a scalability demonstration that creates and adapts selected meshes on a large distributed-memory system.
The Morton- or z-curve is one example for a space filling curve: Given a level of refinement L, it maps the interval [0, 2**dL) one-to-one to a set of d-dimensional cubes of edge length 2**-L that form a subdivision of the unit cube. Similar curves have been proposed for triangular and tetrahedral unit domains. In contrast to the Hilbert curve that is continuous, the Morton-type curves produce jumps. We prove that any contiguous subinterval of the curve divides the domain into a bounded number of face-connected subdomains. For the hypercube case and arbitrary dimension, the subdomains are star-shaped and the bound is indeed two. For the simplicial case in dimensions 2 and 3, the bound is proportional to the depth of refinement L. We supplement the paper with theoretical and computational studies on the frequency of jumps for a quantitative assessment.
We describe a parallel, adaptive, multi-block algorithm for explicit integration of time dependent partial differential equations on two-dimensional Cartesian grids. The grid layout we consider consists of a nested hierarchy of fixed size, non-overlapping, logically Cartesian grids stored as leaves in a quadtree. Dynamic grid refinement and parallel partitioning of the grids is done through the use of the highly scalable quadtree/octree library p4est. Because our concept is multi-block, we are able to easily solve on a variety of geometries including the cubed sphere. In this paper, we pay special attention to providing details of the parallel ghost-filling algorithm needed to ensure that both corner and edge ghost regions around each grid hold valid values. We have implemented this algorithm in the ForestClaw code using single-grid solvers from ClawPack, a software package for solving hyperbolic PDEs using finite volumes methods. We show weak and strong scalability results for scalar advection problems on two-dimensional manifold domains on 1 to 64Ki MPI processes, demonstrating neglible regridding overhead.
In tree based adaptive mesh refinement, elements are partitioned between processes using a space filling curve. The curve establishes an ordering between all elements that derive from the same root element, the tree. When representing more complex geometries by patching together several trees, the roots of these trees form an unstructured coarse mesh. We present an algorithm to partition the elements of the coarse mesh such that (a) the fine mesh can be load-balanced to equal element counts per process regardless of the element-to-tree map and (b) each process that holds fine mesh elements has access to the meta data of all relevant trees. As an additional feature, the algorithm partitions the meta data of relevant ghost (halo) trees as well. We develop in detail how each process computes the communication pattern for the partition routine without handshaking and with minimal data movement. We demonstrate the scalability of this approach on up to 917e3 MPI ranks and .37e12 coarse mesh elements, measuring run times of one second or less.
The forest-of-octrees approach to parallel adaptive mesh refinement and coarsening (AMR) has recently been demonstrated in the context of a number of large-scale PDE-based applications. Although linear octrees, which store only leaf octants, have an underlying tree structure by definition, it is not often exploited in previously published mesh-related algorithms. This is because the branches are not explicitly stored, and because the topological relationships in meshes, such as the adjacency between cells, introduce dependencies that do not respect the octree hierarchy. In this work we combine hierarchical and topological relationships between octree branches to design efficient recursive algorithms. We present three important algorithms with recursive implementations. The first is a parallel search for leaves matching any of a set of multiple search criteria. The second is a ghost layer construction algorithm that handles arbitrarily refined octrees that are not covered by previous algorithms, which require a 2:1 condition between neighboring leaves. The third is a universal mesh topology iterator. This iterator visits every cell in a domain partition, as well as every interface (face, edge and corner) between these cells. The iterator calculates the local topological information for every interface that it visits, taking into account the nonconforming interfaces that increase the complexity of describing the local topology. To demonstrate the utility of the topology iterator, we use it to compute the numbering and encoding of higher-order $C^0$ nodal basis functions. We analyze the complexity of the new recursive algorithms theoretically, and assess their performance, both in terms of single-processor efficiency and in terms of parallel scalability, demonstrating good weak and strong scaling up to 458k cores of the JUQUEEN supercomputer.
We present a new hybrid paradigm for parallel adaptive mesh refinement (AMR) that combines the scalability and lightweight architecture of tree-based AMR with the computational efficiency of patch-based solvers for hyperbolic conservation laws. The key idea is to interpret each leaf of the AMR hierarchy as one uniform compute patch in $\sR^d$ with $m^d$ degrees of freedom, where $m$ is customarily between 8 and 32. Thus, computation on each patch can be optimized for speed, while we inherit the flexibility of adaptive meshes. In our work we choose to integrate with the p4est AMR library since it allows us to compose the mesh from multiple mapped octrees and enables the cubed sphere and other nontrivial multiblock geometries. We describe aspects of the parallel implementation and close with scalings for both MPI-only and OpenMP/MPI hybrid runs, where the largest MPI run executes on 16,384 CPU cores.