arXiv subjects

Erik Schnetter

Publications and source records attributed to Erik Schnetter.

At least 37 records · Page 2Linked to original sources

arXiv2022

GRaM-X: A new GPU-accelerated dynamical spacetime GRMHD code for Exascale computing with the Einstein Toolkit

We present GRaM-X (General Relativistic accelerated Magnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems. GRaM-X supports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit called CarpetX which in turn leverages AMReX, an AMR library developed for use by the United States DOE's Exascale Computing Project (ECP). We use the Z4c formalism to evolve the equations of GR and the Valencia formulation to evolve the equations of GRMHD. GRaM-X supports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D MHD shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D TOV star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find that GRaM-X shows a weak scaling efficiency of $\sim 40-50\%$ on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF's supercomputer Summit.

astro-ph.IM↗

arXiv2022

Bridging HPC Communities through the Julia Programming Language

The Julia programming language has evolved into a modern alternative to fill existing gaps in scientific computing and data science applications. Julia leverages a unified and coordinated single-language and ecosystem paradigm and has a proven track record of achieving high performance without sacrificing user productivity. These aspects make Julia a viable alternative to high-performance computing's (HPC's) existing and increasingly costly many-body workflow composition strategy in which traditional HPC languages (e.g., Fortran, C, C++) are used for simulations, and higher-level languages (e.g., Python, R, MATLAB) are used for data analysis and interactive computing. Julia's rapid growth in language capabilities, package ecosystem, and community make it a promising universal language for HPC. This paper presents the views of a multidisciplinary group of researchers from academia, government, and industry that advocate for an HPC software development paradigm that emphasizes developer productivity, workflow portability, and low barriers for entry. We believe that the Julia programming language, its ecosystem, and its community provide modern and powerful capabilities that enable this group's objectives. Crucially, we believe that Julia can provide a feasible and less costly approach to programming scientific applications and workflows that target HPC facilities. In this work, we examine the current practice and role of Julia as a common, end-to-end programming model to address major challenges in scientific reproducibility, data-driven AI/machine learning, co-design and workflows, scalability and performance portability in heterogeneous computing, network communication, data management, and community education. As a result, the diversification of current investments to fulfill the needs of the upcoming decade is crucial as more supercomputing centers prepare for the exascale era.

cs.DC↗

arXiv2022

Curvature Invariants and the Geometric Horizon Conjecture in a Binary Black Hole Merger

We study curvature invariants in a binary black hole merger. It has been conjectured that one could define a quasi-local and foliation independent black hole horizon by finding the level--$0$ set of a suitable curvature invariant of the Riemann tensor. The conjecture is the geometric horizon conjecture and the associated horizon is the geometric horizon. We study this conjecture by tracing the level--$0$ set of the complex scalar polynomial invariant, $\mathcal{D}$, through a quasi-circular binary black hole merger. We approximate these level--$0$ sets of $\mathcal{D}$ with level--$\varepsilon$ sets of $|\mathcal{D}|$ for small $\varepsilon$. We locate the local minima of $|\mathcal{D}|$ and find that the positions of these local minima correspond closely to the level--$\varepsilon$ sets of $|\mathcal{D}|$ and we also compare with the level--$0$ sets of $\text{Re}(\mathcal{D})$. The analysis provides evidence that the level--$\varepsilon$ sets track a unique geometric horizon. By studying the behaviour of the zero sets of $\text{Re}(\mathcal{D})$ and $\text{Im}(\mathcal{D})$ and also by studying the MOTSs and apparent horizons of the initial black holes, we observe that the level--$\varepsilon$ set that best approximates the geometric horizon is given by $\varepsilon = 10^{-3}$.

gr-qc↗

arXiv2022

Quasinormal modes and their overtones at the common horizon in a binary black hole merger

It is expected that all astrophysical black holes in equilibrium are well described by the Kerr solution. Moreover, any black hole far away from equilibrium, such as one initially formed in a compact binary merger or by the collapse of a massive star, will eventually reach a final equilibrium Kerr state. At sufficiently late times in this process of reaching equilibrium, we expect that the black hole is modeled as a perturbation around the final state. The emitted gravitational waves will then be damped sinusoids with frequencies and damping times given by the quasinormal mode spectrum of the final Kerr black hole. An observational test of this scenario, often referred to as black hole spectroscopy, is one of the major goals of gravitational wave astronomy. It was recently suggested that the quasinormal mode description including the higher overtones might hold even right after the remnant black hole is first formed. At these times, the black hole is expected to be highly dynamical and nonlinear effects are likely to be important. In this paper we investigate this remarkable scenario in terms of the horizon dynamics. Working with high accuracy simulations of a simple configuration, namely the head-on collision of two nonspinning black holes with unequal masses, we study the dynamics of the final common horizon in terms of its shear and its multipole moments. We show that they are indeed well described by a superposition of ringdown modes as long as a sufficiently large number of higher overtones are included. This description holds even for the highly dynamical final black hole shortly after its formation. We discuss the implications and caveats of this result for black hole spectroscopy and for our understanding of the approach to equilibrium.

gr-qc↗

arXiv2021

Geometric horizons in binary black hole mergers

We numerically study the algebraic properties of the Weyl tensor through the merger of two non-spinning black holes (BHs). We are particularly interested in the conjecture that for such a vacuum spacetime, which is zeroth-order algebraically general, a geometric horizon (GH), on which the spacetime is algebraically special and which is identified by the vanishing of a complex scalar invariant (${\mathcal{D}}$), characterizes a smooth foliation independent surface (horizon) associated with the BH. In the first simulation we investigate the level-$0$ sets of $\text{Re}({\mathcal{D}})$ (since $\text{Im}({\mathcal{D}})= 0$) in the head-on collision of two unequal mass BHs. In the second simulation we shall investigate the level-$\varepsilon$ sets of $|{\mathcal{D}}|$ through a quasi-circular merger of two non-spinning, equal mass BHs. The numerical results, as displayed in the figures presented, provide evidence that a (unique) smooth GH can be identified throughout all stages of the binary BH merger.

gr-qc↗

arXiv2020

A magnetar engine for short GRBs and kilonovae

We investigate the influence of magnetic fields on the evolution of binary neutron-star (BNS) merger remnants via three-dimensional (3D) dynamical-spacetime general-relativistic (GR) magnetohydrodynamic (MHD) simulations. We evolve a postmerger remnant with an initial poloidal magnetic field, resolve the magnetoturbulence driven by shear flows, and include a microphysical finite-temperature equation of state (EOS). A neutrino leakage scheme that captures the overall energetics and lepton number exchange is also included. We find that turbulence induced by the magnetorotational instability (MRI) in the hypermassive neutron star (HMNS) amplifies magnetic field to beyond magnetar-strength ($10^{15}\, \mathrm{G}$). The ultra-strong toroidal field is able to launch a relativistic jet from the HMNS. We also find a magnetized wind that ejects neutron-rich material with a rate of $\dot{M}_{\mathrm{ej}} \simeq 1 \times10^{-1}\, \mathrm{M_{\odot}\, s^{-1}}$. The total ejecta mass in our simulation is $5\times 10^{-3}\, \mathrm{M_{\odot}}$. This makes the ejecta from the HMNS an important component in BNS mergers and a promising source of $r$-process elements that can power a kilonova. The jet from the HMNS reaches a terminal Lorentz factor of $\sim 5$ in our highest-resolution simulation. The formation of this jet is aided by neutrino-cooling preventing the accretion disk from protruding into the polar region. As neutrino pair-annihilation and radiative processes in the jet (which were not included in the simulations) will boost the Lorentz factor in the jet further, our simulations demonstrate that magnetars formed in BNS mergers are a viable engine for short gamma-ray bursts (sGRBs).

astro-ph.HE↗

arXiv2020

Horizons in a binary black hole merger I: Geometry and area increase

Recent advances in numerical relativity have revealed how marginally trapped surfaces behave when black holes merge. It is now known that interesting topological features emerge during the merger, and marginally trapped surfaces can have self-intersections. This paper presents the most detailed study yet of the physical and geometric aspects of this scenario. For the case of a head-on collision of non-spinning black holes, we study in detail the world tube formed by the evolution of marginally trapped surfaces. In the first of this two-part study, we focus on geometrical properties of the dynamical horizons, i.e. the world tube traced out by the time evolution of marginally outer trapped surfaces. We show that even the simple case of a head-on collision of non-spinning black holes contains a rich variety of geometric and topological properties and is generally more complex than considered previously in the literature. The dynamical horizons are shown to have mixed signature and are not future marginally trapped everywhere. We analyze the area increase of the marginal surfaces along a sequence which connects the two initially disjoint horizons with the final common horizon. While the area does increase overall along this sequence, it is not monotonic. We find short durations of anomalous area change which, given the connection of area with entropy, might have interesting physical consequences. We investigate the possible reasons for this effect and show that it is consistent with existing proofs of the area increase law.

gr-qc↗

arXiv2020

Horizons in a binary black hole merger II: Fluxes, multipole moments and stability

We study in detail the dynamics and stability of marginally trapped surfaces during a binary black hole merger. This is the second in a two-part study. The first part studied the basic geometric aspects of the world tubes traced out by the marginal surfaces and the status of the area increase law. Here we continue and study the dynamics of the horizons during the merger, again for the head-on collision of two non-spinning black holes. In particular we follow the spectrum of the stability operator during the course of the merger for all the horizons present in the problem and implement systematic spectrum statistics for its analysis. We also study more physical aspects of the merger, namely the fluxes of energy which cross the horizon and cause the area to change. We construct a natural coordinate system on the horizon and decompose the various fields appearing in the flux, primarily the shear of the outgoing null normal, in spin weighted spherical harmonics. For each of the modes we extract the decay rates as the final black hole approaches equilibrium. The late part of the decay is consistent with the expected quasi-normal mode frequencies, while the early part displays a much steeper fall-off. Similarly, we calculate the decay of the horizon multipole moments, again finding two different regimes. Finally, seeking an explanation for this behavior, motivated by the membrane paradigm interpretation, we attempt to identify the different dynamical timescales of the area increase. This leads to the definition of a ``slowness parameter'' for predicting the onset of transition from a faster to a slower decay.

gr-qc↗

arXiv2020

News from horizons in binary black hole mergers

In a binary black hole merger, it is known that the inspiral portion of the waveform corresponds to two distinct horizons orbiting each other, and the merger and ringdown signals correspond to the final horizon being formed and settling down to equilibrium. However, we still lack a detailed understanding of the relation between the horizon geometry in these three regimes and the observed waveform. Here we show that the well known inspiral chirp waveform has a clear counterpart on black hole horizons, namely, the shear of the outgoing null rays at the horizon. We demonstrate that the shear behaves very much like a compact binary coalescence waveform with increasing frequency and amplitude. Furthermore, the parameters of the system estimated from the horizon agree with those estimated from the waveform. This implies that even though black hole horizons are causally disconnected from us, assuming general relativity to be true, we can potentially infer some of their detailed properties from gravitational wave observations.

gr-qc↗

arXiv2019

The interior of a binary black hole merger

We find strong numerical evidence for a new phenomenon in a binary black hole spacetime, namely the merger of marginally outer trapped surfaces (MOTSs). By simulating the head-on collision of two non-spinning unequal mass black holes, we observe that the MOTS associated with the final black hole merges with the two initially disjoint surfaces corresponding to the two initial black holes. This yields a connected sequence of MOTSs interpolating between the initial and final state all the way through the non-linear binary black hole merger process. In addition, we show the existence of a MOTS with self-intersections formed immediately after the merger. This scenario now allows us to track physical quantities (such as mass, angular momentum, higher multipoles, and fluxes) across the merger, which can be potentially compared with the gravitational wave signal in the wave-zone, and with observations by gravitational wave detectors. This also suggests a possibility of proving the Penrose inequality mathematically for generic astrophysical binary back hole configurations.

gr-qc↗

arXiv2019

Self-intersecting marginally outer trapped surfaces

We have shown previously that a merger of marginally outer trapped surfaces (MOTSs) occurs in a binary black hole merger and that there is a continuous sequence of MOTSs which connects the initial two black holes to the final one. In this paper, we confirm this scenario numerically and we detail further improvements in the numerical methods for locating MOTSs. With these improvements, we confirm the merger scenario and demonstrate the existence of self-intersecting MOTSs formed in the immediate aftermath of the merger. These results will allow us to track physical quantities across the non-linear merger process and to potentially infer properties of the merger from gravitational wave observations.

gr-qc↗

arXiv2019

Spectral Methods in the Presence of Discontinuities

Spectral methods provide an elegant and efficient way of numerically solving differential equations of all kinds. For smooth problems, truncation error for spectral methods vanishes exponentially in the infinity norm and $L_2$-norm. However, for non-smooth problems, convergence is significantly worse---the $L_2$-norm of the error for a discontinuous problem will converge at a sub-linear rate and the infinity norm will not converge at all. We explore and improve upon a post-processing technique---optimally convergent mollifiers---to recover exponential convergence from a poorly-converging spectral reconstruction of non-smooth data. This is an important first step towards using these techniques for simulations of realistic systems.

math.NA↗

arXiv2018

The existence and stability of marginally trapped surfaces

Marginally outer trapped surfaces (MOTSs, or marginal surfaces in short) are routinely used in numerical simulations of black hole spacetimes. They are an invaluable tool for locating and characterizing black holes quasi-locally in real time while the simulation is ongoing. It is often believed that a MOTS can behave unpredictably under time evolution; an existing MOTS can disappear, and a new one can appear without any apparent reason. In this paper we show that in fact the behavior of a MOTS is perfectly predictable and its behavior is dictated by a single real parameter, the \emph{stability parameter}, which can be monitored during the course of a numerical simulation. We demonstrate the utility of the stability parameter to fully understand the variety of marginal surfaces that can be present in binary black hole initial data. We also develop a new horizon finder capable of locating very highly distorted marginal surfaces and we show that even in these cases, the stability parameter perfectly predicts the existence and stability of marginal surfaces.

gr-qc↗

arXiv2018

A large scale dynamo and magnetoturbulence in rapidly rotating core-collapse supernovae

Magnetohydrodynamic (MHD) turbulence is of key importance in many high-energy astrophysical systems, where MHD instabilities can amplify local magnetic field over very short time scales. Specifically, the magnetorotational instability (MRI) and dynamo action have been suggested as a mechanism to grow magnetar-strength magnetic field ($\ge10^{15} G$) and magnetorotationally power the explosion of a rotating massive star. Such stars are progenitor candidates for type Ic-bl hypernova explosions and make up all supernovae connected to long gamma-ray bursts (GRBs). The MRI has been studied with local high-resolution shearing box simulations in 3D or with global 2D simulations, but it is an open question whether MRI-driven turbulence can result in the creation of a large-scale ordered and dynamically relevant field. Here we report results from global 3D general-relativistic magnetohydrodynamic (GRMHD) turbulence simulations and show that MRI-driven MHD turbulence in rapidly rotating protoneutron stars produces an inverse cascade of energy. We find a large-scale ordered toroidal field that is consistent with the formation of bipolar magnetorotationally driven outflows. Our results demonstrate that rapidly rotating massive stars are plausible progenitors for both type Ic-bl supernovae and long GRBs, present a viable formation scenario for magnetars, and may account for potentially magnetar-powered superluminous supernovae.

astro-ph.HE↗

arXiv2018

Dynamics of marginally trapped surfaces in a binary black hole merger: Growth and approach to equilibrium

The behavior of quasi-local black hole horizons in a binary black hole merger is studied numerically. We compute the horizon multipole moments, fluxes and other quantities on black hole horizons throughout the merger. These lead to a better qualitative and quantitative understanding of the coalescence of two black holes; how the final black hole is formed, initially grows and then settles down to a Kerr black hole. We calculate the rate at which the final black hole approaches equilibrium in a fully non-perturbative situation and identify a time at which the linear ringdown phase begins. Finally, we provide additional support for the conjecture that fields at the horizon are correlated with fields in the wave-zone by comparing the in-falling gravitational wave flux at the horizon to the outgoing flux as estimated from the gravitational waveform.

gr-qc↗

arXiv2017

R-process Nucleosynthesis from Three-Dimensional Magnetorotational Core-Collapse Supernovae

We investigate r-process nucleosynthesis in three-dimensional (3D) general-relativistic magnetohydrodynamic simulations of rapidly rotating strongly magnetized core collapse. The simulations include a microphysical finite-temperature equation of state and a leakage scheme that captures the overall energetics and lepton number exchange due to postbounce neutrino emission and absorption. We track the composition of the ejected material using the nuclear reaction network SkyNet. Our results show that the 3D dynamics of magnetorotational core-collapse supernovae (CCSN) are important for their nucleosynthetic signature. We find that production of r-process material beyond the second peak is reduced by a factor of 100 when the magnetorotational jets produced by the rapidly rotating core undergo a kink instability. Our results indicate that 3D magnetorotationally powered CCSNe are a robust r-process source only if they are obtained by the collapse of cores with unrealistically large precollapse magnetic fields of order $10^{13}$G. Additionally, a comparison simulation that we restrict to axisymmetry, results in overly optimistic r-process production for lower magnetic field strengths.

astro-ph.HE↗

arXiv2017

Sparse Grid Discretizations based on a Discontinuous Galerkin Method

We examine and extend Sparse Grids as a discretization method for partial differential equations (PDEs). Solving a PDE in $D$ dimensions has a cost that grows as $O(N^D)$ with commonly used methods. Even for moderate $D$ (e.g. $D=3$), this quickly becomes prohibitively expensive for increasing problem size $N$. This effect is known as the Curse of Dimensionality. Sparse Grids offer an alternative discretization method with a much smaller cost of $O(N \log^{D-1}N)$. In this paper, we introduce the reader to Sparse Grids, and extend the method via a Discontinuous Galerkin approach. We then solve the scalar wave equation in up to $6+1$ dimensions, comparing cost and accuracy between full and sparse grids. Sparse Grids perform far superior, even in three dimensions. Our code is freely available as open source, and we encourage the reader to reproduce the results we show.

math.NA↗

arXiv2017

SpECTRE: A Task-based Discontinuous Galerkin Code for Relativistic Astrophysics

We introduce a new relativistic astrophysics code, SpECTRE, that combines a discontinuous Galerkin method with a task-based parallelism model. SpECTRE's goal is to achieve more accurate solutions for challenging relativistic astrophysics problems such as core-collapse supernovae and binary neutron star mergers. The robustness of the discontinuous Galerkin method allows for the use of high-resolution shock capturing methods in regions where (relativistic) shocks are found, while exploiting high-order accuracy in smooth regions. A task-based parallelism model allows efficient use of the largest supercomputers for problems with a heterogeneous workload over disparate spatial and temporal scales. We argue that the locality and algorithmic structure of discontinuous Galerkin methods will exhibit good scalability within a task-based parallelism framework. We demonstrate the code on a wide variety of challenging benchmark problems in (non)-relativistic (magneto)-hydrodynamics. We demonstrate the code's scalability including its strong scaling on the NCSA Blue Waters supercomputer up to the machine's full capacity of 22,380 nodes using 671,400 threads.

astro-ph.HE↗