-
AsterX: a new open-source GPU-accelerated GRMHD code for dynamical spacetimes
Authors:
Jay V. Kalinani,
Liwei Ji,
Lorenzo Ennoggi,
Federico G. Lopez Armengol,
Lucas Timotheo Sanches,
Bing-Jyun Tsao,
Steven R Brandt,
Manuela Campanelli,
Riccardo Ciolfi,
Bruno Giacomazzo,
Roland Haas,
Erik Schnetter,
Yosef Zlochower
Abstract:
We present AsterX, a novel open-source, GPU-accelerated, fully general relativistic magnetohydrodynamic (GRMHD) code designed for dynamic spacetimes in 3D Cartesian coordinates, and tailored for exascale computing. We utilize block-structured adaptive mesh refinement (AMR) through CarpetX, the new driver for the Einstein Toolkit, which is built on AMReX, a software framework for massively parallel…
▽ More
We present AsterX, a novel open-source, GPU-accelerated, fully general relativistic magnetohydrodynamic (GRMHD) code designed for dynamic spacetimes in 3D Cartesian coordinates, and tailored for exascale computing. We utilize block-structured adaptive mesh refinement (AMR) through CarpetX, the new driver for the Einstein Toolkit, which is built on AMReX, a software framework for massively parallel applications. AsterX employs the Valencia formulation for GRMHD, coupled with the 'Z4c' formalism for spacetime evolution, while incorporating high resolution shock capturing schemes to accurately handle the hydrodynamics. AsterX has undergone rigorous testing in both static and dynamic spacetime, demonstrating remarkable accuracy and agreement with other codes in literature. Benchmarking the code through scaling tests on OLCF's Frontier supercomputer, we demonstrate a weak scaling efficiency of about 67%-77% on 4096 nodes compared to an 8-node performance.
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
Ameliorating the Courant-Friedrichs-Lewy condition in spherical coordinates: A double FFT filter method for general relativistic MHD in dynamical spacetimes
Authors:
Liwei Ji,
Vassilios Mewes,
Yosef Zlochower,
Lorenzo Ennoggi,
Federico G. Lopez Armengol,
Manuela Campanelli,
Federico Cipolletta,
Zachariah B. Etienne
Abstract:
Numerical simulations of merging compact objects and their remnants form the theoretical foundation for gravitational wave and multi-messenger astronomy. While Cartesian-coordinate-based adaptive mesh refinement is commonly used for simulations, spherical-like coordinates are more suitable for nearly spherical remnants and azimuthal flows due to lower numerical dissipation in the evolution of flui…
▽ More
Numerical simulations of merging compact objects and their remnants form the theoretical foundation for gravitational wave and multi-messenger astronomy. While Cartesian-coordinate-based adaptive mesh refinement is commonly used for simulations, spherical-like coordinates are more suitable for nearly spherical remnants and azimuthal flows due to lower numerical dissipation in the evolution of fluid angular momentum, as well as requiring fewer numbers of computational cells. However, the use of spherical coordinates to numerically solve hyperbolic partial differential equations can result in severe Courant-Friedrichs-Lewy (CFL) stability condition timestep limitations, which can make simulations prohibitively expensive. This paper addresses this issue for the numerical solution of coupled spacetime and general relativistic magnetohydrodynamics evolutions by introducing a double FFT filter and implementing it within the fully MPI-parallelized SphericalNR framework in the Einstein Toolkit. We demonstrate the effectiveness and robustness of the filtering algorithm by applying it to a number of challenging code tests, and show that it passes these tests effectively, demonstrating convergence while also increasing the timestep significantly compared to unfiltered simulations.
△ Less
Submitted 9 November, 2023; v1 submitted 2 May, 2023;
originally announced May 2023.
-
Addition of tabulated equation of state and neutrino leakage support to IllinoisGRMHD
Authors:
Leonardo R. Werneck,
Zachariah B. Etienne,
Ariadna Murguia-Berthier,
Roland Haas,
Federico Cipolletta,
Scott C. Noble,
Lorenzo Ennoggi,
Federico G. Lopez Armengol,
Bruno Giacomazzo,
Thiago Assumpção,
Joshua Faber,
Tanmayee Gupte,
Bernard J. Kelly,
Julian H. Krolik
Abstract:
We have added support for realistic, microphysical, finite-temperature equations of state (EOS) and neutrino physics via a leakage scheme to IllinoisGRMHD, an open-source GRMHD code for dynamical spacetimes in the Einstein Toolkit. These new features are provided by two new, NRPy+-based codes: NRPyEOS, which performs highly efficient EOS table lookups and interpolations, and NRPyLeakage, which imp…
▽ More
We have added support for realistic, microphysical, finite-temperature equations of state (EOS) and neutrino physics via a leakage scheme to IllinoisGRMHD, an open-source GRMHD code for dynamical spacetimes in the Einstein Toolkit. These new features are provided by two new, NRPy+-based codes: NRPyEOS, which performs highly efficient EOS table lookups and interpolations, and NRPyLeakage, which implements a new, AMR-capable neutrino leakage scheme in the Einstein Toolkit. We have performed a series of strenuous validation tests that demonstrate the robustness of these new codes, particularly on the Cartesian AMR grids provided by Carpet. Furthermore, we show results from fully dynamical GRMHD simulations of single unmagnetized neutron stars, and magnetized binary neutron star mergers. This new version of IllinoisGRMHD, as well as NRPyEOS and NRPyLeakage, is pedagogically documented in Jupyter notebooks and fully open source. The codes will be proposed for inclusion in an upcoming version of the Einstein Toolkit.
△ Less
Submitted 14 December, 2023; v1 submitted 30 August, 2022;
originally announced August 2022.
-
Handing off the outcome of binary neutron star mergers for accurate and long-term post-merger simulations
Authors:
Federico G. Lopez Armengol,
Zachariah B. Etienne,
Scott C. Noble,
Bernard J. Kelly,
Leonardo R. Werneck,
Brendan Drachler,
Manuela Campanelli,
Federico Cipolletta,
Yosef Zlochower,
Ariadna Murguia-Berthier,
Lorenzo Ennoggi,
Mark Avara,
Riccardo Ciolfi,
Joshua Faber,
Grace Fiacco,
Bruno Giacomazzo,
Tanmayee Gupte,
Trung Ha,
Julian H. Krolik,
Vassilios Mewes,
Richard O'Shaughnessy,
Jesús M. Rueda-Becerril,
Jeremy Schnittman
Abstract:
We perform binary neutron star (BNS) merger simulations in full dynamical general relativity with IllinoisGRMHD, on a Cartesian grid with adaptive-mesh refinement. After the remnant black hole has become nearly stationary, the evolution of the surrounding accretion disk on Cartesian grids over long timescales (1s) is suboptimal, as Cartesian coordinates over-resolve the angular coordinates at larg…
▽ More
We perform binary neutron star (BNS) merger simulations in full dynamical general relativity with IllinoisGRMHD, on a Cartesian grid with adaptive-mesh refinement. After the remnant black hole has become nearly stationary, the evolution of the surrounding accretion disk on Cartesian grids over long timescales (1s) is suboptimal, as Cartesian coordinates over-resolve the angular coordinates at large distances, and the accreting plasma flows obliquely across coordinate lines dissipating angular momentum artificially from the disk. To address this, we present the Handoff, a set of computational tools that enables the transfer of general relativistic magnetohydrodynamic (GRMHD) and spacetime data from IllinoisGRMHD to HARM3D, a GRMHD code that specializes in modeling black hole accretion disks in static spacetimes over long timescales, making use of general coordinate systems with spherical topology. We demonstrate that the Handoff allows for a smooth and reliable transition of GRMHD fields and spacetime data, enabling us to efficiently and reliably evolve BNS dynamics well beyond merger. We also discuss future plans, which involve incorporating advanced equations of state and neutrino physics into BNS simulations using the \handoff approach.
△ Less
Submitted 31 October, 2022; v1 submitted 17 December, 2021;
originally announced December 2021.
-
Implementing a new recovery scheme for primitive variables in the general relativistic magnetohydrodynamic code Spritz
Authors:
Jay V. Kalinani,
Riccardo Ciolfi,
Wolfgang Kastaun,
Bruno Giacomazzo,
Federico Cipolletta,
Lorenzo Ennoggi
Abstract:
General relativistic magnetohydrodynamic (GRMHD) simulations represent a fundamental tool to probe various underlying mechanisms at play during binary neutron star (BNS) and neutron star (NS) - black hole (BH) mergers. Contemporary flux-conservative GRMHD codes numerically evolve a set of conservative equations based on `conserved' variables which then need to be converted back into the fundamenta…
▽ More
General relativistic magnetohydrodynamic (GRMHD) simulations represent a fundamental tool to probe various underlying mechanisms at play during binary neutron star (BNS) and neutron star (NS) - black hole (BH) mergers. Contemporary flux-conservative GRMHD codes numerically evolve a set of conservative equations based on `conserved' variables which then need to be converted back into the fundamental (`primitive') variables. The corresponding conservative-to-primitive variable recovery procedure, based on root-finding algorithms, constitutes one of the core elements of such GRMHD codes. Recently, a new robust, accurate and efficient recovery scheme called RePrimAnd was introduced, which has demonstrated the ability to always converge to a unique solution. The scheme provides fine-grained error policies to handle invalid states caused by evolution errors, and also provides analytical bounds for the error of all primitive variables. In this work, we describe the technical aspects of implementing the RePrimAnd scheme into the GRMHD code Spritz. To check our implementation as well as to assess the various features of the scheme, we perform a number of GRMHD tests in three dimensions. Our tests, which include critical cases such as a NS collapse to a BH as well as the early evolution (~50 ms) of a Fishbone-Moncrief BH-accrection disk system, show that RePrimAnd is able to support magnetized, low density environments with magnetic-to-fluid pressure ratios as high as 10^4, in situations where the previously used recovery scheme fails.
△ Less
Submitted 12 May, 2022; v1 submitted 22 July, 2021;
originally announced July 2021.
-
HARM3D+NUC: A new method for simulating the post-merger phase of binary neutron star mergers with GRMHD, tabulated EOS and neutrino leakage
Authors:
Ariadna Murguia-Berthier,
Scott C. Noble,
Luke F. Roberts,
Enrico Ramirez-Ruiz,
Leonardo R. Werneck,
Michael Kolacki,
Zachariah B. Etienne,
Mark Avara,
Manuela Campanelli,
Riccardo Ciolfi,
Federico Cipolletta,
Brendan Drachler,
Lorenzo Ennoggi,
Joshua Faber,
Grace Fiacco,
Bruno Giacomazzo,
Tanmayee Gupte,
Trung Ha,
Bernard J. Kelly,
Julian H. Krolik,
Federico G. Lopez Armengol,
Ben Margalit,
Tim Moon,
Richard O'Shaughnessy,
Jesús M. Rueda-Becerril
, et al. (3 additional authors not shown)
Abstract:
The first binary neutron star merger has already been detected in gravitational waves. The signal was accompanied by an electromagnetic counterpart including a kilonova component powered by the decay of radioactive nuclei, as well as a short $γ$-ray burst. In order to understand the radioactively-powered signal, it is necessary to simulate the outflows and their nucleosynthesis from the post-merge…
▽ More
The first binary neutron star merger has already been detected in gravitational waves. The signal was accompanied by an electromagnetic counterpart including a kilonova component powered by the decay of radioactive nuclei, as well as a short $γ$-ray burst. In order to understand the radioactively-powered signal, it is necessary to simulate the outflows and their nucleosynthesis from the post-merger disk. Simulating the disk and predicting the composition of the outflows requires general relativistic magnetohydrodynamical (GRMHD) simulations that include a realistic, finite-temperature equation of state (EOS) and self-consistently calculating the impact of neutrinos. In this work, we detail the implementation of a finite-temperature EOS and the treatment of neutrinos in the GRMHD code HARM3D+NUC, based on HARM3D. We include formal tests of both the finite-temperature EOS and the neutrino leakage scheme. We further test the code by showing that, given conditions similar to those of published remnant disks following neutron star mergers, it reproduces both recombination of free nucleons to a neutron-rich composition and excitation of a thermal wind.
△ Less
Submitted 31 January, 2022; v1 submitted 9 June, 2021;
originally announced June 2021.
