-
AsterX: a new open-source GPU-accelerated GRMHD code for dynamical spacetimes
Authors:
Jay V. Kalinani,
Liwei Ji,
Lorenzo Ennoggi,
Federico G. Lopez Armengol,
Lucas Timotheo Sanches,
Bing-Jyun Tsao,
Steven R Brandt,
Manuela Campanelli,
Riccardo Ciolfi,
Bruno Giacomazzo,
Roland Haas,
Erik Schnetter,
Yosef Zlochower
Abstract:
We present AsterX, a novel open-source, GPU-accelerated, fully general relativistic magnetohydrodynamic (GRMHD) code designed for dynamic spacetimes in 3D Cartesian coordinates, and tailored for exascale computing. We utilize block-structured adaptive mesh refinement (AMR) through CarpetX, the new driver for the Einstein Toolkit, which is built on AMReX, a software framework for massively parallel…
▽ More
We present AsterX, a novel open-source, GPU-accelerated, fully general relativistic magnetohydrodynamic (GRMHD) code designed for dynamic spacetimes in 3D Cartesian coordinates, and tailored for exascale computing. We utilize block-structured adaptive mesh refinement (AMR) through CarpetX, the new driver for the Einstein Toolkit, which is built on AMReX, a software framework for massively parallel applications. AsterX employs the Valencia formulation for GRMHD, coupled with the 'Z4c' formalism for spacetime evolution, while incorporating high resolution shock capturing schemes to accurately handle the hydrodynamics. AsterX has undergone rigorous testing in both static and dynamic spacetime, demonstrating remarkable accuracy and agreement with other codes in literature. Benchmarking the code through scaling tests on OLCF's Frontier supercomputer, we demonstrate a weak scaling efficiency of about 67%-77% on 4096 nodes compared to an 8-node performance.
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
HPX with Spack and Singularity Containers: Evaluating Overheads for HPX/Kokkos using an astrophysics application
Authors:
Patrick Diehl,
Steven R. Brandt,
Gregor Daiß,
Hartmut Kaiser
Abstract:
Cloud computing for high performance computing resources is an emerging topic. This service is of interest to researchers who care about reproducible computing, for software packages with complex installations, and for companies or researchers who need the compute resources only occasionally or do not want to run and maintain a supercomputer on their own. The connection between HPC and containers…
▽ More
Cloud computing for high performance computing resources is an emerging topic. This service is of interest to researchers who care about reproducible computing, for software packages with complex installations, and for companies or researchers who need the compute resources only occasionally or do not want to run and maintain a supercomputer on their own. The connection between HPC and containers is exemplified by the fact that Microsoft Azure's Eagle cloud service machine is number three on the November 23 Top 500 list. For cloud services, the HPC application and dependencies are installed in containers, e.g. Docker, Singularity, or something else, and these containers are executed on the physical hardware. Although containerization leverages the existing Linux kernel and should not impose overheads on the computation, there is the possibility that machine-specific optimizations might be lost, particularly machine-specific installs of commonly used packages. In this paper, we will use an astrophysics application using HPX-Kokkos and measure overheads on homogeneous resources, e.g. Supercomputer Fugaku, using CPUs only and on heterogenous resources, e.g. LSU's hybrid CPU and GPU system. We will report on challenges in compiling, running, and using the containers as well as performance performance differences.
△ Less
Submitted 7 May, 2024; v1 submitted 11 February, 2024;
originally announced May 2024.
-
GRaM-X: A new GPU-accelerated dynamical spacetime GRMHD code for Exascale computing with the Einstein Toolkit
Authors:
Swapnil Shankar,
Philipp Mösta,
Steven R. Brandt,
Roland Haas,
Erik Schnetter,
Yannick de Graaf
Abstract:
We present GRaM-X (General Relativistic accelerated Magnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems. GRaM-X supports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit called CarpetX which in turn…
▽ More
We present GRaM-X (General Relativistic accelerated Magnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems. GRaM-X supports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit called CarpetX which in turn leverages AMReX, an AMR library developed for use by the United States DOE's Exascale Computing Project (ECP). We use the Z4c formalism to evolve the equations of GR and the Valencia formulation to evolve the equations of GRMHD. GRaM-X supports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D MHD shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D TOV star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find that GRaM-X shows a weak scaling efficiency of $\sim 40-50\%$ on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF's supercomputer Summit.
△ Less
Submitted 21 November, 2022; v1 submitted 31 October, 2022;
originally announced October 2022.
