Showing 1–32 of 32 results for author: Frontiere, N

Exploring the Core-galaxy Connection

Authors: Isabele Souza Vitório, Michael Buehlmann, Eve Kovacs, Patricia Larsen, Nicholas Frontiere, Katrin Heitmann

Abstract: Halo core tracking is a novel concept designed to efficiently follow halo substructure in large simulations. We have recently developed this concept in gravity-only simulations to investigate the galaxy-halo connection in the context of empirical and semi-analytic models. Here, we incorporate information from hydrodynamics simulations, with an emphasis on establishing a connection between cores an… ▽ More Halo core tracking is a novel concept designed to efficiently follow halo substructure in large simulations. We have recently developed this concept in gravity-only simulations to investigate the galaxy-halo connection in the context of empirical and semi-analytic models. Here, we incorporate information from hydrodynamics simulations, with an emphasis on establishing a connection between cores and galaxies. We compare cores across gravity-only, adiabatic hydrodynamics, and subgrid hydrodynamics simulations with the same initial phases. We demonstrate that cores are stable entities whose halo-centric radial profiles match across the simulations. We further develop a methodology that uses merging and infall mass cuts to group cores in the hydrodynamics simulation, creating on average, a one-to-one match to corresponding galaxies. We apply this methodology to cores from the gravity-only simulation, thus creating a proxy for galaxies which approximate the populations from the hydrodynamics simulation. Our results pave the way to incorporate inputs from smaller-scale hydrodynamics simulations directly into large-scale gravity-only runs in a principled manner. △ Less

Submitted 3 July, 2024; v1 submitted 28 June, 2024; originally announced July 2024.

Comments: 14 pages, 13 figures

The New Worlds Simulations: Large-scale Simulations across Three Cosmologies

Authors: Katrin Heitmann, Thomas Uram, Nicholas Frontiere, Salman Habib, Adrian Pope, Silvio Rizzi, Joe Insley

Abstract: In this paper we describe the set of ``New Worlds Simulations'', three very large cosmology simulations, Qo'noS, Vulcan, and Ferenginar, that were carried out on the Summit supercomputer with the Hardware/Hybrid Cosmology Code, HACC. The gravity-only simulations follow the evolution of structure in the Universe by each employing 12,288^3 particles in (3 Gpc/h)^3 volumes, leading to a mass resoluti… ▽ More In this paper we describe the set of ``New Worlds Simulations'', three very large cosmology simulations, Qo'noS, Vulcan, and Ferenginar, that were carried out on the Summit supercomputer with the Hardware/Hybrid Cosmology Code, HACC. The gravity-only simulations follow the evolution of structure in the Universe by each employing 12,288^3 particles in (3 Gpc/h)^3 volumes, leading to a mass resolution of m_p~10^9 Msun/h. The simulations cover three different cosmologies, one LambdaCDM model, consistent with measurements from Planck, one simulation with massive neutrinos, and one simulation with a varying dark energy equation of state. All simulations have the same phases to allow a detailed comparison of the results and the investigation of the impact of different cosmological parameters. We present measurements of some basic statistics, such as matter power spectra, correlation function, halo mass function and concentration-mass relation and investigate the differences due to the varying cosmologies. Given the large volume and high resolution, these simulations provide excellent bases for creating synthetic skies. A subset of the data is made publicly available as part of this paper. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: 10 pages, 13 figures, simulation data portal: https://cosmology.alcf.anl.gov/transfer/newworlds

A Performance-Portable SYCL Implementation of CRK-HACC for Exascale

Authors: Esteban M. Rangel, S. John Pennycook, Adrian Pope, Nicholas Frontiere, Zhiqiang Ma, Varsha Madananth

Abstract: The first generation of exascale systems will include a variety of machine architectures, featuring GPUs from multiple vendors. As a result, many developers are interested in adopting portable programming models to avoid maintaining multiple versions of their code. It is necessary to document experiences with such programming models to assist developers in understanding the advantages and disadvan… ▽ More The first generation of exascale systems will include a variety of machine architectures, featuring GPUs from multiple vendors. As a result, many developers are interested in adopting portable programming models to avoid maintaining multiple versions of their code. It is necessary to document experiences with such programming models to assist developers in understanding the advantages and disadvantages of different approaches. To this end, this paper evaluates the performance portability of a SYCL implementation of a large-scale cosmology application (CRK-HACC) running on GPUs from three different vendors: AMD, Intel, and NVIDIA. We detail the process of migrating the original code from CUDA to SYCL and show that specializing kernels for specific targets can greatly improve performance portability without significantly impacting programmer productivity. The SYCL version of CRK-HACC achieves a performance portability of 0.96 with a code divergence of almost 0, demonstrating that SYCL is a viable programming model for performance-portable applications. △ Less

Submitted 24 October, 2023; originally announced October 2023.

Comments: 12 pages, 13 figures, 2023 International Workshop on Performance, Portability & Productivity in HPC

ACM Class: D.2.7; D.2.8; D.1.3; J.2

Experiences Readying Applications for Exascale

Authors: Paul T. Bauman, Reuben D. Budiardja, Dmytro Bykov, Noel Chalmers, Jacqueline Chen, Nicholas Curtis, Marc Day, Markus Eisenbach, Lucas Esclapez, Alessandro Fanfarillo, William Freitag, Nicholas Frontiere, Antigoni Georgiadou, Joseph Glenski, Kalyana Gottiparthi, Marc T. Henry de Frahan, Gustav R. Jansen, Wayne Joubert, Justin G. Lietz, Jakub Kurzak, Nicholas Malaya, Bronson Messer, Damon McDougall, Paul Mullowney, Stephen Nichols , et al. (7 additional authors not shown)

Abstract: The advent of exascale computing invites an assessment of existing best practices for developing application readiness on the world's largest supercomputers. This work details observations from the last four years in preparing scientific applications to run on the Oak Ridge Leadership Computing Facility's (OLCF) Frontier system. This paper addresses a range of topics in software including programm… ▽ More The advent of exascale computing invites an assessment of existing best practices for developing application readiness on the world's largest supercomputers. This work details observations from the last four years in preparing scientific applications to run on the Oak Ridge Leadership Computing Facility's (OLCF) Frontier system. This paper addresses a range of topics in software including programmability, tuning, and portability considerations that are key to moving applications from existing systems to future installations. A set of representative workloads provides case studies for general system and software testing. We evaluate the use of early access systems for development across several generations of hardware. Finally, we discuss how best practices were identified and disseminated to the community through a wide range of activities including user-guides and trainings. We conclude with recommendations for ensuring application readiness on future leadership computing systems. △ Less

Submitted 2 October, 2023; originally announced October 2023.

Comments: Accepted at SC23

Optimization and Quality Assessment of Baryon Pasting for Intracluster Gas using the Borg Cube Simulation

Authors: F. Kéruzoré, L. E. Bleem, M. Buehlmann, J. D. Emberson, N. Frontiere, S. Habib, K. Heitmann, P. Larsen

Abstract: Synthetic datasets generated from large-volume gravity-only simulations are an important tool in the calibration of cosmological analyses. Their creation often requires accurate inference of baryonic observables from the dark matter field. We explore the effectiveness of a baryon pasting algorithm in providing precise estimations of three-dimensional gas thermodynamic properties based on gravity-o… ▽ More Synthetic datasets generated from large-volume gravity-only simulations are an important tool in the calibration of cosmological analyses. Their creation often requires accurate inference of baryonic observables from the dark matter field. We explore the effectiveness of a baryon pasting algorithm in providing precise estimations of three-dimensional gas thermodynamic properties based on gravity-only simulations. We use the Borg Cube, a pair of simulations originating from identical initial conditions, with one run evolved as a gravity-only simulation, and the other incorporating non-radiative hydrodynamics. Matching halos in both simulations enables comparisons of gas properties on an individual halo basis. This comparative analysis allows us to fit for the model parameters that yield the closest agreement between the gas properties in both runs. To capture the redshift evolution of these parameters, we perform the analysis at five distinct redshift steps, spanning from $z=0$ to $2$. We find that the investigated algorithm, utilizing information solely from the gravity-only simulation, achieves few-percent accuracy in reproducing the median intracluster gas pressure and density, albeit with a scatter of approximately 20%, for cluster-scale objects up to $z=2$. We measure the scaling relation between integrated Compton parameter and cluster mass ($Y_{500c} | M_{500c}$), and find that the imprecision of baryon pasting adds less than 5% to the intrinsic scatter measured in the hydrodynamic simulation. We provide best-fitting values and their redshift evolution, and discuss future investigations that will be undertaken to extend this work. △ Less

Submitted 27 November, 2023; v1 submitted 23 June, 2023; originally announced June 2023.

Comments: 14 pages, 8 figures, 3 tables; accepted in the Open Journal of Astrophysics

Numerical Discreteness Errors in Multi-Species Cosmological N-body Simulations

Authors: Xin Liu, J. D. Emberson, Michael Buehlmann, Nicholas Frontiere, Salman Habib

Abstract: We present a detailed analysis of numerical discreteness errors in two-species, gravity-only, cosmological simulations using the density power spectrum as a diagnostic probe. In a simple setup where both species are initialized with the same total matter transfer function, biased growth of power forms on small scales when the solver force resolution is finer than the mean interparticle separation.… ▽ More We present a detailed analysis of numerical discreteness errors in two-species, gravity-only, cosmological simulations using the density power spectrum as a diagnostic probe. In a simple setup where both species are initialized with the same total matter transfer function, biased growth of power forms on small scales when the solver force resolution is finer than the mean interparticle separation. The artificial bias is more severe when individual density and velocity transfer functions are applied. In particular, significant large-scale offsets in power are measured between simulations with conventional offset grid initial conditions when compared against converged high-resolution results where the force resolution scale is matched to the interparticle separation. These offsets persist even when the cosmology is chosen so that the two particle species have the same mass, indicating that the error is sourced from discreteness in the total matter field as opposed to unequal particle mass. We further investigate two mitigation strategies to address discreteness errors: the frozen potential method and softened interspecies short-range forces. The former evolves particles under the approximately "frozen" total matter potential in linear theory at early times, while the latter filters cross-species gravitational interactions on small scales in low density regions. By modeling closer to the continuum limit, both mitigation strategies demonstrate considerable reductions in large-scale power spectrum offsets. △ Less

Submitted 18 April, 2023; v1 submitted 1 March, 2023; originally announced March 2023.

Comments: Accepted for publication in MNRAS

Galaxy Clustering in the Mira-Titan Universe I: Emulators for the redshift space galaxy correlation function and galaxy-galaxy lensing

Authors: Juliana Kwan, Shun Saito, Alexie Leauthaud, Katrin Heitmann, Salman Habib, Nicholas Frontiere, Hong Guo, Song Huang, Adrian Pope, Sergio Rodríguez-Torres

Abstract: We construct accurate emulators for the projected and redshift space galaxy correlation functions and excess surface density as measured by galaxy-galaxy lensing, based on Halo Occupation Distribution (HOD) modeling. Using the complete Mira-Titan suite of 111 $N$-body simulations, our emulators vary over eight cosmological parameters and include the effects of neutrino mass and dynamical dark ener… ▽ More We construct accurate emulators for the projected and redshift space galaxy correlation functions and excess surface density as measured by galaxy-galaxy lensing, based on Halo Occupation Distribution (HOD) modeling. Using the complete Mira-Titan suite of 111 $N$-body simulations, our emulators vary over eight cosmological parameters and include the effects of neutrino mass and dynamical dark energy. We demonstrate that our emulators are sufficiently accurate for the analysis of the BOSS DR12 CMASS galaxy sample over the range 0.5 < r < 50 Mpc/h. Furthermore, we show that our emulators are capable of recovering unbiased cosmological constraints from realistic mock catalogs over the same range. Our mock catalog tests show the efficacy of combining small scale galaxy-galaxy lensing with redshift space clustering and that we can constrain the growth rate and σ_8 to 7% and 4.5% respectively for a CMASS-like sample using only the measurements covered by our emulator. With the inclusion of a CMB prior on H_0, this reduces to a 2% measurement on the growth rate. △ Less

Submitted 6 June, 2023; v1 submitted 23 February, 2023; originally announced February 2023.

Comments: 31 pages, 13 figures, revised to match published version

Improving initialization and evolution accuracy of cosmological neutrino simulations

Authors: James M. Sullivan, J. D. Emberson, Salman Habib, Nicholas Frontiere

Abstract: Neutrino mass constraints are a primary focus of current and future large-scale structure (LSS) surveys. Non-linear LSS models rely heavily on cosmological simulations -- the impact of massive neutrinos should therefore be included in these simulations in a realistic, computationally tractable, and controlled manner. A recent proposal to reduce the related computational cost employs a symmetric ne… ▽ More Neutrino mass constraints are a primary focus of current and future large-scale structure (LSS) surveys. Non-linear LSS models rely heavily on cosmological simulations -- the impact of massive neutrinos should therefore be included in these simulations in a realistic, computationally tractable, and controlled manner. A recent proposal to reduce the related computational cost employs a symmetric neutrino momentum sampling strategy in the initial conditions. We implement a modified version of this strategy into the Hardware/Hybrid Accelerated Cosmology Code (HACC) and perform convergence tests on its internal parameters. We illustrate that this method can impart $\mathcal{O}(1\%)$ numerical artifacts on the total matter field on small scales, similar to previous findings, and present a method to remove these artifacts using Fourier-space filtering of the neutrino density field. Moreover, we show that the converged neutrino power spectrum does not follow linear theory predictions on relatively large scales at early times at the $15\%$ level, prompting a more careful study of systematics in particle-based neutrino simulations. We also present an improved method for backscaling linear transfer functions for initial conditions in massive neutrino cosmologies that is based on achieving the same relative neutrino growth as computed with Boltzmann solvers. Our self-consistent backscaling method yields sub-percent accuracy in the total matter growth function. Comparisons for the non-linear power spectrum with the Mira-Titan emulator at a neutrino mass of $m_ν=0.15~\mathrm{eV}$ are in very good agreement with the expected level of errors in the emulator and in the direct N-body simulation. △ Less

Submitted 10 June, 2024; v1 submitted 17 February, 2023; originally announced February 2023.

Comments: 33 pages, 8 figures, 1 table. Published version

Journal ref: JCAP Volume 2023, Issue 06, id.003, 34 pp

Modeling the Galaxy Distribution in Clusters using Halo Cores

Authors: Danila Korytov, Esteban Rangel, Lindsey Bleem, Nicholas Frontiere, Salman Habib, Katrin Heitmann, Joseph Hollowed, Adrian Pope

Abstract: The galaxy distribution in dark matter-dominated halos is expected to approximately trace the details of the underlying dark matter substructure. In this paper we introduce halo `core-tracking' as a way to efficiently follow the small-scale substructure in cosmological simulations and apply the technique to model the galaxy distribution in observed clusters. The method relies on explicitly trackin… ▽ More The galaxy distribution in dark matter-dominated halos is expected to approximately trace the details of the underlying dark matter substructure. In this paper we introduce halo `core-tracking' as a way to efficiently follow the small-scale substructure in cosmological simulations and apply the technique to model the galaxy distribution in observed clusters. The method relies on explicitly tracking the set of particles identified as belonging to a halo's central density core, once a halo has attained a certain threshold mass. The halo cores are then followed throughout the entire evolution of the simulation. The aim of core-tracking is to simplify substructure analysis tasks by avoiding the use of subhalos and, at the same time, to more easily account for the so-called ``orphan'' galaxies, which have lost substantial dark mass due to tidal stripping. We show that simple models based on halo cores can reproduce the number and spatial distribution of galaxies found in optically-selected clusters in the Sloan Digital Sky Survey. We also discuss future applications of the core-tracking methodology in studying the galaxy-halo connection. △ Less

Submitted 11 July, 2023; v1 submitted 8 February, 2023; originally announced February 2023.

Comments: 17 pages, 20 figures, 1 Appendix; version accepted by OJA

Journal ref: Open Journal of Astrophysics 6 (2023)

Modeling the Lyman-$α$ forest with Eulerian and SPH hydrodynamical methods

Authors: Solène Chabanier, J. D. Emberson, Zarija Lukić, Jesus Pulido, Salman Habib, Esteban Rangel, Jean Sexton, Nicholas Frontiere, Michael Buehlmann

Abstract: We compare two state-of-the-art numerical codes to study the overall accuracy in modeling the intergalactic medium and reproducing Lyman-$α$ forest observables for DESI and high-resolution data sets. The codes employ different approaches to solving both gravity and modeling the gas hydrodynamics. The first code, Nyx, solves the Poisson equation using the Particle-Mesh (PM) method and the Euler equ… ▽ More We compare two state-of-the-art numerical codes to study the overall accuracy in modeling the intergalactic medium and reproducing Lyman-$α$ forest observables for DESI and high-resolution data sets. The codes employ different approaches to solving both gravity and modeling the gas hydrodynamics. The first code, Nyx, solves the Poisson equation using the Particle-Mesh (PM) method and the Euler equations using a finite volume method. The second code, \CRKHACC, uses a Tree-PM method to solve for gravity, and an improved Lagrangian smoothed particle hydrodynamics (SPH) technique, where fluid elements are modeled with particles, to treat the intergalactic gas. We compare the convergence behavior of the codes in flux statistics as well as the degree to which the codes agree in the converged limit. We find good agreement overall with differences being less than observational uncertainties, and a particularly notable $\lesssim$1\% agreement in the 1D flux power spectrum. This agreement was achieved by applying a tessellation methodology for reconstructing the density in \CRKHACC instead of using an SPH kernel as is standard practice. We show that use of the SPH kernel can lead to significant and unnecessary biases in flux statistics; this is especially prominent at high redshifts, $z \sim 5$, as the Lyman-$α$ forest mostly comes from lower-density regions which are intrinsically poorly sampled by SPH particles. △ Less

Submitted 11 July, 2022; originally announced July 2022.

Comments: submitted to MNRAS

Simulating Hydrodynamics in Cosmology with CRK-HACC

Authors: Nicholas Frontiere, J. D. Emberson, Michael Buehlmann, Joseph Adamo, Salman Habib, Katrin Heitmann, Claude-André Faucher-Giguère

Abstract: We introduce CRK-HACC, an extension of the Hardware/Hybrid Accelerated Cosmology Code (HACC), to resolve gas hydrodynamics in large-scale structure formation simulations of the universe. The new framework couples the HACC gravitational N-body solver with a modern smoothed particle hydrodynamics (SPH) approach called CRKSPH. $\underline{\text{C}}$onservative $\underline{\text{R}}$eproducing… ▽ More We introduce CRK-HACC, an extension of the Hardware/Hybrid Accelerated Cosmology Code (HACC), to resolve gas hydrodynamics in large-scale structure formation simulations of the universe. The new framework couples the HACC gravitational N-body solver with a modern smoothed particle hydrodynamics (SPH) approach called CRKSPH. $\underline{\text{C}}$onservative $\underline{\text{R}}$eproducing $\underline{\text{K}}$ernel $\underline{\text{SPH}}$ utilizes smoothing functions that exactly interpolate linear fields while manifestly preserving conservation laws (momentum, mass, and energy). The CRKSPH method has been incorporated to accurately model baryonic effects in cosmology simulations - an important addition targeting the generation of precise synthetic sky predictions for upcoming observational surveys. CRK-HACC inherits the codesign strategies of the HACC solver and is built to run on modern GPU-accelerated supercomputers. In this work, we summarize the primary solver components and present a number of standard validation tests to demonstrate code accuracy, including idealized hydrodynamic and cosmological setups, as well as self-similarity measurements. △ Less

Submitted 22 November, 2022; v1 submitted 6 February, 2022; originally announced February 2022.

Farpoint: A High-Resolution Cosmology Simulation at the Gigaparsec Scale

Authors: Nicholas Frontiere, Katrin Heitmann, Esteban Rangel, Patricia Larsen, Adrian Pope, Imran Sultan, Thomas Uram, Salman Habib, Silvio Rizzi, Joe Insley

Abstract: In this paper we introduce the Farpoint simulation, the latest member of the Hardware/Hybrid Accelerated Cosmology Code (HACC) gravity-only simulation family. The domain covers a volume of (1000$h^{-1}$Mpc)$^3$ and evolves close to two trillion particles, corresponding to a mass resolution of $m_p\sim 4.6\cdot 10^7 h^{-1}$M$_\odot$. These specifications enable comprehensive investigations of the g… ▽ More In this paper we introduce the Farpoint simulation, the latest member of the Hardware/Hybrid Accelerated Cosmology Code (HACC) gravity-only simulation family. The domain covers a volume of (1000$h^{-1}$Mpc)$^3$ and evolves close to two trillion particles, corresponding to a mass resolution of $m_p\sim 4.6\cdot 10^7 h^{-1}$M$_\odot$. These specifications enable comprehensive investigations of the galaxy-halo connection, capturing halos down to small masses. Further, the large volume resolves scales typical of modern surveys with good statistical coverage of high mass halos. The simulation was carried out on the GPU-accelerated system Summit, one of the fastest supercomputers currently available. We provide specifics about the Farpoint run and present an initial set of results. The high mass resolution facilitates precise measurements of important global statistics, such as the halo concentration-mass relation and the correlation function down to small scales. Selected subsets of the simulation data products are publicly available via the HACC Simulation Data Portal. △ Less

Submitted 28 February, 2022; v1 submitted 4 September, 2021; originally announced September 2021.

Journal ref: The Astrophysical Journal Supplement Series, 259(1):15, 2022

The Last Journey. II. SMACC -- Subhalo Mass-loss Analysis using Core Catalogs

Authors: Imran Sultan, Nicholas Frontiere, Salman Habib, Katrin Heitmann, Eve Kovacs, Patricia Larsen, Esteban Rangel

Abstract: This paper introduces SMACC -- Subhalo Mass-loss Analysis using Core Catalogs. SMACC adds a mass model to substructure merger trees based on halo "core tracking." Our approach avoids the need for running expensive subhalo finding algorithms and instead uses subhalo mass-loss modeling to assign masses to halo cores. We present details of the SMACC methodology and demonstrate its excellent performan… ▽ More This paper introduces SMACC -- Subhalo Mass-loss Analysis using Core Catalogs. SMACC adds a mass model to substructure merger trees based on halo "core tracking." Our approach avoids the need for running expensive subhalo finding algorithms and instead uses subhalo mass-loss modeling to assign masses to halo cores. We present details of the SMACC methodology and demonstrate its excellent performance in describing halo substructure and its evolution. Validation of the approach is carried out using cosmological simulations at significantly different resolutions. We apply SMACC to the 1.24 trillion-particle Last Journey simulation and construct core catalogs with the additional mass information. These catalogs can be readily used as input to semi-analytic models or subhalo abundance matching approaches to determine approximate galaxy distributions, as well as for in-depth studies of small-scale structure evolution. △ Less

Submitted 16 December, 2020; originally announced December 2020.

Comments: 15 pages, 10 figures. Submitted to ApJ

Building Halo Merger Trees from the Q Continuum Simulation

Authors: Esteban Rangel, Nicholas Frontiere, Salman Habib, Katrin Heitmann, Wei-keng Liao, Ankit Agrawal, Alok Choudhary

Abstract: Cosmological N-body simulations rank among the most computationally intensive efforts today. A key challenge is the analysis of structure, substructure, and the merger history for many billions of compact particle clusters, called halos. Effectively representing the merging history of halos is essential for many galaxy formation models used to generate synthetic sky catalogs, an important applicat… ▽ More Cosmological N-body simulations rank among the most computationally intensive efforts today. A key challenge is the analysis of structure, substructure, and the merger history for many billions of compact particle clusters, called halos. Effectively representing the merging history of halos is essential for many galaxy formation models used to generate synthetic sky catalogs, an important application of modern cosmological simulations. Generating realistic mock catalogs requires computing the halo formation history from simulations with large volumes and billions of halos over many time steps, taking hundreds of terabytes of analysis data. We present fast parallel algorithms for producing halo merger trees and tracking halo substructure from a single-level, density-based clustering algorithm. Merger trees are created from analyzing the halo-particle membership function in adjacent snapshots, and substructure is identified by tracking the "cores" of merging halos -- sets of particles near the halo center. Core tracking is performed after creating merger trees and uses the relationships found during tree construction to associate substructures with hosts. The algorithms are implemented with MPI and evaluated on a Cray XK7 supercomputer using up to 16,384 processes on data from HACC, a modern cosmological simulation framework. We present results for creating merger trees from 101 analysis snapshots taken from the Q Continuum, a large volume, high mass resolution, cosmological simulation evolving half a trillion particles. △ Less

Submitted 19 August, 2020; originally announced August 2020.

Comments: 2017 IEEE 24th International Conference on High Performance Computing

Journal ref: 2017 IEEE 24th International Conference on High Performance Computing (HiPC), pp. 398-407. IEEE, 2017

The Last Journey. I. An Extreme-Scale Simulation on the Mira Supercomputer

Authors: Katrin Heitmann, Nicholas Frontiere, Esteban Rangel, Patricia Larsen, Adrian Pope, Imran Sultan, Thomas Uram, Salman Habib, Hal Finkel, Danila Korytov, Eve Kovacs, Silvio Rizzi, Joe Insley

Abstract: The Last Journey is a large-volume, gravity-only, cosmological N-body simulation evolving more than 1.24 trillion particles in a periodic box with a side-length of 5.025Gpc. It was implemented using the HACC simulation and analysis framework on the BG/Q system, Mira. The cosmological parameters are chosen to be consistent with the results from the Planck satellite. A range of analysis tools have b… ▽ More The Last Journey is a large-volume, gravity-only, cosmological N-body simulation evolving more than 1.24 trillion particles in a periodic box with a side-length of 5.025Gpc. It was implemented using the HACC simulation and analysis framework on the BG/Q system, Mira. The cosmological parameters are chosen to be consistent with the results from the Planck satellite. A range of analysis tools have been run in situ to enable a diverse set of science projects, and at the same time, to keep the resulting data amount manageable. Analysis outputs have been generated starting at redshift z~10 to allow for construction of synthetic galaxy catalogs using a semi-analytic modeling approach in post-processing. As part of our in situ analysis pipeline we employ a new method for tracking halo sub-structures, introducing the concept of subhalo cores. The production of multi-wavelength synthetic sky maps is facilitated by generating particle lightcones in situ, also beginning at z~10. We provide an overview of the simulation set-up and the generated data products; a first set of analysis results is presented. A subset of the data is publicly available. △ Less

Submitted 8 January, 2021; v1 submitted 2 June, 2020; originally announced June 2020.

Comments: 14 pages, 9 figures. Accepted for publication in ApJS. New visualization and new results for the matter correlation function added, minor edits. The Last Journey data products can be accessed here: https://cosmology.alcf.anl.gov/

The Mira-Titan Universe. III. Emulation of the Halo Mass Function

Authors: Sebastian Bocquet, Katrin Heitmann, Salman Habib, Earl Lawrence, Thomas Uram, Nicholas Frontiere, Adrian Pope, Hal Finkel

Abstract: We construct an emulator for the halo mass function over group and cluster mass scales for a range of cosmologies, including the effects of dynamical dark energy and massive neutrinos. The emulator is based on the recently completed Mira-Titan Universe suite of cosmological $N$-body simulations. The main set of simulations spans 111 cosmological models with 2.1 Gpc boxes. We extract halo catalogs… ▽ More We construct an emulator for the halo mass function over group and cluster mass scales for a range of cosmologies, including the effects of dynamical dark energy and massive neutrinos. The emulator is based on the recently completed Mira-Titan Universe suite of cosmological $N$-body simulations. The main set of simulations spans 111 cosmological models with 2.1 Gpc boxes. We extract halo catalogs in the redshift range $z=[0.0, 2.0]$ and for masses $M_{200\mathrm{c}}\geq 10^{13}M_\odot/h$. The emulator covers an 8-dimensional hypercube spanned by {$Ω_\mathrm{m}h^2$, $Ω_\mathrm{b}h^2$, $Ω_νh^2$, $σ_8$, $h$, $n_s$, $w_0$, $w_a$}; spatial flatness is assumed. We obtain smooth halo mass functions by fitting piecewise second-order polynomials to the halo catalogs and employ Gaussian process regression to construct the emulator while keeping track of the statistical noise in the input halo catalogs and uncertainties in the regression process. For redshifts $z\lesssim1$, the typical emulator precision is better than $2\%$ for $10^{13}-10^{14} M_\odot/h$ and $<10\%$ for $M\simeq 10^{15}M_\odot/h$. For comparison, fitting functions using the traditional universal form for the halo mass function can be biased at up to 30\% at $M\simeq 10^{14}M_\odot/h$ for $z=0$. Our emulator is publicly available at \url{https://github.com/SebastianBocquet/MiraTitanHMFemulator}. △ Less

Submitted 5 August, 2020; v1 submitted 26 March, 2020; originally announced March 2020.

Comments: 22 pages, 10 figures, 2 tables. Accepted for publication in ApJ (v2). For associated emulator code, see https://github.com/SebastianBocquet/MiraTitanHMFemulator

CosmoDC2: A Synthetic Sky Catalog for Dark Energy Science with LSST

Authors: Danila Korytov, Andrew Hearin, Eve Kovacs, Patricia Larsen, Esteban Rangel, Joseph Hollowed, Andrew J. Benson, Katrin Heitmann, Yao-Yuan Mao, Anita Bahmanyar, Chihway Chang, Duncan Campbell, Joseph Derose, Hal Finkel, Nicholas Frontiere, Eric Gawiser, Salman Habib, Benjamin Joachimi, François Lanusse, Nan Li, Rachel Mandelbaum, Christopher Morrison, Jeffrey A. Newman, Adrian Pope, Eli Rykoff , et al. (5 additional authors not shown)

Abstract: This paper introduces cosmoDC2, a large synthetic galaxy catalog designed to support precision dark energy science with the Large Synoptic Survey Telescope (LSST). CosmoDC2 is the starting point for the second data challenge (DC2) carried out by the LSST Dark Energy Science Collaboration (LSST DESC). The catalog is based on a trillion-particle, 4.225 Gpc^3 box cosmological N-body simulation, the `… ▽ More This paper introduces cosmoDC2, a large synthetic galaxy catalog designed to support precision dark energy science with the Large Synoptic Survey Telescope (LSST). CosmoDC2 is the starting point for the second data challenge (DC2) carried out by the LSST Dark Energy Science Collaboration (LSST DESC). The catalog is based on a trillion-particle, 4.225 Gpc^3 box cosmological N-body simulation, the `Outer Rim' run. It covers 440 deg^2 of sky area to a redshift of z=3 and is complete to a magnitude depth of 28 in the r-band. Each galaxy is characterized by a multitude of properties including stellar mass, morphology, spectral energy distributions, broadband filter magnitudes, host halo information and weak lensing shear. The size and complexity of cosmoDC2 requires an efficient catalog generation methodology; our approach is based on a new hybrid technique that combines data-driven empirical approaches with semi-analytic galaxy modeling. A wide range of observation-based validation tests has been implemented to ensure that cosmoDC2 enables the science goals of the planned LSST DESC DC2 analyses. This paper also represents the official release of the cosmoDC2 data set, including an efficient reader that facilitates interaction with the data. △ Less

Submitted 27 July, 2019; v1 submitted 15 July, 2019; originally announced July 2019.

Comments: 27 pages, 17 figures, submitted to APJS

The Outer Rim Simulation: A Path to Many-Core Supercomputers

Authors: Katrin Heitmann, Hal Finkel, Adrian Pope, Vitali Morozov, Nicholas Frontiere, Salman Habib, Esteban Rangel, Thomas Uram, Danila Korytov, Hillary Child, Samuel Flender, Joe Insley, Silvio Rizzi

Abstract: We describe the Outer Rim cosmological simulation, one of the largest high-resolution N-body simulations performed to date, aimed at promoting science to be carried out with large-scale structure surveys. The simulation covers a volume of (4.225Gpc)^3 and evolves more than one trillion particles. It was executed on Mira, a BlueGene/Q system at the Argonne Leadership Computing Facility. We discuss… ▽ More We describe the Outer Rim cosmological simulation, one of the largest high-resolution N-body simulations performed to date, aimed at promoting science to be carried out with large-scale structure surveys. The simulation covers a volume of (4.225Gpc)^3 and evolves more than one trillion particles. It was executed on Mira, a BlueGene/Q system at the Argonne Leadership Computing Facility. We discuss some of the computational challenges posed by a system like Mira, a many-core supercomputer, and how the simulation code, HACC, has been designed to overcome these challenges. We have carried out a large range of analyses on the simulation data and we report on the results as well as the data products that have been generated. The full data set generated by the simulation totals more than 5PB of data, making data curation and data handling a large challenge in of itself. The simulation results have been used to generate synthetic catalogs for large-scale structure surveys, including DESI and eBOSS, as well as CMB experiments. A detailed catalog for the LSST DESC data challenges has been created as well. We publicly release some of the Outer Rim halo catalogs, downsampled particle information, and lightcone data. △ Less

Submitted 28 April, 2019; v1 submitted 26 April, 2019; originally announced April 2019.

Comments: 10 pages, 10 figures. Submitted to ApJS. The Outer Rim data products can be accessed here: https://cosmology.alcf.anl.gov/

HACC Cosmological Simulations: First Data Release

Authors: Katrin Heitmann, Thomas D. Uram, Hal Finkel, Nicholas Frontiere, Salman Habib, Adrian Pope, Esteban Rangel, Joseph Hollowed, Danila Korytov, Patricia Larsen, Benjamin S. Allen, Kyle Chard, Ian Foster

Abstract: We describe the first major public data release from cosmological simulations carried out with Argonne's HACC code. This initial release covers a range of datasets from large gravity-only simulations. The data products include halo information for multiple redshifts, down-sampled particles, and lightcone outputs. We provide data from two very large LCDM simulations as well as beyond-LCDM simulatio… ▽ More We describe the first major public data release from cosmological simulations carried out with Argonne's HACC code. This initial release covers a range of datasets from large gravity-only simulations. The data products include halo information for multiple redshifts, down-sampled particles, and lightcone outputs. We provide data from two very large LCDM simulations as well as beyond-LCDM simulations spanning eleven w0-wa cosmologies. Our release platform uses Petrel, a research data service, located at the Argonne Leadership Computing Facility. Petrel offers fast data transfer mechanisms and authentication via Globus, enabling simple and efficient access to stored datasets. Easy browsing of the available data products is provided via a web portal that allows the user to navigate simulation products efficiently. The data hub will be extended by adding more types of data products and by enabling computational capabilities to allow direct interactions with simulation results. △ Less

Submitted 3 October, 2019; v1 submitted 26 April, 2019; originally announced April 2019.

Comments: 8 pages, 5 figures. Final version published in ApJS. The HACC Simulation Data Portal can be accessed here: https://cosmology.alcf.anl.gov/

Journal ref: Astrophys.J.Suppl. 244 (2019) no.1, 17

Effects of Massive Neutrinos and Dynamical Dark Energy on the Cluster Mass Function

Authors: Rahul Biswas, Katrin Heitmann, Salman Habib, Amol Upadhye, Adrian Pope, Nicholas Frontiere

Abstract: The presence of massive neutrinos affects the growth of large-scale structure in the universe, leaving a potentially observable imprint on the abundance and properties of massive dark matter-dominated halos. Cosmological surveys detect large numbers of these halos in the form of rich groups and clusters, using the information as an input to constraining the properties of dark energy. We use a suit… ▽ More The presence of massive neutrinos affects the growth of large-scale structure in the universe, leaving a potentially observable imprint on the abundance and properties of massive dark matter-dominated halos. Cosmological surveys detect large numbers of these halos in the form of rich groups and clusters, using the information as an input to constraining the properties of dark energy. We use a suite of N-body simulations that include the effects of massive neutrinos as well as of dynamical dark energy to study the properties of the mass function. As in our previous work, we follow an approach valid at low neutrino mass, where the neutrino overdensities are assumed to be too small to act as a significant nonlinear source term for gravity. We study how well a universal form for the halo mass function describes our numerical results, finding that the use of an appropriate linear power spectrum within the formalism yields a good match to the simulation results, correctly accounting for the (neutrino mass-dependent) suppression of the mass function. △ Less

Submitted 30 January, 2019; originally announced January 2019.

Comments: 11 pages, 5 figures, to be submitted to PRD

The Borg Cube Simulation: Cosmological Hydrodynamics with CRK-SPH

Authors: J. D. Emberson, Nicholas Frontiere, Salman Habib, Katrin Heitmann, Patricia Larsen, Hal Finkel, Adrian Pope

Abstract: A challenging requirement posed by next-generation observations is a firm theoretical grasp of the impact of baryons on structure formation. Cosmological hydrodynamic simulations modeling gas physics are vital in this regard. A high degree of modeling flexibility exists in this space making it important to explore a range of methods in order to gauge the accuracy of simulation predictions. We pres… ▽ More A challenging requirement posed by next-generation observations is a firm theoretical grasp of the impact of baryons on structure formation. Cosmological hydrodynamic simulations modeling gas physics are vital in this regard. A high degree of modeling flexibility exists in this space making it important to explore a range of methods in order to gauge the accuracy of simulation predictions. We present results from the first cosmological simulation using Conservative Reproducing Kernel Smoothed Particle Hydrodynamics (CRK-SPH). We employ two simulations: one evolved purely under gravity and the other with non-radiative hydrodynamics. Each contains 2x2304^3 cold dark matter plus baryon particles in an 800 Mpc/h box. We compare statistics to previous non-radiative simulations including power spectra, mass functions, baryon fractions, and concentration. We find self-similar radial profiles of gas temperature, entropy, and pressure and show that a simple analytic model recovers these results to better than 40% over two orders of magnitude in mass. We quantify the level of non-thermal pressure support in halos and demonstrate that hydrostatic mass estimates are biased low by 24% (10%) for halos of mass 10^15 (10^13) Msun/h. We compute angular power spectra for the thermal and kinematic Sunyaev-Zel'dovich effects and find good agreement with the low-l Planck measurements. Finally, artificial scattering between particles of unequal mass is shown to have a large impact on the gravity-only run and we highlight the importance of better understanding this issue in hydrodynamic applications. This is the first in a simulation campaign using CRK-SPH with future work including subresolution gas treatments. △ Less

Submitted 3 June, 2019; v1 submitted 8 November, 2018; originally announced November 2018.

Comments: Accepted for publication in the Astrophysical Journal

Journal ref: ApJ, 877, 85, 2019

Halo Profiles and the Concentration-Mass Relation for a ΛCDM Universe

Authors: Hillary L. Child, Salman Habib, Katrin Heitmann, Nicholas Frontiere, Hal Finkel, Adrian Pope, Vitali Morozov

Abstract: Profiles of dark matter-dominated halos at the group and cluster scales play an important role in modern cosmology. Using results from two very large cosmological $N$-body simulations, which increase the available volume at their mass resolution by roughly two orders of magnitude, we robustly determine the halo concentration-mass $(c-M)$ relation over a wide range of masses, employing multiple met… ▽ More Profiles of dark matter-dominated halos at the group and cluster scales play an important role in modern cosmology. Using results from two very large cosmological $N$-body simulations, which increase the available volume at their mass resolution by roughly two orders of magnitude, we robustly determine the halo concentration-mass $(c-M)$ relation over a wide range of masses, employing multiple methods of concentration measurement. We characterize individual halo profiles, as well as stacked profiles, relevant for galaxy-galaxy lensing and next-generation cluster surveys; the redshift range covered is $0\leq z \leq 4$, with a minimum halo mass of $M_{200c}\sim2\times10^{11} M_\odot$. Despite the complexity of a proper description of a halo (environmental effects, merger history, nonsphericity, relaxation state), when the mass is scaled by the nonlinear mass scale $M_\star(z)$, we find that a simple non-power-law form for the $c-M/M_\star$ relation provides an excellent description of our simulation results across eight decades in $M/M_{\star}$ and for $0\leq z \leq 4$. Over the mass range covered, the $c-M$ relation has two asymptotic forms: an approximate power law below a mass threshold $M/M_\star\sim 500-1000$, transitioning to a constant value, $c_0\sim 3$ at higher masses. The relaxed halo fraction decreases with mass, transitioning to a constant value of $\sim 0.5$ above the same mass threshold. We compare Navarro-Frenk-White (NFW) and Einasto fits to stacked profiles in narrow mass bins at different redshifts; as expected, the Einasto profile provides a better description of the simulation results. At cluster scales at low redshift, however, both NFW and Einasto profiles are in very good agreement with the simulation results, consistent with recent weak lensing observations. △ Less

Submitted 7 May, 2018; v1 submitted 26 April, 2018; originally announced April 2018.

Comments: 19 pages, 20 figures; minor corrections and additional reference

Journal ref: ApJ 859, 55 (2018)

The Mira-Titan Universe II: Matter Power Spectrum Emulation

Authors: Earl Lawrence, Katrin Heitmann, Juliana Kwan, Amol Upadhye, Derek Bingham, Salman Habib, David Higdon, Adrian Pope, Hal Finkel, Nicholas Frontiere

Abstract: We introduce a new cosmic emulator for the matter power spectrum covering eight cosmological parameters. Targeted at optical surveys, the emulator provides accurate predictions out to a wavenumber k~5/Mpc and redshift z<=2. Besides covering the standard set of LCDM parameters, massive neutrinos and a dynamical dark energy of state are included. The emulator is built on a sample set of 36 cosmologi… ▽ More We introduce a new cosmic emulator for the matter power spectrum covering eight cosmological parameters. Targeted at optical surveys, the emulator provides accurate predictions out to a wavenumber k~5/Mpc and redshift z<=2. Besides covering the standard set of LCDM parameters, massive neutrinos and a dynamical dark energy of state are included. The emulator is built on a sample set of 36 cosmological models, carefully chosen to provide accurate predictions over the wide and large parameter space. For each model, we have performed a high-resolution simulation, augmented with sixteen medium-resolution simulations and TimeRG perturbation theory results to provide accurate coverage of a wide k-range; the dataset generated as part of this project is more than 1.2Pbyte. With the current set of simulated models, we achieve an accuracy of approximately 4%. Because the sampling approach used here has established convergence and error-control properties, follow-on results with more than a hundred cosmological models will soon achieve ~1% accuracy. We compare our approach with other prediction schemes that are based on halo model ideas and remapping approaches. The new emulator code is publicly available. △ Less

Submitted 9 May, 2017; originally announced May 2017.

Comments: 12 pages, 12 figures, emulator code publicly available here: https://github.com/lanl/CosmicEmu

CRKSPH - A Conservative Reproducing Kernel Smoothed Particle Hydrodynamics Scheme

Authors: Nicholas Frontiere, Cody D. Raskin, J. Michael Owen

Abstract: We present a formulation of smoothed particle hydrodynamics (SPH) that utilizes a first-order consistent reproducing kernel, a smoothing function that exactly interpolates linear fields with particle tracers. Previous formulations using reproducing kernel (RK) interpolation have had difficulties maintaining conservation of momentum due to the fact the RK kernels are not, in general, spatially symm… ▽ More We present a formulation of smoothed particle hydrodynamics (SPH) that utilizes a first-order consistent reproducing kernel, a smoothing function that exactly interpolates linear fields with particle tracers. Previous formulations using reproducing kernel (RK) interpolation have had difficulties maintaining conservation of momentum due to the fact the RK kernels are not, in general, spatially symmetric. Here, we utilize a reformulation of the fluid equations such that mass, linear momentum, and energy are all rigorously conserved without any assumption about kernel symmetries, while additionally maintaining approximate angular momentum conservation. Our approach starts from a rigorously consistent interpolation theory, where we derive the evolution equations to enforce the appropriate conservation properties, at the sacrifice of full consistency in the momentum equation. Additionally, by exploiting the increased accuracy of the RK method's gradient, we formulate a simple limiter for the artificial viscosity that reduces the excess diffusion normally incurred by the ordinary SPH artificial viscosity. Collectively, we call our suite of modifications to the traditional SPH scheme Conservative Reproducing Kernel SPH, or CRKSPH. CRKSPH retains many benefits of traditional SPH methods (such as preserving Galilean invariance and manifest conservation of mass, momentum, and energy) while improving on many of the shortcomings of SPH, particularly the overly aggressive artificial viscosity and zeroth-order inaccuracy. We compare CRKSPH to two different modern SPH formulations (pressure based SPH and compatibly differenced SPH), demonstrating the advantages of our new formulation when modeling fluid mixing, strong shock, and adiabatic phenomena. △ Less

Submitted 27 December, 2016; v1 submitted 2 May, 2016; originally announced May 2016.

Journal ref: CRKSPH - A Conservative Reproducing Kernel Smoothed Particle Hydrodynamics Scheme, J. Comput. Phys. 332 (2017) 160-209

ASCR/HEP Exascale Requirements Review Report

Authors: Salman Habib, Robert Roser, Richard Gerber, Katie Antypas, Katherine Riley, Tim Williams, Jack Wells, Tjerk Straatsma, A. Almgren, J. Amundson, S. Bailey, D. Bard, K. Bloom, B. Bockelman, A. Borgland, J. Borrill, R. Boughezal, R. Brower, B. Cowan, H. Finkel, N. Frontiere, S. Fuess, L. Ge, N. Gnedin, S. Gottlieb , et al. (29 additional authors not shown)

Abstract: This draft report summarizes and details the findings, results, and recommendations derived from the ASCR/HEP Exascale Requirements Review meeting held in June, 2015. The main conclusions are as follows. 1) Larger, more capable computing and data facilities are needed to support HEP science goals in all three frontiers: Energy, Intensity, and Cosmic. The expected scale of the demand at the 2025 ti… ▽ More This draft report summarizes and details the findings, results, and recommendations derived from the ASCR/HEP Exascale Requirements Review meeting held in June, 2015. The main conclusions are as follows. 1) Larger, more capable computing and data facilities are needed to support HEP science goals in all three frontiers: Energy, Intensity, and Cosmic. The expected scale of the demand at the 2025 timescale is at least two orders of magnitude -- and in some cases greater -- than that available currently. 2) The growth rate of data produced by simulations is overwhelming the current ability, of both facilities and researchers, to store and analyze it. Additional resources and new techniques for data analysis are urgently needed. 3) Data rates and volumes from HEP experimental facilities are also straining the ability to store and analyze large and complex data volumes. Appropriately configured leadership-class facilities can play a transformational role in enabling scientific discovery from these datasets. 4) A close integration of HPC simulation and data analysis will aid greatly in interpreting results from HEP experiments. Such an integration will minimize data movement and facilitate interdependent workflows. 5) Long-range planning between HEP and ASCR will be required to meet HEP's research needs. To best use ASCR HPC resources the experimental HEP program needs a) an established long-term plan for access to ASCR computational and data resources, b) an ability to map workflows onto HPC resources, c) the ability for ASCR facilities to accommodate workflows run by collaborations that can have thousands of individual members, d) to transition codes to the next-generation HPC platforms that will be available at ASCR facilities, e) to build up and train a workforce capable of developing and using simulations and analysis to support HEP scientific research on next-generation systems. △ Less

Submitted 31 March, 2016; v1 submitted 30 March, 2016; originally announced March 2016.

Comments: 77 pages, 13 Figures; draft report, subject to further revision

The Mira-Titan Universe: Precision Predictions for Dark Energy Surveys

Authors: Katrin Heitmann, Derek Bingham, Earl Lawrence, Steven Bergner, Salman Habib, David Higdon, Adrian Pope, Rahul Biswas, Hal Finkel, Nicholas Frontiere, Suman Bhattacharya

Abstract: Ground and space-based sky surveys enable powerful cosmological probes based on measurements of galaxy properties and the distribution of galaxies in the Universe. These probes include weak lensing, baryon acoustic oscillations, abundance of galaxy clusters, and redshift space distortions; they are essential to improving our knowledge of the nature of dark energy. On the theory and modeling front,… ▽ More Ground and space-based sky surveys enable powerful cosmological probes based on measurements of galaxy properties and the distribution of galaxies in the Universe. These probes include weak lensing, baryon acoustic oscillations, abundance of galaxy clusters, and redshift space distortions; they are essential to improving our knowledge of the nature of dark energy. On the theory and modeling front, large-scale simulations of cosmic structure formation play an important role in interpreting the observations and in the challenging task of extracting cosmological physics at the needed precision. These simulations must cover a parameter range beyond the standard six cosmological parameters and need to be run at high mass and force resolution. One key simulation-based task is the generation of accurate theoretical predictions for observables, via the method of emulation. Using a new sampling technique, we explore an 8-dimensional parameter space including massive neutrinos and a variable dark energy equation of state. We construct trial emulators using two surrogate models (the linear power spectrum and an approximate halo mass function). The new sampling method allows us to build precision emulators from just 26 cosmological models and to increase the emulator accuracy by adding new sets of simulations in a prescribed way. This allows emulator fidelity to be systematically improved as new observational data becomes available and higher accuracy is required. Finally, using one LCDM cosmology as an example, we study the demands imposed on a simulation campaign to achieve the required statistics and accuracy when building emulators for dark energy investigations. △ Less

Submitted 11 August, 2015; originally announced August 2015.

Comments: 14 pages, 13 figures

Redshift-space distortions in massive neutrino and evolving dark energy cosmologies

Authors: Amol Upadhye, Juliana Kwan, Adrian Pope, Katrin Heitmann, Salman Habib, Hal Finkel, Nicholas Frontiere

Abstract: Large-scale structure surveys in the coming years will measure the redshift-space power spectrum to unprecedented accuracy, allowing for powerful new tests of the LambdaCDM picture as well as measurements of particle physics parameters such as the neutrino masses. We extend the Time-RG perturbative framework to redshift space, computing the power spectrum P_s(k,mu) in massive neutrino cosmologies… ▽ More Large-scale structure surveys in the coming years will measure the redshift-space power spectrum to unprecedented accuracy, allowing for powerful new tests of the LambdaCDM picture as well as measurements of particle physics parameters such as the neutrino masses. We extend the Time-RG perturbative framework to redshift space, computing the power spectrum P_s(k,mu) in massive neutrino cosmologies with time-dependent dark energy equations of state w(z). Time-RG is uniquely capable of incorporating scale-dependent growth into the P_s(k,mu) computation, which is important for massive neutrinos as well as modified gravity models. Although changes to w(z) and the neutrino mass fraction both affect the late-time scale-dependence of the non-linear power spectrum, we find that the two effects depend differently on the line-of-sight angle mu. Finally, we use the HACC N-body code to quantify errors in the perturbative calculations. For a LambdaCDM model at redshift z=1, our procedure predicts the monopole~(quadrupole) to 1% accuracy up to a wave number 0.19h/Mpc (0.28h/Mpc), compared to 0.08h/Mpc (0.07h/Mpc) for the Kaiser approximation and 0.19h/Mpc (0.16h/Mpc) for the current state-of-the-art perturbation scheme. Our calculation agrees with the simulated redshift-space power spectrum even for neutrino masses above the current bound, and for rapidly-evolving dark energy equations of state, |dw/dz| ~ 1. Along with this article, we make our redshift-space Time-RG implementation publicly available as the code redTime. △ Less

Submitted 29 February, 2016; v1 submitted 24 June, 2015; originally announced June 2015.

Comments: 18 pages, 17 figures, 4 tables. Matches version accepted by PRD. redTime code available at http://www.hep.anl.gov/cosmology/pert.html

Journal ref: Phys. Rev. D 93, 063515 (2016)

The Q Continuum Simulation: Harnessing the Power of GPU Accelerated Supercomputers

Authors: Katrin Heitmann, Nicholas Frontiere, Chris Sewell, Salman Habib, Adrian Pope, Hal Finkel, Silvio Rizzi, Joe Insley, Suman Bhattacharya

Abstract: Modeling large-scale sky survey observations is a key driver for the continuing development of high resolution, large-volume, cosmological simulations. We report the first results from the 'Q Continuum' cosmological N-body simulation run carried out on the GPU-accelerated supercomputer Titan. The simulation encompasses a volume of (1300 Mpc)^3 and evolves more than half a trillion particles, leadi… ▽ More Modeling large-scale sky survey observations is a key driver for the continuing development of high resolution, large-volume, cosmological simulations. We report the first results from the 'Q Continuum' cosmological N-body simulation run carried out on the GPU-accelerated supercomputer Titan. The simulation encompasses a volume of (1300 Mpc)^3 and evolves more than half a trillion particles, leading to a particle mass resolution of ~1.5 X 10^8 M_sun. At this mass resolution, the Q Continuum run is currently the largest cosmology simulation available. It enables the construction of detailed synthetic sky catalogs, encompassing different modeling methodologies, including semi-analytic modeling and sub-halo abundance matching in a large, cosmological volume. Here we describe the simulation and outputs in detail and present first results for a range of cosmological statistics, such as mass power spectra, halo mass functions, and halo mass-concentration relations for different epochs. We also provide details on challenges connected to running a simulation on almost 90% of Titan, one of the fastest supercomputers in the world, including our usage of Titan's GPU accelerators. △ Less

Submitted 12 November, 2014; originally announced November 2014.

Comments: 11 pages, 12 figures

HACC: Simulating Sky Surveys on State-of-the-Art Supercomputing Architectures

Authors: Salman Habib, Adrian Pope, Hal Finkel, Nicholas Frontiere, Katrin Heitmann, David Daniel, Patricia Fasel, Vitali Morozov, George Zagaris, Tom Peterka, Venkatram Vishwanath, Zarija Lukic, Saba Sehrish, Wei-keng Liao

Abstract: Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the 'Dark Universe', dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino… ▽ More Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the 'Dark Universe', dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers that enable these simulations. Here we report on HACC (Hardware/Hybrid Accelerated Cosmology Code), a recently developed and evolving cosmology N-body code framework, designed to run efficiently on diverse computing architectures and to scale to millions of cores and beyond. HACC can run on all current supercomputer architectures and supports a variety of programming models and algorithms. It has been demonstrated at scale on Cell- and GPU-accelerated systems, standard multi-core node clusters, and Blue Gene systems. HACC's design allows for ease of portability, and at the same time, high levels of sustained performance on the fastest supercomputers available. We present a description of the design philosophy of HACC, the underlying algorithms and code structure, and outline implementation details for several specific architectures. We show selected accuracy and performance results from some of the largest high resolution cosmological simulations so far performed, including benchmarks evolving more than 3.6 trillion particles. △ Less

Submitted 8 October, 2014; originally announced October 2014.

Comments: 23 pages, 20 figures

Cosmic Emulation: Fast Predictions for the Galaxy Power Spectrum

Authors: Juliana Kwan, Katrin Heitmann, Salman Habib, Nikhil Padmanabhan, Hal Finkel, Nick Frontiere, Adrian Pope

Abstract: The halo occupation distribution (HOD) approach has proven to be an effective method for modeling galaxy clustering and bias. In this approach, galaxies of a given type are probabilistically assigned to individual halos in N-body simulations. In this paper, we present a fast emulator for predicting the fully nonlinear galaxy power spectrum over a range of freely specifiable HOD modeling parameters… ▽ More The halo occupation distribution (HOD) approach has proven to be an effective method for modeling galaxy clustering and bias. In this approach, galaxies of a given type are probabilistically assigned to individual halos in N-body simulations. In this paper, we present a fast emulator for predicting the fully nonlinear galaxy power spectrum over a range of freely specifiable HOD modeling parameters. The emulator is constructed using results from 100 HOD models run on a large LCDM N-body simulation, with Gaussian Process interpolation applied to a PCA-based representation of the galaxy power spectrum. The total error is currently ~3% (~2% in the simulation and ~1% in the emulation process) from z=1 to z=0, over the considered parameter range. We use the emulator to investigate parametric dependencies in the HOD model, as well as the behavior of galaxy bias as a function of HOD parameters. The emulator is publicly available at http://www.hep.anl.gov/cosmology/CosmicEmu/emu.html. △ Less

Submitted 16 August, 2015; v1 submitted 25 November, 2013; originally announced November 2013.

Comments: Replaced to match published version. The emulator can be downloaded at http://www.hep.anl.gov/cosmology/CosmicEmu/emu.html

Large-Scale Structure Formation with Massive Neutrinos and Dynamical Dark Energy

Authors: Amol Upadhye, Rahul Biswas, Adrian Pope, Katrin Heitmann, Salman Habib, Hal Finkel, Nicholas Frontiere

Abstract: Over the next decade, cosmological measurements of the large-scale structure of the Universe will be sensitive to the combined effects of dynamical dark energy and massive neutrinos. The matter power spectrum is a key repository of this information. We extend higher-order perturbative methods for computing the power spectrum to investigate these effects over quasi-linear scales. Through comparison… ▽ More Over the next decade, cosmological measurements of the large-scale structure of the Universe will be sensitive to the combined effects of dynamical dark energy and massive neutrinos. The matter power spectrum is a key repository of this information. We extend higher-order perturbative methods for computing the power spectrum to investigate these effects over quasi-linear scales. Through comparison with N-body simulations we establish the regime of validity of a Time-Renormalization Group (Time-RG) perturbative treatment that includes dynamical dark energy and massive neutrinos. We also quantify the accuracy of Standard (SPT), Renormalized (RPT) and Lagrangian Resummation (LPT) perturbation theories without massive neutrinos. We find that an approximation that neglects neutrino clustering as a source for nonlinear matter clustering predicts the Baryon Acoustic Oscillation (BAO) peak position to 0.25% accuracy for redshifts 1 < z < 3, justifying the use of LPT for BAO reconstruction in upcoming surveys. We release a modified version of the public Copter code which includes the additional physics discussed in the paper. △ Less

Submitted 28 April, 2014; v1 submitted 23 September, 2013; originally announced September 2013.

Comments: Code available at http://www.hep.anl.gov/cosmology/pert.html . 16 pages, 10 figures. Matches version accepted by PRD

Journal ref: Phys. Rev. D 89, 103515 (2014)

The Universe at Extreme Scale: Multi-Petaflop Sky Simulation on the BG/Q

Authors: Salman Habib, Vitali Morozov, Hal Finkel, Adrian Pope, Katrin Heitmann, Kalyan Kumaran, Tom Peterka, Joe Insley, David Daniel, Patricia Fasel, Nicholas Frontiere, Zarija Lukic

Abstract: Remarkable observational advances have established a compelling cross-validated model of the Universe. Yet, two key pillars of this model -- dark matter and dark energy -- remain mysterious. Sky surveys that map billions of galaxies to explore the `Dark Universe', demand a corresponding extreme-scale simulation capability; the HACC (Hybrid/Hardware Accelerated Cosmology Code) framework has been de… ▽ More Remarkable observational advances have established a compelling cross-validated model of the Universe. Yet, two key pillars of this model -- dark matter and dark energy -- remain mysterious. Sky surveys that map billions of galaxies to explore the `Dark Universe', demand a corresponding extreme-scale simulation capability; the HACC (Hybrid/Hardware Accelerated Cosmology Code) framework has been designed to deliver this level of performance now, and into the future. With its novel algorithmic structure, HACC allows flexible tuning across diverse architectures, including accelerated and multi-core systems. On the IBM BG/Q, HACC attains unprecedented scalable performance -- currently 13.94 PFlops at 69.2% of peak and 90% parallel efficiency on 1,572,864 cores with an equal number of MPI ranks, and a concurrency of 6.3 million. This level of performance was achieved at extreme problem sizes, including a benchmark run with more than 3.6 trillion particles, significantly larger than any cosmological simulation yet performed. △ Less

Submitted 19 November, 2012; originally announced November 2012.

Comments: 11 pages, 11 figures, final version of paper for talk presented at SC12