-
The Outer Rim Simulation: A Path to Many-Core Supercomputers
Authors:
Katrin Heitmann,
Hal Finkel,
Adrian Pope,
Vitali Morozov,
Nicholas Frontiere,
Salman Habib,
Esteban Rangel,
Thomas Uram,
Danila Korytov,
Hillary Child,
Samuel Flender,
Joe Insley,
Silvio Rizzi
Abstract:
We describe the Outer Rim cosmological simulation, one of the largest high-resolution N-body simulations performed to date, aimed at promoting science to be carried out with large-scale structure surveys. The simulation covers a volume of (4.225Gpc)^3 and evolves more than one trillion particles. It was executed on Mira, a BlueGene/Q system at the Argonne Leadership Computing Facility. We discuss…
▽ More
We describe the Outer Rim cosmological simulation, one of the largest high-resolution N-body simulations performed to date, aimed at promoting science to be carried out with large-scale structure surveys. The simulation covers a volume of (4.225Gpc)^3 and evolves more than one trillion particles. It was executed on Mira, a BlueGene/Q system at the Argonne Leadership Computing Facility. We discuss some of the computational challenges posed by a system like Mira, a many-core supercomputer, and how the simulation code, HACC, has been designed to overcome these challenges. We have carried out a large range of analyses on the simulation data and we report on the results as well as the data products that have been generated. The full data set generated by the simulation totals more than 5PB of data, making data curation and data handling a large challenge in of itself. The simulation results have been used to generate synthetic catalogs for large-scale structure surveys, including DESI and eBOSS, as well as CMB experiments. A detailed catalog for the LSST DESC data challenges has been created as well. We publicly release some of the Outer Rim halo catalogs, downsampled particle information, and lightcone data.
△ Less
Submitted 28 April, 2019; v1 submitted 26 April, 2019;
originally announced April 2019.
-
Halo Profiles and the Concentration-Mass Relation for a ΛCDM Universe
Authors:
Hillary L. Child,
Salman Habib,
Katrin Heitmann,
Nicholas Frontiere,
Hal Finkel,
Adrian Pope,
Vitali Morozov
Abstract:
Profiles of dark matter-dominated halos at the group and cluster scales play an important role in modern cosmology. Using results from two very large cosmological $N$-body simulations, which increase the available volume at their mass resolution by roughly two orders of magnitude, we robustly determine the halo concentration-mass $(c-M)$ relation over a wide range of masses, employing multiple met…
▽ More
Profiles of dark matter-dominated halos at the group and cluster scales play an important role in modern cosmology. Using results from two very large cosmological $N$-body simulations, which increase the available volume at their mass resolution by roughly two orders of magnitude, we robustly determine the halo concentration-mass $(c-M)$ relation over a wide range of masses, employing multiple methods of concentration measurement. We characterize individual halo profiles, as well as stacked profiles, relevant for galaxy-galaxy lensing and next-generation cluster surveys; the redshift range covered is $0\leq z \leq 4$, with a minimum halo mass of $M_{200c}\sim2\times10^{11} M_\odot$. Despite the complexity of a proper description of a halo (environmental effects, merger history, nonsphericity, relaxation state), when the mass is scaled by the nonlinear mass scale $M_\star(z)$, we find that a simple non-power-law form for the $c-M/M_\star$ relation provides an excellent description of our simulation results across eight decades in $M/M_{\star}$ and for $0\leq z \leq 4$. Over the mass range covered, the $c-M$ relation has two asymptotic forms: an approximate power law below a mass threshold $M/M_\star\sim 500-1000$, transitioning to a constant value, $c_0\sim 3$ at higher masses. The relaxed halo fraction decreases with mass, transitioning to a constant value of $\sim 0.5$ above the same mass threshold. We compare Navarro-Frenk-White (NFW) and Einasto fits to stacked profiles in narrow mass bins at different redshifts; as expected, the Einasto profile provides a better description of the simulation results. At cluster scales at low redshift, however, both NFW and Einasto profiles are in very good agreement with the simulation results, consistent with recent weak lensing observations.
△ Less
Submitted 7 May, 2018; v1 submitted 26 April, 2018;
originally announced April 2018.
-
US Cosmic Visions: New Ideas in Dark Matter 2017: Community Report
Authors:
Marco Battaglieri,
Alberto Belloni,
Aaron Chou,
Priscilla Cushman,
Bertrand Echenard,
Rouven Essig,
Juan Estrada,
Jonathan L. Feng,
Brenna Flaugher,
Patrick J. Fox,
Peter Graham,
Carter Hall,
Roni Harnik,
JoAnne Hewett,
Joseph Incandela,
Eder Izaguirre,
Daniel McKinsey,
Matthew Pyle,
Natalie Roe,
Gray Rybka,
Pierre Sikivie,
Tim M. P. Tait,
Natalia Toro,
Richard Van De Water,
Neal Weiner
, et al. (226 additional authors not shown)
Abstract:
This white paper summarizes the workshop "U.S. Cosmic Visions: New Ideas in Dark Matter" held at University of Maryland on March 23-25, 2017.
This white paper summarizes the workshop "U.S. Cosmic Visions: New Ideas in Dark Matter" held at University of Maryland on March 23-25, 2017.
△ Less
Submitted 14 July, 2017;
originally announced July 2017.
-
Optical Response of Strained- and Unstrained-Silicon Cold-Electron Bolometers
Authors:
T. L. R. Brien,
P. A. R. Ade,
P. S. Barry,
C. J. Dunscombe,
D. R. Leadley,
D. V. Morozov,
M. Myronov,
E. H. C. Parker,
M. J. Prest,
M. Prunnila,
R. V. Sudiwala,
T. E. Whall,
P. D. Mauskopf
Abstract:
We describe the optical characterisation of two silicon cold-electron bolometers each consisting of a small ($32 \times 14~\mathrm{μm}$) island of degenerately doped silicon with superconducting aluminium contacts. Radiation is coupled into the silicon absorber with a twin-slot antenna designed to couple to 160-GHz radiation through a silicon lens.The first device has a highly doped silicon absorb…
▽ More
We describe the optical characterisation of two silicon cold-electron bolometers each consisting of a small ($32 \times 14~\mathrm{μm}$) island of degenerately doped silicon with superconducting aluminium contacts. Radiation is coupled into the silicon absorber with a twin-slot antenna designed to couple to 160-GHz radiation through a silicon lens.The first device has a highly doped silicon absorber, the second has a highly doped strained-silicon absorber.Using a novel method of cross-correlating the outputs from two parallel amplifiers, we measure noise-equivalent powers of $3.0 \times 10^{-16}$ and $6.6 \times 10^{-17}~\mathrm{W\,Hz^{-1/2}}$ for the control and strained device, respectively, when observing radiation from a 77-K source. In the case of the strained device, the noise-equivalent power is limited by the photon noise.
△ Less
Submitted 10 March, 2016;
originally announced March 2016.
-
HACC: Simulating Sky Surveys on State-of-the-Art Supercomputing Architectures
Authors:
Salman Habib,
Adrian Pope,
Hal Finkel,
Nicholas Frontiere,
Katrin Heitmann,
David Daniel,
Patricia Fasel,
Vitali Morozov,
George Zagaris,
Tom Peterka,
Venkatram Vishwanath,
Zarija Lukic,
Saba Sehrish,
Wei-keng Liao
Abstract:
Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the 'Dark Universe', dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino…
▽ More
Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the 'Dark Universe', dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers that enable these simulations. Here we report on HACC (Hardware/Hybrid Accelerated Cosmology Code), a recently developed and evolving cosmology N-body code framework, designed to run efficiently on diverse computing architectures and to scale to millions of cores and beyond. HACC can run on all current supercomputer architectures and supports a variety of programming models and algorithms. It has been demonstrated at scale on Cell- and GPU-accelerated systems, standard multi-core node clusters, and Blue Gene systems. HACC's design allows for ease of portability, and at the same time, high levels of sustained performance on the fastest supercomputers available. We present a description of the design philosophy of HACC, the underlying algorithms and code structure, and outline implementation details for several specific architectures. We show selected accuracy and performance results from some of the largest high resolution cosmological simulations so far performed, including benchmarks evolving more than 3.6 trillion particles.
△ Less
Submitted 8 October, 2014;
originally announced October 2014.
-
A Strained Silicon Cold Electron Bolometer using Schottky Contacts
Authors:
T. L. R. Brien,
P. A. R. Ade,
P. S. Barry,
C. Dunscombe,
D. R. Leadley,
D. V. Morozov,
M. Myronov,
E. H. C. Parker,
M. Prunnila,
M. J. Prest,
R. V. Sudiwala,
T. E. Whall,
P. D. Mauskopf
Abstract:
We describe optical characterisation of a Strained Silicon Cold Electron Bolometer (CEB), operating on a $350~\mathrm{mK}$ stage, designed for absorption of millimetre-wave radiation. The silicon Cold Electron Bolometer utilises Schottky contacts between a superconductor and an n++ doped silicon island to detect changes in the temperature of the charge carriers in the silicon, due to variations in…
▽ More
We describe optical characterisation of a Strained Silicon Cold Electron Bolometer (CEB), operating on a $350~\mathrm{mK}$ stage, designed for absorption of millimetre-wave radiation. The silicon Cold Electron Bolometer utilises Schottky contacts between a superconductor and an n++ doped silicon island to detect changes in the temperature of the charge carriers in the silicon, due to variations in absorbed radiation. By using strained silicon as the absorber, we decrease the electron-phonon coupling in the device and increase the responsivity to incoming power. The strained silicon absorber is coupled to a planar aluminium twin-slot antenna designed to couple to $160~\mathrm{GHz}$ and that serves as the superconducting contacts. From the measured optical responsivity and spectral response, we calculate a maximum optical efficiency of $50~\%$ for radiation coupled into the device by the planar antenna and an overall noise equivalent power (NEP), referred to absorbed optical power, of $1.1 \times 10^{-16}~\mathrm{\mbox{W Hz}^{-1/2}}$ when the detector is observing a $300~\mathrm{K}$ source through a $4~\mathrm{K}$ throughput limiting aperture. Even though this optical system is not optimised we measure a system noise equivalent temperature difference (NETD) of $6~\mathrm{\mbox{mK Hz}^{-1/2}}$. We measure the noise of the device using a cross-correlation of time stream data measured simultaneously with two junction field-effect transistor (JFET) amplifiers, with a base correlated noise level of $300~\mathrm{\mbox{pV Hz}^{-1/2}}$ and find that the total noise is consistent with a combination of photon noise, current shot noise and electron-phonon thermal noise.
△ Less
Submitted 31 July, 2014; v1 submitted 8 July, 2014;
originally announced July 2014.
-
The Universe at Extreme Scale: Multi-Petaflop Sky Simulation on the BG/Q
Authors:
Salman Habib,
Vitali Morozov,
Hal Finkel,
Adrian Pope,
Katrin Heitmann,
Kalyan Kumaran,
Tom Peterka,
Joe Insley,
David Daniel,
Patricia Fasel,
Nicholas Frontiere,
Zarija Lukic
Abstract:
Remarkable observational advances have established a compelling cross-validated model of the Universe. Yet, two key pillars of this model -- dark matter and dark energy -- remain mysterious. Sky surveys that map billions of galaxies to explore the `Dark Universe', demand a corresponding extreme-scale simulation capability; the HACC (Hybrid/Hardware Accelerated Cosmology Code) framework has been de…
▽ More
Remarkable observational advances have established a compelling cross-validated model of the Universe. Yet, two key pillars of this model -- dark matter and dark energy -- remain mysterious. Sky surveys that map billions of galaxies to explore the `Dark Universe', demand a corresponding extreme-scale simulation capability; the HACC (Hybrid/Hardware Accelerated Cosmology Code) framework has been designed to deliver this level of performance now, and into the future. With its novel algorithmic structure, HACC allows flexible tuning across diverse architectures, including accelerated and multi-core systems.
On the IBM BG/Q, HACC attains unprecedented scalable performance -- currently 13.94 PFlops at 69.2% of peak and 90% parallel efficiency on 1,572,864 cores with an equal number of MPI ranks, and a concurrency of 6.3 million. This level of performance was achieved at extreme problem sizes, including a benchmark run with more than 3.6 trillion particles, significantly larger than any cosmological simulation yet performed.
△ Less
Submitted 19 November, 2012;
originally announced November 2012.
