-
Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping
Authors:
Wushi Dong,
Murat Keceli,
Rafael Vescovi,
Hanyu Li,
Corey Adams,
Elise Jennings,
Samuel Flender,
Tom Uram,
Venkatram Vishwanath,
Nicola Ferrier,
Narayanan Kasthuri,
Peter Littlewood
Abstract:
Mapping all the neurons in the brain requires automatic reconstruction of entire cells from volume electron microscopy data. The flood-filling network (FFN) architecture has demonstrated leading performance for segmenting structures from this data. However, the training of the network is computationally expensive. In order to reduce the training time, we implemented synchronous and data-parallel d…
▽ More
Mapping all the neurons in the brain requires automatic reconstruction of entire cells from volume electron microscopy data. The flood-filling network (FFN) architecture has demonstrated leading performance for segmenting structures from this data. However, the training of the network is computationally expensive. In order to reduce the training time, we implemented synchronous and data-parallel distributed training using the Horovod library, which is different from the asynchronous training scheme used in the published FFN code. We demonstrated that our distributed training scaled well up to 2048 Intel Knights Landing (KNL) nodes on the Theta supercomputer. Our trained models achieved similar level of inference performance, but took less training time compared to previous methods. Our study on the effects of different batch sizes on FFN training suggests ways to further improve training efficiency. Our findings on optimal learning rate and batch sizes agree with previous works.
△ Less
Submitted 9 December, 2019; v1 submitted 13 May, 2019;
originally announced May 2019.
-
The Outer Rim Simulation: A Path to Many-Core Supercomputers
Authors:
Katrin Heitmann,
Hal Finkel,
Adrian Pope,
Vitali Morozov,
Nicholas Frontiere,
Salman Habib,
Esteban Rangel,
Thomas Uram,
Danila Korytov,
Hillary Child,
Samuel Flender,
Joe Insley,
Silvio Rizzi
Abstract:
We describe the Outer Rim cosmological simulation, one of the largest high-resolution N-body simulations performed to date, aimed at promoting science to be carried out with large-scale structure surveys. The simulation covers a volume of (4.225Gpc)^3 and evolves more than one trillion particles. It was executed on Mira, a BlueGene/Q system at the Argonne Leadership Computing Facility. We discuss…
▽ More
We describe the Outer Rim cosmological simulation, one of the largest high-resolution N-body simulations performed to date, aimed at promoting science to be carried out with large-scale structure surveys. The simulation covers a volume of (4.225Gpc)^3 and evolves more than one trillion particles. It was executed on Mira, a BlueGene/Q system at the Argonne Leadership Computing Facility. We discuss some of the computational challenges posed by a system like Mira, a many-core supercomputer, and how the simulation code, HACC, has been designed to overcome these challenges. We have carried out a large range of analyses on the simulation data and we report on the results as well as the data products that have been generated. The full data set generated by the simulation totals more than 5PB of data, making data curation and data handling a large challenge in of itself. The simulation results have been used to generate synthetic catalogs for large-scale structure surveys, including DESI and eBOSS, as well as CMB experiments. A detailed catalog for the LSST DESC data challenges has been created as well. We publicly release some of the Outer Rim halo catalogs, downsampled particle information, and lightcone data.
△ Less
Submitted 28 April, 2019; v1 submitted 26 April, 2019;
originally announced April 2019.
-
Investigating Cluster Astrophysics and Cosmology with Cross-Correlation of the Thermal Sunyaev-Zel'dovich Effect and Weak Lensing
Authors:
Ken Osato,
Samuel Flender,
Daisuke Nagai,
Masato Shirasaki,
Naoki Yoshida
Abstract:
Recent detections of the cross-correlation of the thermal Sunyaev-Zel'dovich (tSZ) effect and weak gravitational lensing (WL) enable unique studies of cluster astrophysics and cosmology. In this work, we present constraints on the amplitude of the non-thermal pressure fraction in galaxy clusters, $α_0$, and the amplitude of the matter power spectrum, $σ_8$, using measurements of the tSZ power spec…
▽ More
Recent detections of the cross-correlation of the thermal Sunyaev-Zel'dovich (tSZ) effect and weak gravitational lensing (WL) enable unique studies of cluster astrophysics and cosmology. In this work, we present constraints on the amplitude of the non-thermal pressure fraction in galaxy clusters, $α_0$, and the amplitude of the matter power spectrum, $σ_8$, using measurements of the tSZ power spectrum from Planck, and the tSZ-WL cross-correlation from Planck and the Red Cluster Sequence Lensing Survey. We fit the data to a semi-analytic model with the covariance matrix using $N$-body simulations. We find that the tSZ power spectrum alone prefers $σ_8 \sim 0.85$ and a large fraction of non-thermal pressure ($α_0 \sim 0.2$-$0.3$). The tSZ-WL cross-correlation on the other hand prefers a significantly lower $σ_8 \sim 0.6$, and low $α_0 \sim 0.05$. We show that this tension can be mitigated by allowing for a steep slope in the stellar-mass-halo-mass relation, which would cause a reduction of the gas in low-mass halos. In such a model, the combined data prefer $σ_8 \sim 0.7$ and $α_0 \sim 0.2$, consistent with predictions from hydrodynamical simulations.
△ Less
Submitted 24 January, 2018; v1 submitted 27 June, 2017;
originally announced June 2017.
-
The DESI Experiment Part II: Instrument Design
Authors:
DESI Collaboration,
Amir Aghamousa,
Jessica Aguilar,
Steve Ahlen,
Shadab Alam,
Lori E. Allen,
Carlos Allende Prieto,
James Annis,
Stephen Bailey,
Christophe Balland,
Otger Ballester,
Charles Baltay,
Lucas Beaufore,
Chris Bebek,
Timothy C. Beers,
Eric F. Bell,
José Luis Bernal,
Robert Besuner,
Florian Beutler,
Chris Blake,
Hannes Bleuler,
Michael Blomqvist,
Robert Blum,
Adam S. Bolton,
Cesar Briceno
, et al. (268 additional authors not shown)
Abstract:
DESI (Dark Energy Spectropic Instrument) is a Stage IV ground-based dark energy experiment that will study baryon acoustic oscillations and the growth of structure through redshift-space distortions with a wide-area galaxy and quasar redshift survey. The DESI instrument is a robotically-actuated, fiber-fed spectrograph capable of taking up to 5,000 simultaneous spectra over a wavelength range from…
▽ More
DESI (Dark Energy Spectropic Instrument) is a Stage IV ground-based dark energy experiment that will study baryon acoustic oscillations and the growth of structure through redshift-space distortions with a wide-area galaxy and quasar redshift survey. The DESI instrument is a robotically-actuated, fiber-fed spectrograph capable of taking up to 5,000 simultaneous spectra over a wavelength range from 360 nm to 980 nm. The fibers feed ten three-arm spectrographs with resolution $R= λ/Δλ$ between 2000 and 5500, depending on wavelength. The DESI instrument will be used to conduct a five-year survey designed to cover 14,000 deg$^2$. This powerful instrument will be installed at prime focus on the 4-m Mayall telescope in Kitt Peak, Arizona, along with a new optical corrector, which will provide a three-degree diameter field of view. The DESI collaboration will also deliver a spectroscopic pipeline and data management system to reduce and archive all data for eventual public use.
△ Less
Submitted 13 December, 2016; v1 submitted 31 October, 2016;
originally announced November 2016.
-
The DESI Experiment Part I: Science,Targeting, and Survey Design
Authors:
DESI Collaboration,
Amir Aghamousa,
Jessica Aguilar,
Steve Ahlen,
Shadab Alam,
Lori E. Allen,
Carlos Allende Prieto,
James Annis,
Stephen Bailey,
Christophe Balland,
Otger Ballester,
Charles Baltay,
Lucas Beaufore,
Chris Bebek,
Timothy C. Beers,
Eric F. Bell,
José Luis Bernal,
Robert Besuner,
Florian Beutler,
Chris Blake,
Hannes Bleuler,
Michael Blomqvist,
Robert Blum,
Adam S. Bolton,
Cesar Briceno
, et al. (268 additional authors not shown)
Abstract:
DESI (Dark Energy Spectroscopic Instrument) is a Stage IV ground-based dark energy experiment that will study baryon acoustic oscillations (BAO) and the growth of structure through redshift-space distortions with a wide-area galaxy and quasar redshift survey. To trace the underlying dark matter distribution, spectroscopic targets will be selected in four classes from imaging data. We will measure…
▽ More
DESI (Dark Energy Spectroscopic Instrument) is a Stage IV ground-based dark energy experiment that will study baryon acoustic oscillations (BAO) and the growth of structure through redshift-space distortions with a wide-area galaxy and quasar redshift survey. To trace the underlying dark matter distribution, spectroscopic targets will be selected in four classes from imaging data. We will measure luminous red galaxies up to $z=1.0$. To probe the Universe out to even higher redshift, DESI will target bright [O II] emission line galaxies up to $z=1.7$. Quasars will be targeted both as direct tracers of the underlying dark matter distribution and, at higher redshifts ($ 2.1 < z < 3.5$), for the Ly-$α$ forest absorption features in their spectra, which will be used to trace the distribution of neutral hydrogen. When moonlight prevents efficient observations of the faint targets of the baseline survey, DESI will conduct a magnitude-limited Bright Galaxy Survey comprising approximately 10 million galaxies with a median $z\approx 0.2$. In total, more than 30 million galaxy and quasar redshifts will be obtained to measure the BAO feature and determine the matter power spectrum, including redshift space distortions.
△ Less
Submitted 13 December, 2016; v1 submitted 31 October, 2016;
originally announced November 2016.
-
Constraints on the optical depth of galaxy groups and clusters
Authors:
Samuel Flender,
Daisuke Nagai,
Michael McDonald
Abstract:
Future data from galaxy redshift surveys, combined with high-resolutions maps of the cosmic microwave background, will enable measurements of the pairwise kinematic Sunyaev-Zel'dovich (kSZ) signal with unprecedented statistical significance. This signal probes the matter-velocity correlation function, scaled by the average optical depth ($τ$) of the galaxy groups and clusters in the sample, and is…
▽ More
Future data from galaxy redshift surveys, combined with high-resolutions maps of the cosmic microwave background, will enable measurements of the pairwise kinematic Sunyaev-Zel'dovich (kSZ) signal with unprecedented statistical significance. This signal probes the matter-velocity correlation function, scaled by the average optical depth ($τ$) of the galaxy groups and clusters in the sample, and is thus of fundamental importance for cosmology. However, in order to translate pairwise kSZ measurements into cosmological constraints, external constraints on $τ$ are necessary. In this work, we present a new model for the intra-cluster medium, which takes into account star-formation, feedback, non-thermal pressure, and gas cooling. Our semi-analytic model is computationally efficient and can reproduce results of recent hydrodynamical simulations of galaxy cluster formation. By calibrating the model using recent X-ray measurements of gas density profiles of clusters and $M_{\mathrm{gas}}-M$ relations of groups and clusters, we show that the integrated $τ$ is constrained to better than 6% (with a confidence level of 95%), which sets the tightest constraints on the external prior on the optical depth of groups and clusters reported in the literature to date. Our strong prior on $τ$ should significantly improve cosmological constraints derived from future pairwise kSZ measurements and shed new insights into the nature of dark energy, modified gravity, and neutrino mass.
△ Less
Submitted 1 May, 2017; v1 submitted 25 October, 2016;
originally announced October 2016.
-
Detection of the kinematic Sunyaev-Zel'dovich effect with DES Year 1 and SPT
Authors:
B. Soergel,
S. Flender,
K. T. Story,
L. Bleem,
T. Giannantonio,
G. Efstathiou,
E. Rykoff,
B. A. Benson,
T. Crawford,
S. Dodelson,
S. Habib,
K. Heitmann,
G. Holder,
B. Jain,
E. Rozo,
A. Saro,
J. Weller,
F. B. Abdalla,
S. Allam,
J. Annis,
R. Armstrong,
A. Benoit-Lévy,
G. M. Bernstein,
J. E. Carlstrom,
A. Carnero Rosell
, et al. (58 additional authors not shown)
Abstract:
We detect the kinematic Sunyaev-Zel'dovich (kSZ) effect with a statistical significance of $4.2 σ$ by combining a cluster catalogue derived from the first year data of the Dark Energy Survey (DES) with CMB temperature maps from the South Pole Telescope Sunyaev-Zel'dovich (SPT-SZ) Survey. This measurement is performed with a differential statistic that isolates the pairwise kSZ signal, providing th…
▽ More
We detect the kinematic Sunyaev-Zel'dovich (kSZ) effect with a statistical significance of $4.2 σ$ by combining a cluster catalogue derived from the first year data of the Dark Energy Survey (DES) with CMB temperature maps from the South Pole Telescope Sunyaev-Zel'dovich (SPT-SZ) Survey. This measurement is performed with a differential statistic that isolates the pairwise kSZ signal, providing the first detection of the large-scale, pairwise motion of clusters using redshifts derived from photometric data. By fitting the pairwise kSZ signal to a theoretical template we measure the average central optical depth of the cluster sample, $\barτ_e = (3.75 \pm 0.89)\cdot 10^{-3}$. We compare the extracted signal to realistic simulations and find good agreement with respect to the signal-to-noise, the constraint on $\barτ_e$, and the corresponding gas fraction. High-precision measurements of the pairwise kSZ signal with future data will be able to place constraints on the baryonic physics of galaxy clusters, and could be used to probe gravity on scales $ \gtrsim 100$ Mpc.
△ Less
Submitted 25 July, 2016; v1 submitted 12 March, 2016;
originally announced March 2016.
-
Simulations of the Pairwise Kinematic Sunyaev-Zel'dovich Signal
Authors:
Samuel Flender,
Lindsey Bleem,
Hal Finkel,
Salman Habib,
Katrin Heitmann,
Gilbert Holder
Abstract:
The pairwise kinematic Sunyaev-Zel'dovich (kSZ) signal from galaxy clusters is a probe of their line-of-sight momenta, and thus a potentially valuable source of cosmological information. In addition to the momenta, the amplitude of the measured signal depends on the properties of the intra-cluster gas and observational limitations such as errors in determining cluster centers and redshifts. In thi…
▽ More
The pairwise kinematic Sunyaev-Zel'dovich (kSZ) signal from galaxy clusters is a probe of their line-of-sight momenta, and thus a potentially valuable source of cosmological information. In addition to the momenta, the amplitude of the measured signal depends on the properties of the intra-cluster gas and observational limitations such as errors in determining cluster centers and redshifts. In this work we simulate the pairwise kSZ signal of clusters at z<1, using the output from a cosmological N-body simulation and including the properties of the intra-cluster gas via a model that can be varied in post-processing. We find that modifications to the gas profile due to star formation and feedback reduce the pairwise kSZ amplitude of clusters by ~50%, relative to the naive 'gas traces mass' assumption. We demonstrate that mis-centering can reduce the overall amplitude of the pairwise kSZ signal by up to 10%, while redshift errors can lead to an almost complete suppression of the signal at small separations. We confirm that a high-significance detection is expected from the combination of data from current-generation, high-resolution CMB experiments, such as the South Pole Telescope, and cluster samples from optical photometric surveys, such as the Dark Energy Survey. Furthermore, we forecast that future experiments such as Advanced ACTPol in conjunction with data from the Dark Energy Spectroscopic Instrument will yield detection significances of at least 20σ, and up to 57σ in an optimistic scenario. Our simulated maps are publicly available at: http://www.hep.anl.gov/cosmology/ksz.html
△ Less
Submitted 30 June, 2016; v1 submitted 9 November, 2015;
originally announced November 2015.
-
The small scale power asymmetry in the cosmic microwave background
Authors:
Samuel Flender,
Shaun Hotchkiss
Abstract:
We investigate the hemispherical power asymmetry in the cosmic microwave background on small angular scales. We find an anomalously high asymmetry in the multipole range l=601-2048, with a naive statistical significance of 6.5 sigma. However, we show that this extreme anomaly is simply a coincidence of three other effects, relativistic power modulation, edge effects from the mask applied, and inte…
▽ More
We investigate the hemispherical power asymmetry in the cosmic microwave background on small angular scales. We find an anomalously high asymmetry in the multipole range l=601-2048, with a naive statistical significance of 6.5 sigma. However, we show that this extreme anomaly is simply a coincidence of three other effects, relativistic power modulation, edge effects from the mask applied, and inter-scale correlations. After correcting for all of these effects, the significance level drops to ~1 sigma, i.e., there is no anomalous intrinsic asymmetry in the small angular scales. Using this null result, we derive a constraint on a potential dipolar modulation amplitude, A(k)<0.0045 on the ~10 Mpc-scale, at 95% C.L. This new constraint must be satisfied by any theoretical model attempting to explain the hemispherical asymmetry at large angular scales.
△ Less
Submitted 27 September, 2013; v1 submitted 23 July, 2013;
originally announced July 2013.
-
The stacked ISW signal of rare superstructures in ΛCDM
Authors:
Samuel Flender,
Shaun Hotchkiss,
Seshadri Nadathur
Abstract:
A detection of the stacked integrated Sachs-Wolfe (ISW) signal in the CMB of rare superstructures identified in the SDSS Luminous Red Galaxy catalogue has been reported at very high statistical significance. The magnitude of the observed signal has previously been argued to be more than 3 standard deviations larger than the theoretical ΛCDM expectation. However, this calculation was made in the li…
▽ More
A detection of the stacked integrated Sachs-Wolfe (ISW) signal in the CMB of rare superstructures identified in the SDSS Luminous Red Galaxy catalogue has been reported at very high statistical significance. The magnitude of the observed signal has previously been argued to be more than 3 standard deviations larger than the theoretical ΛCDM expectation. However, this calculation was made in the linear approximation, and relied on assumptions that may potentially have caused the ΛCDM expectation to be underestimated. Here we update the theoretical model calculation and compare it with an analysis of ISW maps obtained from N-body simulations of a ΛCDM universe. The differences between model predictions and the map analyses are found to be small and cannot explain the discrepancy with observation, which remains at >3 s.d. significance. We discuss the cosmological significance of this anomaly and speculate on the potential of alternative models to explain it.
△ Less
Submitted 21 January, 2013; v1 submitted 4 December, 2012;
originally announced December 2012.
-
Newtonian versus relativistic cosmology
Authors:
Samuel F. Flender,
Dominik J. Schwarz
Abstract:
We show how the relativistic matter and velocity power spectra behave in different gauges. We construct a new gauge where both spectra coincide with Newtonian theory on all scales. However, in this gauge there are geometric quantities present which do not exist in Newtonian theory, for example the local variation of the Hubble parameter. Comparing this quantity to second order Newtonian quantities…
▽ More
We show how the relativistic matter and velocity power spectra behave in different gauges. We construct a new gauge where both spectra coincide with Newtonian theory on all scales. However, in this gauge there are geometric quantities present which do not exist in Newtonian theory, for example the local variation of the Hubble parameter. Comparing this quantity to second order Newtonian quantities, we find that Newtonian theory is inaccurate on scales larger than 10 Mpc. This stresses the importance of relativistic corrections to Newtonian cosmological N-body simulations on these scales.
△ Less
Submitted 5 September, 2012; v1 submitted 9 July, 2012;
originally announced July 2012.
