-
Fast Emulation and Modular Calibration for Simulators with Functional Response
Authors:
Grant Hutchings,
Derek Bingham,
Earl Lawrence
Abstract:
Scalable surrogate models enable efficient emulation of computer models (or simulators), particularly when dealing with large ensembles of runs. While Gaussian Process (GP) models are commonly employed for emulation, they face limitations in scaling to truly large datasets. Furthermore, when dealing with dense functional output, such as spatial or time-series data, additional complexities arise, r…
▽ More
Scalable surrogate models enable efficient emulation of computer models (or simulators), particularly when dealing with large ensembles of runs. While Gaussian Process (GP) models are commonly employed for emulation, they face limitations in scaling to truly large datasets. Furthermore, when dealing with dense functional output, such as spatial or time-series data, additional complexities arise, requiring careful handling to ensure fast emulation. This work presents a highly scalable emulator for functional data, building upon the works of Kennedy and O'Hagan (2001) and Higdon et al. (2008), while incorporating the local approximate Gaussian Process framework proposed by Gramacy and Apley (2015). The emulator utilizes global GP lengthscale parameter estimates to scale the input space, leading to a substantial improvement in prediction speed. We demonstrate that our fast approximation-based emulator can serve as a viable alternative to the methods outlined in Higdon et al. (2008) for functional response, while drastically reducing computational costs. The proposed emulator is applied to quickly calibrate the multiphysics continuum hydrodynamics simulator FLAG with a large ensemble of 20000 runs. The methods presented are implemented in the R package FlaGP.
△ Less
Submitted 25 May, 2024;
originally announced May 2024.
-
The Mira-Titan Universe IV. High Precision Power Spectrum Emulation
Authors:
Kelly R. Moran,
Katrin Heitmann,
Earl Lawrence,
Salman Habib,
Derek Bingham,
Amol Upadhye,
Juliana Kwan,
David Higdon,
Richard Payne
Abstract:
Modern cosmological surveys are delivering datasets characterized by unprecedented quality and statistical completeness; this trend is expected to continue into the future as new ground- and space-based surveys come online. In order to maximally extract cosmological information from these observations, matching theoretical predictions are needed. At low redshifts, the surveys probe the nonlinear r…
▽ More
Modern cosmological surveys are delivering datasets characterized by unprecedented quality and statistical completeness; this trend is expected to continue into the future as new ground- and space-based surveys come online. In order to maximally extract cosmological information from these observations, matching theoretical predictions are needed. At low redshifts, the surveys probe the nonlinear regime of structure formation where cosmological simulations are the primary means of obtaining the required information. The computational cost of sufficiently resolved large-volume simulations makes it prohibitive to run very large ensembles. Nevertheless, precision emulators built on a tractable number of high-quality simulations can be used to build very fast prediction schemes to enable a variety of cosmological inference studies. We have recently introduced the Mira-Titan Universe simulation suite designed to construct emulators for a range of cosmological probes. The suite covers the standard six cosmological parameters $\{ω_m,ω_b, σ_8, h, n_s, w_0\}$ and, in addition, includes massive neutrinos and a dynamical dark energy equation of state, $\{ω_ν, w_a\}$. In this paper we present the final emulator for the matter power spectrum based on 111 cosmological simulations, each covering a (2.1Gpc)$^3$ volume and evolving 3200$^3$ particles. An additional set of 1776 lower-resolution simulations and TimeRG perturbation theory results for the power spectrum are used to cover scales straddling the linear to mildly nonlinear regimes. The emulator provides predictions at the two to three percent level of accuracy over a wide range of cosmological parameters and is publicly released as part of this paper.
△ Less
Submitted 25 July, 2022;
originally announced July 2022.
-
Let's practice what we preach: Planning and interpreting simulation studies with design and analysis of experiments
Authors:
Hugh Chipman,
Derek Bingham
Abstract:
Statisticians recommend the Design and Analysis of Experiments (DAE) for evidence-based research but often use tables to present their own simulation studies. Could DAE do better? We outline how DAE methods can be used to plan and analyze simulation studies. Tools for planning include fishbone diagrams, factorial and fractional factorial designs. Analysis is carried out via ANOVA, main-effect and…
▽ More
Statisticians recommend the Design and Analysis of Experiments (DAE) for evidence-based research but often use tables to present their own simulation studies. Could DAE do better? We outline how DAE methods can be used to plan and analyze simulation studies. Tools for planning include fishbone diagrams, factorial and fractional factorial designs. Analysis is carried out via ANOVA, main-effect and interaction plots and other DAE tools. We also demonstrate how Taguchi Robust Parameter Design can be used to study the robustness of methods to a variety of uncontrollable population parameters.
△ Less
Submitted 26 November, 2021;
originally announced November 2021.
-
Uncertainty Quantification of a Computer Model for Binary Black Hole Formation
Authors:
Luyao Lin,
Derek Bingham,
Floor Broekgaarden,
Ilya Mandel
Abstract:
In this paper, a fast and parallelizable method based on Gaussian Processes (GPs) is introduced to emulate computer models that simulate the formation of binary black holes (BBHs) through the evolution of pairs of massive stars. Two obstacles that arise in this application are the a priori unknown conditions of BBH formation and the large scale of the simulation data. We address them by proposing…
▽ More
In this paper, a fast and parallelizable method based on Gaussian Processes (GPs) is introduced to emulate computer models that simulate the formation of binary black holes (BBHs) through the evolution of pairs of massive stars. Two obstacles that arise in this application are the a priori unknown conditions of BBH formation and the large scale of the simulation data. We address them by proposing a local emulator which combines a GP classifier and a GP regression model. The resulting emulator can also be utilized in planning future computer simulations through a proposed criterion for sequential design. By propagating uncertainties of simulation input through the emulator, we are able to obtain the distribution of BBH properties under the distribution of physical parameters.
△ Less
Submitted 2 June, 2021;
originally announced June 2021.
-
LRP2020: Astrostatistics in Canada
Authors:
Gwendolyn Eadie,
Arash Bahramian,
Pauline Barmby,
Radu Craiu,
Derek Bingham,
Renée Hložek,
JJ Kavelaars,
David Stenning,
Samantha Benincasa,
Guillaume Thomas,
Karun Thanjavur,
Jo Bovy,
Jan Cami,
Ray Carlberg,
Sam Lawler,
Adrian Liu,
Henry Ngo,
Mubdi Rahman,
Michael Rupen
Abstract:
(Abridged from Executive Summary) This white paper focuses on the interdisciplinary fields of astrostatistics and astroinformatics, in which modern statistical and computational methods are applied to and developed for astronomical data. Astrostatistics and astroinformatics have grown dramatically in the past ten years, with international organizations, societies, conferences, workshops, and summe…
▽ More
(Abridged from Executive Summary) This white paper focuses on the interdisciplinary fields of astrostatistics and astroinformatics, in which modern statistical and computational methods are applied to and developed for astronomical data. Astrostatistics and astroinformatics have grown dramatically in the past ten years, with international organizations, societies, conferences, workshops, and summer schools becoming the norm. Canada's formal role in astrostatistics and astroinformatics has been relatively limited, but there is a great opportunity and necessity for growth in this area. We conducted a survey of astronomers in Canada to gain information on the training mechanisms through which we learn statistical methods and to identify areas for improvement. In general, the results of our survey indicate that while astronomers see statistical methods as critically important for their research, they lack focused training in this area and wish they had received more formal training during all stages of education and professional development. These findings inform our recommendations for the LRP2020 on how to increase interdisciplinary connections between astronomy and statistics at the institutional, national, and international levels over the next ten years. We recommend specific, actionable ways to increase these connections, and discuss how interdisciplinary work can benefit not only research but also astronomy's role in training Highly Qualified Personnel (HQP) in Canada.
△ Less
Submitted 19 October, 2019;
originally announced October 2019.
-
The Mira-Titan Universe II: Matter Power Spectrum Emulation
Authors:
Earl Lawrence,
Katrin Heitmann,
Juliana Kwan,
Amol Upadhye,
Derek Bingham,
Salman Habib,
David Higdon,
Adrian Pope,
Hal Finkel,
Nicholas Frontiere
Abstract:
We introduce a new cosmic emulator for the matter power spectrum covering eight cosmological parameters. Targeted at optical surveys, the emulator provides accurate predictions out to a wavenumber k~5/Mpc and redshift z<=2. Besides covering the standard set of LCDM parameters, massive neutrinos and a dynamical dark energy of state are included. The emulator is built on a sample set of 36 cosmologi…
▽ More
We introduce a new cosmic emulator for the matter power spectrum covering eight cosmological parameters. Targeted at optical surveys, the emulator provides accurate predictions out to a wavenumber k~5/Mpc and redshift z<=2. Besides covering the standard set of LCDM parameters, massive neutrinos and a dynamical dark energy of state are included. The emulator is built on a sample set of 36 cosmological models, carefully chosen to provide accurate predictions over the wide and large parameter space. For each model, we have performed a high-resolution simulation, augmented with sixteen medium-resolution simulations and TimeRG perturbation theory results to provide accurate coverage of a wide k-range; the dataset generated as part of this project is more than 1.2Pbyte. With the current set of simulated models, we achieve an accuracy of approximately 4%. Because the sampling approach used here has established convergence and error-control properties, follow-on results with more than a hundred cosmological models will soon achieve ~1% accuracy. We compare our approach with other prediction schemes that are based on halo model ideas and remapping approaches. The new emulator code is publicly available.
△ Less
Submitted 9 May, 2017;
originally announced May 2017.
-
Estimating parameter uncertainty in binding-energy models by the frequency-domain bootstrap
Authors:
G. F. Bertsch,
Derek Bingham
Abstract:
We propose using the frequency-domain bootstrap (FDB) to estimate errors of modeling parameters when the modeling error is itself a major source of uncertainty. Unlike the usual bootstrap or the simple $χ^2$ analysis, the FDB can take into account correlations between errors. It is also very fast compared to the the Gaussian process Bayesian estimate as often implemented for computer model calibra…
▽ More
We propose using the frequency-domain bootstrap (FDB) to estimate errors of modeling parameters when the modeling error is itself a major source of uncertainty. Unlike the usual bootstrap or the simple $χ^2$ analysis, the FDB can take into account correlations between errors. It is also very fast compared to the the Gaussian process Bayesian estimate as often implemented for computer model calibration. The method is illustrated drop model of nuclear binding energies. We find that the FDB gives a more conservative estimate of the uncertainty in liquid drop parameters in better accord with more empirical estimates. For the nuclear physics application, there no apparent obstacle to apply the method to the more accurate and detailed models based on density-functional theory.
△ Less
Submitted 26 March, 2017;
originally announced March 2017.
-
A regional compound Poisson process for hurricane and tropical storm damage
Authors:
Simon Mak,
Derek Bingham,
Yi Lu
Abstract:
In light of intense hurricane activity along the U.S. Atlantic coast, attention has turned to understanding both the economic impact and behaviour of these storms. The compound Poisson-lognormal process has been proposed as a model for aggregate storm damage, but does not shed light on regional analysis since storm path data are not used. In this paper, we propose a fully Bayesian regional predict…
▽ More
In light of intense hurricane activity along the U.S. Atlantic coast, attention has turned to understanding both the economic impact and behaviour of these storms. The compound Poisson-lognormal process has been proposed as a model for aggregate storm damage, but does not shed light on regional analysis since storm path data are not used. In this paper, we propose a fully Bayesian regional prediction model which uses conditional autoregressive (CAR) models to account for both storm paths and spatial patterns for storm damage. When fitted to historical data, the analysis from our model both confirms previous findings and reveals new insights on regional storm tendencies. Posterior predictive samples can also be used for pricing regional insurance premiums, which we illustrate using three different risk measures.
△ Less
Submitted 11 February, 2016;
originally announced February 2016.
-
Design of Computer Experiments for Optimization, Estimation of Function Contours, and Related Objectives
Authors:
Derek Bingham,
Pritam Ranjan,
William Welch
Abstract:
A computer code or simulator is a mathematical representation of a physical system, for example a set of differential equations. Running the code with given values of the vector of inputs, x, leads to an output y(x) or several such outputs. For instance, one application we use for illustration simulates the average tidal power, y, generated as a function of the turbine location, x = (x1, x2), in t…
▽ More
A computer code or simulator is a mathematical representation of a physical system, for example a set of differential equations. Running the code with given values of the vector of inputs, x, leads to an output y(x) or several such outputs. For instance, one application we use for illustration simulates the average tidal power, y, generated as a function of the turbine location, x = (x1, x2), in the Bay of Fundy, Nova Scotia, Canada (Ranjan et al. 2011). Performing scientific or engineering experiments via such a computer code is often more time and cost effective than running a physical experiment.
Choosing new runs sequentially for optimization, moving y to a target, etc. has been formalized using the concept of expected improvement (Jones et al. 1998). The next experimental run is made where the expected improvement in the function of interest is largest. This expectation is with respect to the predictive distribution of y from a statistical model relating y to x. By considering a set of possible inputs x for the new run, we can choose that which gives the largest expectation.
△ Less
Submitted 22 January, 2016;
originally announced January 2016.
-
The Mira-Titan Universe: Precision Predictions for Dark Energy Surveys
Authors:
Katrin Heitmann,
Derek Bingham,
Earl Lawrence,
Steven Bergner,
Salman Habib,
David Higdon,
Adrian Pope,
Rahul Biswas,
Hal Finkel,
Nicholas Frontiere,
Suman Bhattacharya
Abstract:
Ground and space-based sky surveys enable powerful cosmological probes based on measurements of galaxy properties and the distribution of galaxies in the Universe. These probes include weak lensing, baryon acoustic oscillations, abundance of galaxy clusters, and redshift space distortions; they are essential to improving our knowledge of the nature of dark energy. On the theory and modeling front,…
▽ More
Ground and space-based sky surveys enable powerful cosmological probes based on measurements of galaxy properties and the distribution of galaxies in the Universe. These probes include weak lensing, baryon acoustic oscillations, abundance of galaxy clusters, and redshift space distortions; they are essential to improving our knowledge of the nature of dark energy. On the theory and modeling front, large-scale simulations of cosmic structure formation play an important role in interpreting the observations and in the challenging task of extracting cosmological physics at the needed precision. These simulations must cover a parameter range beyond the standard six cosmological parameters and need to be run at high mass and force resolution. One key simulation-based task is the generation of accurate theoretical predictions for observables, via the method of emulation. Using a new sampling technique, we explore an 8-dimensional parameter space including massive neutrinos and a variable dark energy equation of state. We construct trial emulators using two surrogate models (the linear power spectrum and an approximate halo mass function). The new sampling method allows us to build precision emulators from just 26 cosmological models and to increase the emulator accuracy by adding new sets of simulations in a prescribed way. This allows emulator fidelity to be systematically improved as new observational data becomes available and higher accuracy is required. Finally, using one LCDM cosmology as an example, we study the demands imposed on a simulation campaign to achieve the required statistics and accuracy when building emulators for dark energy investigations.
△ Less
Submitted 11 August, 2015;
originally announced August 2015.
-
Calibrating a large computer experiment simulating radiative shock hydrodynamics
Authors:
Robert B. Gramacy,
Derek Bingham,
James Paul Holloway,
Michael J. Grosskopf,
Carolyn C. Kuranz,
Erica Rutter,
Matt Trantham,
R. Paul Drake
Abstract:
We consider adapting a canonical computer model calibration apparatus, involving coupled Gaussian process (GP) emulators, to a computer experiment simulating radiative shock hydrodynamics that is orders of magnitude larger than what can typically be accommodated. The conventional approach calls for thousands of large matrix inverses to evaluate the likelihood in an MCMC scheme. Our approach replac…
▽ More
We consider adapting a canonical computer model calibration apparatus, involving coupled Gaussian process (GP) emulators, to a computer experiment simulating radiative shock hydrodynamics that is orders of magnitude larger than what can typically be accommodated. The conventional approach calls for thousands of large matrix inverses to evaluate the likelihood in an MCMC scheme. Our approach replaces that costly ideal with a thrifty take on essential ingredients, synergizing three modern ideas in emulation, calibration and optimization: local approximate GP regression, modularization, and mesh adaptive direct search. The new methodology is motivated both by necessity - considering our particular application - and by recent trends in the supercomputer simulation literature. A synthetic data application allows us to explore the merits of several variations in a controlled environment and, together with results on our motivating real-data experiment, lead to noteworthy insights into the dynamics of radiative shocks as well as the limitations of the calibration enterprise generally.
△ Less
Submitted 5 November, 2015; v1 submitted 13 October, 2014;
originally announced October 2014.
-
Monotone Function Estimation for Computer Experiments
Authors:
Shirin Golchi,
Derek R. Bingham,
Hugh Chipman,
David A. Campbell
Abstract:
In statistical modeling of computer experiments sometimes prior information is available about the underlying function. For example, the physical system simulated by the computer code may be known to be monotone with respect to some or all inputs. We develop a Bayesian approach to Gaussian process modelling capable of incorporating monotonicity information for computer model emulation. Markov chai…
▽ More
In statistical modeling of computer experiments sometimes prior information is available about the underlying function. For example, the physical system simulated by the computer code may be known to be monotone with respect to some or all inputs. We develop a Bayesian approach to Gaussian process modelling capable of incorporating monotonicity information for computer model emulation. Markov chain Monte Carlo methods are used to sample from the posterior distribution of the process given the simulator output and monotonicity information. The performance of the proposed approach in terms of predictive accuracy and uncertainty quantification is demonstrated in a number of simulated examples as well as a real queueing system application.
△ Less
Submitted 14 June, 2014; v1 submitted 15 September, 2013;
originally announced September 2013.
-
Parameter tuning for a multi-fidelity dynamical model of the magnetosphere
Authors:
William Kleiber,
Stephan R. Sain,
Matthew J. Heaton,
Michael Wiltberger,
C. Shane Reese,
Derek Bingham
Abstract:
Geomagnetic storms play a critical role in space weather physics with the potential for far reaching economic impacts including power grid outages, air traffic rerouting, satellite damage and GPS disruption. The LFM-MIX is a state-of-the-art coupled magnetospheric-ionospheric model capable of simulating geomagnetic storms. Imbedded in this model are physical equations for turning the magnetohydrod…
▽ More
Geomagnetic storms play a critical role in space weather physics with the potential for far reaching economic impacts including power grid outages, air traffic rerouting, satellite damage and GPS disruption. The LFM-MIX is a state-of-the-art coupled magnetospheric-ionospheric model capable of simulating geomagnetic storms. Imbedded in this model are physical equations for turning the magnetohydrodynamic state parameters into energy and flux of electrons entering the ionosphere, involving a set of input parameters. The exact values of these input parameters in the model are unknown, and we seek to quantify the uncertainty about these parameters when model output is compared to observations. The model is available at different fidelities: a lower fidelity which is faster to run, and a higher fidelity but more computationally intense version. Model output and observational data are large spatiotemporal systems; the traditional design and analysis of computer experiments is unable to cope with such large data sets that involve multiple fidelities of model output. We develop an approach to this inverse problem for large spatiotemporal data sets that incorporates two different versions of the physical model. After an initial design, we propose a sequential design based on expected improvement. For the LFM-MIX, the additional run suggested by expected improvement diminishes posterior uncertainty by ruling out a posterior mode and shrinking the width of the posterior distribution. We also illustrate our approach using the Lorenz `96 system of equations for a simplified atmosphere, using known input parameters. For the Lorenz `96 system, after performing sequential runs based on expected improvement, the posterior mode converges to the true value and the posterior variability is reduced.
△ Less
Submitted 5 December, 2013; v1 submitted 27 March, 2013;
originally announced March 2013.
-
Prediction and Computer Model Calibration Using Outputs From Multi-fidelity Simulators
Authors:
Joslin Goh,
Derek Bingham,
James Paul Holloway,
Michael J. Grosskopf,
Carolyn C. Kuranz,
Erica Rutter
Abstract:
Computer codes are widely used to describe physical processes in lieu of physical observations. In some cases, more than one computer simulator, each with different degrees of fidelity, can be used to explore the physical system. In this work, we combine field observations and model runs from deterministic multi-fidelity computer simulators to build a predictive model for the real process. The res…
▽ More
Computer codes are widely used to describe physical processes in lieu of physical observations. In some cases, more than one computer simulator, each with different degrees of fidelity, can be used to explore the physical system. In this work, we combine field observations and model runs from deterministic multi-fidelity computer simulators to build a predictive model for the real process. The resulting model can be used to perform sensitivity analysis for the system, solve inverse problems and make predictions. Our approach is Bayesian and will be illustrated through a simple example, as well as a real application in predictive science at the Center for Radiative Shock Hydrodynamics at the University of Michigan.
△ Less
Submitted 13 August, 2012;
originally announced August 2012.
-
Efficient emulators of computer experiments using compactly supported correlation functions, with an application to cosmology
Authors:
Cari G. Kaufman,
Derek Bingham,
Salman Habib,
Katrin Heitmann,
Joshua A. Frieman
Abstract:
Statistical emulators of computer simulators have proven to be useful in a variety of applications. The widely adopted model for emulator building, using a Gaussian process model with strictly positive correlation function, is computationally intractable when the number of simulator evaluations is large. We propose a new model that uses a combination of low-order regression terms and compactly sup…
▽ More
Statistical emulators of computer simulators have proven to be useful in a variety of applications. The widely adopted model for emulator building, using a Gaussian process model with strictly positive correlation function, is computationally intractable when the number of simulator evaluations is large. We propose a new model that uses a combination of low-order regression terms and compactly supported correlation functions to recreate the desired predictive behavior of the emulator at a fraction of the computational cost. Following the usual approach of taking the correlation to be a product of correlations in each input dimension, we show how to impose restrictions on the ranges of the correlations, giving sparsity, while also allowing the ranges to trade off against one another, thereby giving good predictive performance. We illustrate the method using data from a computer simulator of photometric redshift with 20,000 simulator evaluations and 80,000 predictions.
△ Less
Submitted 28 February, 2012; v1 submitted 4 July, 2011;
originally announced July 2011.
-
A new and flexible method for constructing designs for computer experiments
Authors:
C. Devon Lin,
Derek Bingham,
Randy R. Sitter,
Boxin Tang
Abstract:
We develop a new method for constructing "good" designs for computer experiments. The method derives its power from its basic structure that builds large designs using small designs. We specialize the method for the construction of orthogonal Latin hypercubes and obtain many results along the way. In terms of run sizes, the existence problem of orthogonal Latin hypercubes is completely solved. We…
▽ More
We develop a new method for constructing "good" designs for computer experiments. The method derives its power from its basic structure that builds large designs using small designs. We specialize the method for the construction of orthogonal Latin hypercubes and obtain many results along the way. In terms of run sizes, the existence problem of orthogonal Latin hypercubes is completely solved. We also present an explicit result showing how large orthogonal Latin hypercubes can be constructed using small orthogonal Latin hypercubes. Another appealing feature of our method is that it can easily be adapted to construct other designs; we examine how to make use of the method to construct nearly orthogonal and cascading Latin hypercubes.
△ Less
Submitted 2 October, 2010;
originally announced October 2010.
-
Existence and construction of randomization defining contrast subspaces for regular factorial designs
Authors:
Pritam Ranjan,
Derek R. Bingham,
Angela M. Dean
Abstract:
Regular factorial designs with randomization restrictions are widely used in practice. This paper provides a unified approach to the construction of such designs using randomization defining contrast subspaces for the representation of randomization restrictions. We use finite projective geometry to determine the existence of designs with the required structure and develop a systematic approach…
▽ More
Regular factorial designs with randomization restrictions are widely used in practice. This paper provides a unified approach to the construction of such designs using randomization defining contrast subspaces for the representation of randomization restrictions. We use finite projective geometry to determine the existence of designs with the required structure and develop a systematic approach for their construction. An attractive feature is that commonly used factorial designs with randomization restrictions are special cases of this general representation. Issues related to the use of these designs for particular factorial experiments are also addressed.
△ Less
Submitted 2 September, 2009;
originally announced September 2009.
