-
Considering discrepancy when calibrating a mechanistic electrophysiology model
Authors:
Chon Lok Lei,
Sanmitra Ghosh,
Dominic G. Whittaker,
Yasser Aboelkassem,
Kylie A. Beattie,
Chris D. Cantwell,
Tammo Delhaas,
Charles Houston,
Gustavo Montes Novaes,
Alexander V. Panfilov,
Pras Pathmanathan,
Marina Riabiz,
Rodrigo Weber dos Santos,
John Walmsley,
Keith Worden,
Gary R. Mirams,
Richard D. Wilkinson
Abstract:
Uncertainty quantification (UQ) is a vital step in using mathematical models and simulations to take decisions. The field of cardiac simulation has begun to explore and adopt UQ methods to characterise uncertainty in model inputs and how that propagates through to outputs or predictions. In this perspective piece we draw attention to an important and under-addressed source of uncertainty in our pr…
▽ More
Uncertainty quantification (UQ) is a vital step in using mathematical models and simulations to take decisions. The field of cardiac simulation has begun to explore and adopt UQ methods to characterise uncertainty in model inputs and how that propagates through to outputs or predictions. In this perspective piece we draw attention to an important and under-addressed source of uncertainty in our predictions -- that of uncertainty in the model structure or the equations themselves. The difference between imperfect models and reality is termed model discrepancy, and we are often uncertain as to the size and consequences of this discrepancy. Here we provide two examples of the consequences of discrepancy when calibrating models at the ion channel and action potential scales. Furthermore, we attempt to account for this discrepancy when calibrating and validating an ion channel model using different methods, based on modelling the discrepancy using Gaussian processes (GPs) and autoregressive-moving-average (ARMA) models, then highlight the advantages and shortcomings of each approach. Finally, suggestions and lines of enquiry for future work are provided.
△ Less
Submitted 23 April, 2020; v1 submitted 13 January, 2020;
originally announced January 2020.
-
Quantifying Age and Model Uncertainties in Paleoclimate Data and Dynamical Climate Models with a Joint Inferential Analysis
Authors:
Jake Carson,
Michel Crucifix,
Simon P. Preston,
Richard D. Wilkinson
Abstract:
A major goal in paleoclimate science is to reconstruct historical climates using proxies for climate variables such as those observed in sediment cores, and in the process learn about climate dynamics. This is hampered by uncertainties in how sediment core depths relate to ages, how proxy quantities relate to climate variables, how climate models are specified, and the values of parameters in clim…
▽ More
A major goal in paleoclimate science is to reconstruct historical climates using proxies for climate variables such as those observed in sediment cores, and in the process learn about climate dynamics. This is hampered by uncertainties in how sediment core depths relate to ages, how proxy quantities relate to climate variables, how climate models are specified, and the values of parameters in climate models. Quantifying these uncertainties is key in drawing well founded conclusions. Analyses are often performed in separate stages with, for example, a sediment core's depth-age relation being estimated as stage one, then fed as an input to calibrate climate models as stage two. Here, we show that such "multi-stage" approaches can lead to misleading conclusions. We develop a joint inferential approach for climate reconstruction, model calibration, and age model estimation. We focus on the glacial-interglacial cycle over the past 780 kyr, analysing two sediment cores that span this range. Our age estimates are largely in agreement with previous studies, but provides the full joint specification of all uncertainties, estimation of model parameters, and the model evidence. By sampling plausible chronologies from the posterior distribution, we demonstrate that downstream scientific conclusions can differ greatly both between different sampled chronologies, and in comparison with conclusions obtained in the complete joint inferential analysis. We conclude that multi-stage analyses are insufficient when dealing with uncertainty, and that to draw sound conclusions the full joint inferential analysis must be performed.
△ Less
Submitted 17 April, 2019; v1 submitted 22 March, 2018;
originally announced March 2018.
-
Detecting non-binomial sex allocation when developmental mortality operates
Authors:
Richard D. Wilkinson,
Apostolos Kapranas,
Ian C. W. Hardy
Abstract:
Optimal sex allocation theory is one of the most intricately developed areas of evolutionary ecology. Under a range of conditions, particularly under population sub-division, selection favours sex being allocated to offspring non-randomly, generating non-binomial variances of offspring group sex ratios. Detecting non-binomial sex allocation is complicated by stochastic developmental mortality, as…
▽ More
Optimal sex allocation theory is one of the most intricately developed areas of evolutionary ecology. Under a range of conditions, particularly under population sub-division, selection favours sex being allocated to offspring non-randomly, generating non-binomial variances of offspring group sex ratios. Detecting non-binomial sex allocation is complicated by stochastic developmental mortality, as offspring sex can often only be identified on maturity with the sex of non-maturing offspring remaining unknown. We show that current approaches for detecting non-binomiality have limited ability to detect non-binomial sex allocation when developmental mortality has occurred. We present a new procedure using an explicit model of sex allocation and mortality and develop a Bayesian model selection approach (available as an R package). We use the double and multiplicative binomial distributions to model over- and under-dispersed sex allocation and show how to calculate Bayes factors for comparing these alternative models to the null hypothesis of binomial sex allocation. The ability to detect non-binomial sex allocation is greatly increased, particularly in cases where mortality is common. The use of Bayesian methods allows for the quantification of the evidence in favour of each hypothesis, and our modelling approach provides an improved descriptive capability over existing approaches. We use a simulation study to demonstrate substantial improvements in power for detecting non-binomial sex allocation in situations where current methods fail, and we illustrate the approach in real scenarios using empirically obtained datasets on the sexual composition of groups of gregarious parasitoid wasps.
△ Less
Submitted 10 August, 2016;
originally announced August 2016.
-
ABC for climate: dealing with expensive simulators
Authors:
Philip B. Holden,
Neil R. Edwards,
James Hensman,
Richard D. Wilkinson
Abstract:
This paper is due to appear as a chapter of the forthcoming Handbook of Approximate Bayesian Computation (ABC) by S. Sisson, L. Fan, and M. Beaumont. We describe the challenge of calibrating climate simulators, and discuss the differences in emphasis in climate science compared to many of the more traditional ABC application areas. The primary difficulty is how to do inference with a computational…
▽ More
This paper is due to appear as a chapter of the forthcoming Handbook of Approximate Bayesian Computation (ABC) by S. Sisson, L. Fan, and M. Beaumont. We describe the challenge of calibrating climate simulators, and discuss the differences in emphasis in climate science compared to many of the more traditional ABC application areas. The primary difficulty is how to do inference with a computationally expensive simulator which we can only afford to run a small number of times, and we describe how Gaussian process emulators are used as surrogate models in this case. We introduce the idea of history matching, which is a non-probabilistic calibration method, which divides the parameter space into (not im)plausible and implausible regions. History matching can be shown to be a special case of ABC, but with a greater emphasis on defining realistic simulator discrepancy bounds, and using these to define tolerances and metrics. We describe a design approach for choosing parameter values at which to run the simulator, and illustrate the approach on a toy climate model, showing that with careful design we can find the plausible region with a very small number of model evaluations. Finally, we describe how calibrated GENIE-1 (an earth system model of intermediate complexity) predictions have been used, and why it is important to accurately characterise parametric uncertainty.
△ Less
Submitted 11 November, 2015;
originally announced November 2015.
-
Bayesian model selection for the glacial-interglacial cycle
Authors:
Jake Carson,
Michel Crucifix,
Simon Preston,
Richard D. Wilkinson
Abstract:
A prevailing viewpoint in palaeoclimate science is that a single palaeoclimate record contains insufficient information to discriminate between most competing explanatory models. Results we present here suggest the contrary. Using SMC^2 combined with novel Brownian bridge type proposals for the state trajectories, we show that even with relatively short time series it is possible to estimate Bayes…
▽ More
A prevailing viewpoint in palaeoclimate science is that a single palaeoclimate record contains insufficient information to discriminate between most competing explanatory models. Results we present here suggest the contrary. Using SMC^2 combined with novel Brownian bridge type proposals for the state trajectories, we show that even with relatively short time series it is possible to estimate Bayes factors to sufficient accuracy to be able to select between competing models. The results show that Monte Carlo methodology and computer power have now advanced to the point where a full Bayesian analysis for a wide class of conceptual climate models is now possible. The results also highlight a problem with estimating the chronology of the climate record prior to further statistical analysis, a practice which is common in palaeoclimate science. Using two datasets based on the same record but with different estimated chronologies results in conflicting conclusions about the importance of the orbital forcing on the glacial cycle, and about the internal dynamics generating the glacial cycle, even though the difference between the two estimated chronologies is consistent with dating uncertainty. This highlights a need for chronology estimation and other inferential questions to be addressed in a joint statistical procedure.
△ Less
Submitted 11 November, 2015;
originally announced November 2015.
-
Accelerating ABC methods using Gaussian processes
Authors:
Richard D Wilkinson
Abstract:
Approximate Bayesian computation (ABC) methods are used to approximate posterior distributions using simulation rather than likelihood calculations. We introduce Gaussian process (GP) accelerated ABC, which we show can significantly reduce the number of simulations required. As computational resource is usually the main determinant of accuracy in ABC, GP-accelerated methods can thus enable more ac…
▽ More
Approximate Bayesian computation (ABC) methods are used to approximate posterior distributions using simulation rather than likelihood calculations. We introduce Gaussian process (GP) accelerated ABC, which we show can significantly reduce the number of simulations required. As computational resource is usually the main determinant of accuracy in ABC, GP-accelerated methods can thus enable more accurate inference in some models. GP models of the unknown log-likelihood function are used to exploit continuity and smoothness, reducing the required computation. We use a sequence of models that increase in accuracy, using intermediate models to rule out regions of the parameter space as implausible. The methods will not be suitable for all problems, but when they can be used, can result in significant computational savings. For the Ricker model, we are able to achieve accurate approximations to the posterior distribution using a factor of 100 fewer simulator evaluations than comparable Monte Carlo approaches, and for a population genetics model we are able to approximate the exact posterior for the first time.
△ Less
Submitted 23 February, 2014; v1 submitted 7 January, 2014;
originally announced January 2014.
-
Approximate Bayesian computation (ABC) gives exact results under the assumption of model error
Authors:
Richard D. Wilkinson
Abstract:
Approximate Bayesian computation (ABC) or likelihood-free inference algorithms are used to find approximations to posterior distributions without making explicit use of the likelihood function, depending instead on simulation of sample data sets from the model. In this paper we show that under the assumption of the existence of a uniform additive model error term, ABC algorithms give exact results…
▽ More
Approximate Bayesian computation (ABC) or likelihood-free inference algorithms are used to find approximations to posterior distributions without making explicit use of the likelihood function, depending instead on simulation of sample data sets from the model. In this paper we show that under the assumption of the existence of a uniform additive model error term, ABC algorithms give exact results when sufficient summaries are used. This interpretation allows the approximation made in many previous application papers to be understood, and should guide the choice of metric and tolerance in future work. ABC algorithms can be generalized by replacing the 0-1 cut-off with an acceptance probability that varies with the distance of the simulated data from the observed data. The acceptance density gives the distribution of the error term, enabling the uniform error usually used to be replaced by a general distribution. This generalization can also be applied to approximate Markov chain Monte Carlo algorithms. In light of this work, ABC algorithms can be seen as calibration techniques for implicit stochastic models, inferring parameter values in light of the computer model, data, prior beliefs about the parameter values, and any measurement or model errors.
△ Less
Submitted 23 April, 2013; v1 submitted 20 November, 2008;
originally announced November 2008.
