Climate Dynamics

, Volume 45, Issue 5–6, pp 1299–1324 | Cite as

Identifying and removing structural biases in climate models with history matching

  • Daniel WilliamsonEmail author
  • Adam T. Blaker
  • Charlotte Hampton
  • James Salter


We describe the method of history matching, a method currently used to help quantify parametric uncertainty in climate models, and argue for its use in identifying and removing structural biases in climate models at the model development stage. We illustrate the method using an investigation of the potential to improve upon known ocean circulation biases in a coupled non-flux-adjusted climate model (the third Hadley Centre Climate Model; HadCM3). In particular, we use history matching to investigate whether or not the behaviour of the Antarctic Circumpolar Current (ACC), which is known to be too strong in HadCM3, represents a structural bias that could be corrected using the model parameters. We find that it is possible to improve the ACC strength using the parameters and observe that doing this leads to more realistic representations of the sub-polar and sub-tropical gyres, sea surface salinities (both globally and in the North Atlantic), sea surface temperatures in the sinking regions in the North Atlantic and in the Southern Ocean, North Atlantic Deep Water flows, global precipitation, wind fields and sea level pressure. We then use history matching to locate a region of parameter space predicted not to contain structural biases for ACC and SSTs that is around 1 % of the original parameter space. We explore qualitative features of this space and show that certain key ocean and atmosphere parameters must be tuned carefully together in order to locate climates that satisfy our chosen metrics. Our study shows that attempts to tune climate model parameters that vary only a handful of parameters relevant to a given process at a time will not be as successful or as efficient as history matching.


Tuning Ensembles Emulators HadCM3 Climate model 



This research was funded by the NERC RAPID-RAPIT project (NE/G015368/1). Daniel Williamson was also funded by an EPSRC fellowship, grant number EP/K019112/1. We would like to thank the CPDN team for their work on submitting our ensemble to CPDN users. We’d also like to thank the Institute of Advanced Study at Durham University for funding and hosting our workshop on ocean model discrepancy which formed the motivation for these investigations. In addition, we thank the oceanographers who participated in this workshop. We’d like to thank the CPDN users around the world who contributed their spare computing resource as part of the generation of our ensemble. NCEP and CMAP Precipitation data were provided by the NOAA/OAR/ESRL PSD, Boulder, Colorado, USA, from their website at

Supplementary material

382_2014_2378_MOESM1_ESM.pdf (72 kb)
Supplementary material 1 (pdf 72 KB)
382_2014_2378_MOESM2_ESM.txt (5.1 mb)
Supplementary material 2 (txt 5259 KB)
382_2014_2378_MOESM3_ESM.txt (8 kb)
Supplementary material 3 (txt 7 KB)


  1. Acreman DM, Jeffery CD (2007) The use of Argo for validation and tuning of mixed layer models. Ocean Model 19:53–69CrossRefGoogle Scholar
  2. Annan JD, Hargreaves JC, Edwards NR, Marsh R (2005) Parameter estimation in an intermediate complexity earth system model using an ensemble Kalman filter. Ocean Model 8:135–154CrossRefGoogle Scholar
  3. Annan JD, Lunt DJ, Hargreaves JC, Valdes PJ (2005) Parameter estimation in an atmospheric GCM using the ensemble Kalman filter. Nonlinear Process Geophys 12:363–371CrossRefGoogle Scholar
  4. Annan JD, Hargreaves JC, Ohgaito R, Abe-Ouchi A, Emori S (2005) Efficiently constraining climate sensitivity with ensembles of paleoclimate simulations. SOLA 1:181–184. doi: 10.2151/sola.2005-047 CrossRefGoogle Scholar
  5. Bellprat O, Kotlarski S, Luthi D, Schar C (2012) Objective calibration of regional climate models. J Geophys Res 117:D23115CrossRefGoogle Scholar
  6. Blaker AT, Hirschi JJ-M, McCarthy G, Sinha B, Taws S, Marsh R, de Cuevas BA, Alderson SG, Coward AC (2014) Historical analogues of the recent extreme minima observed in the Atlantic meridional overturning circulation at 26ºN. Clim Dyn. doi: 10.1007/s00382-014-2274-6
  7. Challenor P, McNeall D, Gattiker J (2009) Assessing the probability of rare climate events. In: O’Hagan A, West M (eds) The handbook of applied Bayesian analysis, chap 10. Oxford University Press, OxfordGoogle Scholar
  8. Collins M, Brierley CM, MacVean M, Booth BBB, Harris GR (2007) The sensitivity of the rate of transient climate change to ocean physics perturbations. J Clim 20:2315–2320CrossRefGoogle Scholar
  9. Craig PS, Goldstein M, Seheult AH, Smith JA, Berger JO (1996) Bayes linear strategies for matching hydrocarbon reservoir history. In: Bernado JM, Dawid AP, Smith AFM (eds) Bayesian statistics 5. Oxford University Press, Oxford, pp 69–95Google Scholar
  10. Craig PS, Goldstein M, Rougier JC, Seheult AH (2001) Bayesian forecasting for complex systems using computer simulators. J Am Stat Assoc 96:717–729CrossRefGoogle Scholar
  11. Cumming JA, Goldstein M (2009) Small sample designs for complex high-dimensional models based on fast approximations. Technometrics 51:377–388CrossRefGoogle Scholar
  12. Cunningham SA, Alderson SG, King BA (2003) Transport and variability of the Antarctic circumpolar current in Drake passage. J Geophys Res 108(C5):8084. doi: 10.1029/2001JC001147 CrossRefGoogle Scholar
  13. Daniel C (1973) One at a time plans. J Am Stat Assoc 68:353–360CrossRefGoogle Scholar
  14. Draper NR, Smith H (1998) Applied regression analysis, 3rd edn. Wiley, New YorkGoogle Scholar
  15. Edwards NR, Cameron D, Rougier JC (2011) Precalibrating an intermediate complexity climate model. Clim Dyn 37:1469–1482CrossRefGoogle Scholar
  16. Fisher R (1926) The arrangement of field experiments. J Minist Agric Great Br 33:503–513Google Scholar
  17. Friedman M, Savage LJ (1947) Planning experiments seeking maxima. In: Eisnenhart C, Hastay MW, Wallis WA (eds) Techniques of statistical analysis. McGraw-Hill, New YorkGoogle Scholar
  18. Gent PR, Danabasoglu G, Donner LJ, Holland MM, Hunke EC, Jayne SR, Lawrence DM, Neale RB, Rasch PJ, Vertenstein M, Worley PH, Yang Z, Zhang M (2011) The community climate system model version 4. J Clim 24:4973–4991CrossRefGoogle Scholar
  19. Gladstone RM, Lee V, Rougier JC, Payne AJ, Hellmer H, Le Brocq A, Shepherd A, Edwards TL, Gregory J, Cornford SL (2012) Calibrated prediction of Pine Island Glacier retreat drink the 21st and 22nd centuries with a coupled flow line model. Earth Planet Sci Lett 333–334:191–199CrossRefGoogle Scholar
  20. Goldstein M, Rougier JC (2009) Reified Bayesian modelling and inference for physical systems. J Stat Plan Inference 139:1221–1239CrossRefGoogle Scholar
  21. Gordon C, Cooper C, Senior CA, Banks H, Gregory JM, Johns TC, Mitchell JFB, Wood RA (2000) The simulation of SST, sea ice extents and ocean heat transports in a version of the Hadley Centre coupled model without flux adjustments. Clim Dyn 16:147–168CrossRefGoogle Scholar
  22. Hargreaves JC, Annan JD, Edwards NR, Marsh R (2004) A efficient climate forecasting method using an intermediate complexity earch system model and the ensemble Kalman filter. Clim Dyn 23:745–760CrossRefGoogle Scholar
  23. Haylock R, O’Hagan A (1996) On inference for outputs of computationally expensive algorithms with uncertainty on the inputs. In: Berger JM, Dawid JO, Smith AFM (eds) Bayesian statistics 5. Oxford University Press, Oxford, pp 629–637Google Scholar
  24. Ingleby B, Huddleston M (2007) Quality control of ocean temperature and salinity profiles—historical and real-time data. J Mar Syst 65:158–175. doi: 10.1016/j.jmarsys.2005.11.019 CrossRefGoogle Scholar
  25. Johns et al (2006) The New Hadley Centre Climate Model (HadGEM1): evaluation of coupled simulations. J Clim 19:1327–1353CrossRefGoogle Scholar
  26. Johns et al (2008) Variability of shallow and deep Western boundary currents off the Bahamas during 2004-05: results from the 26N RAPID-MOC array. J Phys Oceanogr 38:605–623CrossRefGoogle Scholar
  27. Joshi MM, Webb MJ, Maycock AC, Collins M (2010) Stratospheric water vapour and high climate sensitivity in a version of the HadSM3 climate model. Atmos Chem Phys 10:7161–7167CrossRefGoogle Scholar
  28. Kalnay et al (1996) The NCEP/NCAR 40-year reanalysis project. Bull Am Meteorol Soc 77:437–470CrossRefGoogle Scholar
  29. Kennedy MC, O’Hagan A (2000) Predicting the output from a complex computer code when fast approximations are available. Biometrika 87:1–13Google Scholar
  30. Kennedy MC, O’Hagan A (2001) Bayesian calibration of computer models. J R Stat Soc Ser B 63:425–464CrossRefGoogle Scholar
  31. Kraus EB, Turner J (1967) A one dimensional model of the seasonal thermocline II. The general theory and its consequences. Tellus 19:98106CrossRefGoogle Scholar
  32. Le Gratiet L (2014) Bayesian analysis of hierarchical multifidelity codes, SIAM. J Uncertain Quantif 1:244–269CrossRefGoogle Scholar
  33. Lee LA, Carslaw KS, Pringle KJ, Mann GW, Spracklen DV (2011) Emulation of a complex global aerosol model to quantify sensitivity to uncertain parameters. Atmos Chem Phys 11:12253–12273. doi: 10.5194/acp-11-12253-2011 CrossRefGoogle Scholar
  34. Loeppky JL, Sacks J, Welch WJ (2009) Choosing the sample size of a computer experiment: a practical guide. Technometrics 51(4):366–376CrossRefGoogle Scholar
  35. Martin GM, Milton SF, Senior CA, Brooks ME, Ineson S, Reichler T, Kim J (2010) Analysis and reduction of systematic errors through a seamless approach to modelling weather and climate. J Clim 23:5933–5957CrossRefGoogle Scholar
  36. Martin GM et al (2011) The HadGEM2 family of met office unified model climate configurations. Geosci Model Dev 4:723–757Google Scholar
  37. Mauritsen T, Stevens B, Roeckner E, Crueger T, Esch M, Giorgetta M, Haak H, Jungclaus J, Klocke D, Matei D, Mikolajewicz U, Notz D, Pincus R, Schmidt H, Tomassini L (2012) Tuning the climate of a global model. J Adv Model Earth Syst 4:M00A01. doi: 10.1029/2012MS000154 CrossRefGoogle Scholar
  38. McNeall DJ, Challenor PG, Gattiker JR, Stone EJ (2013) The potential of an observational data set for calibration of a computationally expensive computer model. Geosci Model Dev 6:1715–1728CrossRefGoogle Scholar
  39. Meehl GA, Covey C, Delworth T, Latif M, McAvaney B, Mitchell JFB, Stouffer RJ, Taylor KE (2007) The WCRP CMIP3 multi-model dataset: a new era in climate change research. Bull Am Meteorol Soc 88:1383–1394CrossRefGoogle Scholar
  40. Megann AP et al (2010) The sensitivity of a coupled climate model to its ocean component. J Clim 23:5126–5150. doi: 10.1175/2010JCLI3394.1 CrossRefGoogle Scholar
  41. Meijers AJS, Shuckburgh E, Bruneau N, Sallee JB, Bracegirdle TJ, Wang Z (2012) Representation of the Antarctic circumpolar current in the CMIP5 climate models and future changes under warming scenarios. J Geophys Res 117:C12008. doi: 10.1029/2012JC008412 CrossRefGoogle Scholar
  42. Meinen CS, Garzoli SL (2014) Attribution of deep western boundary current variability at 26.5 N. Deep Sea Res I 90:81–90. doi: 10.1016/j.dsr.2014.04.016
  43. Murphy JM, Sexton DMH, Jenkins GJ, Booth BBB, Brown CC, Clark RT, Collins M, Harris GR, Kendon EJ, Betts RA, Brown SJ, Humphrey KA, McCarthy MP, McDonald RE, Stephens A, Wallace C, Warren R, Wilby R, Wood R (2009) UK climate projections science report: climate change projections. Met Office Hadley Centre, Exeter.
  44. Pope VD, Gallani ML, Rowntree PR, Stratton RA (2000) The impact of new physical parameterizations in the Hadley Centre climate model: HadAM3. Clim Dyn 16:123–146CrossRefGoogle Scholar
  45. Pukelsheim F (1994) The three sigma rule. Am Stat 48:88–91Google Scholar
  46. Randall DA et al (2007) Climate models and their evaluation. In: Solomon S et al (eds) Climate change 2007: the physical science basis. Cambridge University Press, Cambridge, pp 589–662Google Scholar
  47. Rougier JC (2007) Probabilistic inference for future climate using an ensemble of climate model evaluations. Clim Change 81:247–264CrossRefGoogle Scholar
  48. Rougier JC (2008) Efficient emulators for multivariate deterministic functions. J Comput Graph Stat 17:827–843CrossRefGoogle Scholar
  49. Rougier JC, Sexton DMH, Murphy JM, Stainforth D (2009) Emulating the sensitivity of the HadSM3 climate model using ensembles from different but related experiments. J Clim 22:3540–3557CrossRefGoogle Scholar
  50. Rougier JC (2013) ‘Intractable and unsolved’: some thoughts on statistical data assimilation with uncertain static parameters. Phil Trans R Soc A 371:20120297. doi: 10.1098/rsta.2012.0297 CrossRefGoogle Scholar
  51. Rowlands DJ, Frame DJ, Ackerley D, Aina T, Booth BBB, Christensen C, Collins M, Faull N, Forest CE, Grandey BS, Gryspeerdt E, Highwood EJ, Ingram WJ, Knight S, Lopez A, Massey N, McNamara F, Meinshausen N, Piani C, Rosier SM, Sanderson BJ, Smith LA, Stone DA, Thurston M, Yamazaki K, Yamazaki YH, Allen MR (2012) Broad range of 2050 warming from an observationally constrained large climate model ensemble. Nat Geosci doi: 10.1038/NGEO1430
  52. Russell JL, Stouffer RJ, Dixon KW (2006) Intercomparisons of the Southern Ocean circulations in IPCC coupled model control simulations. J Clim 19:4560–4575Google Scholar
  53. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4:409–435CrossRefGoogle Scholar
  54. Santner TJ, Williams BJ, Notz WI (2003) The design and analysis of computer experiments. Springer, New YorkCrossRefGoogle Scholar
  55. Schmittner A, Urban NM, Shakun JD, Mahowald NM, Clark PU, Bartlein PJ, Mix AC, Rosell-Mele A (2011) Climate sensitivity estimated from temperature reconstructions of the last glacial maximum. Science 334:1385CrossRefGoogle Scholar
  56. Severijns CA, Hazeleger W (2005) Optimizing parameters in an atmospheric general circulation model. J Clim 18:3527–3535CrossRefGoogle Scholar
  57. Sexton DMH, Murphy JM, Collins M (2011) Multivariate probabilistic projections using imperfect climate models part 1: outline of methodology. Clim Dyn. doi: 10.1007/s00382-011-1208-9
  58. Shaffrey L et al (2009) UK-HiGEM: the new UK high resolution global environment model. Model description and basic evaluation. J Clim 22(8):1861–1896. doi: 10.1175/2008JCLI2508.1 CrossRefGoogle Scholar
  59. Solomon S, Qin D, Manning M, Chen Z, Marquis M, Averyt KB, Tignor M, Miller HL (eds) (2007) Contribution of working group I to the fourth assessment report of the intergovernmental panel on climate change, 2007. Cambridge University Press, CambridgeGoogle Scholar
  60. Uppala et al (2005) The ERA-40 re-analysis. Q J R Meteorol Soc 131:2961–3012CrossRefGoogle Scholar
  61. Vernon I, Goldstein M, Bower RG (2010) Galaxy formation: a Bayesian uncertainty analysis. Bayesian Anal 5(4):619–846 with DiscussionCrossRefGoogle Scholar
  62. Watanabe M, Suzuki T, O’Ishi R, Komuro Y, Watanabe S, Emori S, Takemura T, Chikira M, Ogura T, Sekiguchi M, Takata K, Yamazaki D, Yokohata T, Nozawa T, Hasumi H, Tatebe H, Kimoto M (2010) Improved climate simulation by MIROC5: mean states, variability and climate sensitivity. J Clim 23:6312–6335CrossRefGoogle Scholar
  63. Williamson D (2010) Policy making using computer simulators for complex physical systems: Bayesian decision support for the development of adaptive strategies, Ph.D. thesis, Durham UniversityGoogle Scholar
  64. Williamson D, Goldstein M, Blaker A (2012) Fast linked analyses for scenario based hierarchies. J R Stat Soc Ser C 61(5):665–692. doi: 10.1111/j.1467-9876.2012.01042.x
  65. Williamson D, Goldstein M, Allison L, Blaker A, Challenor P, Jackson L, Yamazaki K (2013) History matching for exploring and reducing climate model parameter space using observations and a large perturbed physics ensemble. Clim Dyn 41:1703-1729. doi: 10.1007/s00382-013-1896-4
  66. Williamson D, Blaker AT (2014) Evolving Bayesian emulators for structurally chaotic time series with application to large climate models. SIAM/ASA J Uncertain Quantif 2(1):1–28. doi: 10.1137/120900915
  67. Williamson D, Vernon IR (2013) Implausibility driven evolutionary Monte Carlo for efficient generation of uniform and optimal designs for multi-wave computer experiments. arXiv:1309.3520 (in revision)Google Scholar
  68. Xie P, Arkin PA (1997) Global precipitation: a 17-year monthly analysis based on gauge observations, satellite estimates, and numerical model outputs. Bull Am Meteorol Soc 78:2539–2558CrossRefGoogle Scholar
  69. Yamazaki K, Rowlands DJ, Aina T, Blaker A, Bowery A, Massey N, Miller J, Rye C, Tett SFB, Williamson D, Yamazaki YH, Allen MR (2012) Obtaining diverse behaviours in a climate model without the use of flux adjustments. J Geophys Res Atmos 118:2781-2793. doi: 10.1002/jgrd.50304

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Daniel Williamson
    • 1
    Email author
  • Adam T. Blaker
    • 2
  • Charlotte Hampton
    • 1
    • 2
  • James Salter
    • 1
  1. 1.College of Engineering, Mathematics and Physical SciencesUniversity of ExeterExeterUK
  2. 2.National Oceanography CentreSouthamptonUK

Personalised recommendations