# Identifying and removing structural biases in climate models with history matching

## Abstract

We describe the method of history matching, a method currently used to help quantify parametric uncertainty in climate models, and argue for its use in identifying and removing structural biases in climate models at the model development stage. We illustrate the method using an investigation of the potential to improve upon known ocean circulation biases in a coupled non-flux-adjusted climate model (the third Hadley Centre Climate Model; HadCM3). In particular, we use history matching to investigate whether or not the behaviour of the Antarctic Circumpolar Current (ACC), which is known to be too strong in HadCM3, represents a structural bias that could be corrected using the model parameters. We find that it is possible to improve the ACC strength using the parameters and observe that doing this leads to more realistic representations of the sub-polar and sub-tropical gyres, sea surface salinities (both globally and in the North Atlantic), sea surface temperatures in the sinking regions in the North Atlantic and in the Southern Ocean, North Atlantic Deep Water flows, global precipitation, wind fields and sea level pressure. We then use history matching to locate a region of parameter space predicted not to contain structural biases for ACC and SSTs that is around 1 % of the original parameter space. We explore qualitative features of this space and show that certain key ocean and atmosphere parameters must be tuned carefully together in order to locate climates that satisfy our chosen metrics. Our study shows that attempts to tune climate model parameters that vary only a handful of parameters relevant to a given process at a time will not be as successful or as efficient as history matching.

## Keywords

Tuning Ensembles Emulators HadCM3 Climate model## Notes

### Acknowledgments

This research was funded by the NERC RAPID-RAPIT project (NE/G015368/1). Daniel Williamson was also funded by an EPSRC fellowship, grant number EP/K019112/1. We would like to thank the CPDN team for their work on submitting our ensemble to CPDN users. We’d also like to thank the Institute of Advanced Study at Durham University for funding and hosting our workshop on ocean model discrepancy which formed the motivation for these investigations. In addition, we thank the oceanographers who participated in this workshop. We’d like to thank the CPDN users around the world who contributed their spare computing resource as part of the generation of our ensemble. NCEP and CMAP Precipitation data were provided by the NOAA/OAR/ESRL PSD, Boulder, Colorado, USA, from their website at http://www.esrl.noaa.gov/psd/.

## Supplementary material

## References

- Acreman DM, Jeffery CD (2007) The use of Argo for validation and tuning of mixed layer models. Ocean Model 19:53–69CrossRefGoogle Scholar
- Annan JD, Hargreaves JC, Edwards NR, Marsh R (2005) Parameter estimation in an intermediate complexity earth system model using an ensemble Kalman filter. Ocean Model 8:135–154CrossRefGoogle Scholar
- Annan JD, Lunt DJ, Hargreaves JC, Valdes PJ (2005) Parameter estimation in an atmospheric GCM using the ensemble Kalman filter. Nonlinear Process Geophys 12:363–371CrossRefGoogle Scholar
- Annan JD, Hargreaves JC, Ohgaito R, Abe-Ouchi A, Emori S (2005) Efficiently constraining climate sensitivity with ensembles of paleoclimate simulations. SOLA 1:181–184. doi: 10.2151/sola.2005-047 CrossRefGoogle Scholar
- Bellprat O, Kotlarski S, Luthi D, Schar C (2012) Objective calibration of regional climate models. J Geophys Res 117:D23115CrossRefGoogle Scholar
- Blaker AT, Hirschi JJ-M, McCarthy G, Sinha B, Taws S, Marsh R, de Cuevas BA, Alderson SG, Coward AC (2014) Historical analogues of the recent extreme minima observed in the Atlantic meridional overturning circulation at 26ºN. Clim Dyn. doi: 10.1007/s00382-014-2274-6
- Challenor P, McNeall D, Gattiker J (2009) Assessing the probability of rare climate events. In: O’Hagan A, West M (eds) The handbook of applied Bayesian analysis, chap 10. Oxford University Press, OxfordGoogle Scholar
- Collins M, Brierley CM, MacVean M, Booth BBB, Harris GR (2007) The sensitivity of the rate of transient climate change to ocean physics perturbations. J Clim 20:2315–2320CrossRefGoogle Scholar
- Craig PS, Goldstein M, Seheult AH, Smith JA, Berger JO (1996) Bayes linear strategies for matching hydrocarbon reservoir history. In: Bernado JM, Dawid AP, Smith AFM (eds) Bayesian statistics 5. Oxford University Press, Oxford, pp 69–95Google Scholar
- Craig PS, Goldstein M, Rougier JC, Seheult AH (2001) Bayesian forecasting for complex systems using computer simulators. J Am Stat Assoc 96:717–729CrossRefGoogle Scholar
- Cumming JA, Goldstein M (2009) Small sample designs for complex high-dimensional models based on fast approximations. Technometrics 51:377–388CrossRefGoogle Scholar
- Cunningham SA, Alderson SG, King BA (2003) Transport and variability of the Antarctic circumpolar current in Drake passage. J Geophys Res 108(C5):8084. doi: 10.1029/2001JC001147 CrossRefGoogle Scholar
- Daniel C (1973) One at a time plans. J Am Stat Assoc 68:353–360CrossRefGoogle Scholar
- Draper NR, Smith H (1998) Applied regression analysis, 3rd edn. Wiley, New YorkGoogle Scholar
- Edwards NR, Cameron D, Rougier JC (2011) Precalibrating an intermediate complexity climate model. Clim Dyn 37:1469–1482CrossRefGoogle Scholar
- Fisher R (1926) The arrangement of field experiments. J Minist Agric Great Br 33:503–513Google Scholar
- Friedman M, Savage LJ (1947) Planning experiments seeking maxima. In: Eisnenhart C, Hastay MW, Wallis WA (eds) Techniques of statistical analysis. McGraw-Hill, New YorkGoogle Scholar
- Gent PR, Danabasoglu G, Donner LJ, Holland MM, Hunke EC, Jayne SR, Lawrence DM, Neale RB, Rasch PJ, Vertenstein M, Worley PH, Yang Z, Zhang M (2011) The community climate system model version 4. J Clim 24:4973–4991CrossRefGoogle Scholar
- Gladstone RM, Lee V, Rougier JC, Payne AJ, Hellmer H, Le Brocq A, Shepherd A, Edwards TL, Gregory J, Cornford SL (2012) Calibrated prediction of Pine Island Glacier retreat drink the 21st and 22nd centuries with a coupled flow line model. Earth Planet Sci Lett 333–334:191–199CrossRefGoogle Scholar
- Goldstein M, Rougier JC (2009) Reified Bayesian modelling and inference for physical systems. J Stat Plan Inference 139:1221–1239CrossRefGoogle Scholar
- Gordon C, Cooper C, Senior CA, Banks H, Gregory JM, Johns TC, Mitchell JFB, Wood RA (2000) The simulation of SST, sea ice extents and ocean heat transports in a version of the Hadley Centre coupled model without flux adjustments. Clim Dyn 16:147–168CrossRefGoogle Scholar
- Hargreaves JC, Annan JD, Edwards NR, Marsh R (2004) A efficient climate forecasting method using an intermediate complexity earch system model and the ensemble Kalman filter. Clim Dyn 23:745–760CrossRefGoogle Scholar
- Haylock R, O’Hagan A (1996) On inference for outputs of computationally expensive algorithms with uncertainty on the inputs. In: Berger JM, Dawid JO, Smith AFM (eds) Bayesian statistics 5. Oxford University Press, Oxford, pp 629–637Google Scholar
- Ingleby B, Huddleston M (2007) Quality control of ocean temperature and salinity profiles—historical and real-time data. J Mar Syst 65:158–175. doi: 10.1016/j.jmarsys.2005.11.019 CrossRefGoogle Scholar
- Johns et al (2006) The New Hadley Centre Climate Model (HadGEM1): evaluation of coupled simulations. J Clim 19:1327–1353CrossRefGoogle Scholar
- Johns et al (2008) Variability of shallow and deep Western boundary currents off the Bahamas during 2004-05: results from the 26N RAPID-MOC array. J Phys Oceanogr 38:605–623CrossRefGoogle Scholar
- Joshi MM, Webb MJ, Maycock AC, Collins M (2010) Stratospheric water vapour and high climate sensitivity in a version of the HadSM3 climate model. Atmos Chem Phys 10:7161–7167CrossRefGoogle Scholar
- Kalnay et al (1996) The NCEP/NCAR 40-year reanalysis project. Bull Am Meteorol Soc 77:437–470CrossRefGoogle Scholar
- Kennedy MC, O’Hagan A (2000) Predicting the output from a complex computer code when fast approximations are available. Biometrika 87:1–13Google Scholar
- Kennedy MC, O’Hagan A (2001) Bayesian calibration of computer models. J R Stat Soc Ser B 63:425–464CrossRefGoogle Scholar
- Kraus EB, Turner J (1967) A one dimensional model of the seasonal thermocline II. The general theory and its consequences. Tellus 19:98106CrossRefGoogle Scholar
- Le Gratiet L (2014) Bayesian analysis of hierarchical multifidelity codes, SIAM. J Uncertain Quantif 1:244–269CrossRefGoogle Scholar
- Lee LA, Carslaw KS, Pringle KJ, Mann GW, Spracklen DV (2011) Emulation of a complex global aerosol model to quantify sensitivity to uncertain parameters. Atmos Chem Phys 11:12253–12273. doi: 10.5194/acp-11-12253-2011 CrossRefGoogle Scholar
- Loeppky JL, Sacks J, Welch WJ (2009) Choosing the sample size of a computer experiment: a practical guide. Technometrics 51(4):366–376CrossRefGoogle Scholar
- Martin GM, Milton SF, Senior CA, Brooks ME, Ineson S, Reichler T, Kim J (2010) Analysis and reduction of systematic errors through a seamless approach to modelling weather and climate. J Clim 23:5933–5957CrossRefGoogle Scholar
- Martin GM et al (2011) The HadGEM2 family of met office unified model climate configurations. Geosci Model Dev 4:723–757Google Scholar
- Mauritsen T, Stevens B, Roeckner E, Crueger T, Esch M, Giorgetta M, Haak H, Jungclaus J, Klocke D, Matei D, Mikolajewicz U, Notz D, Pincus R, Schmidt H, Tomassini L (2012) Tuning the climate of a global model. J Adv Model Earth Syst 4:M00A01. doi: 10.1029/2012MS000154 CrossRefGoogle Scholar
- McNeall DJ, Challenor PG, Gattiker JR, Stone EJ (2013) The potential of an observational data set for calibration of a computationally expensive computer model. Geosci Model Dev 6:1715–1728CrossRefGoogle Scholar
- Meehl GA, Covey C, Delworth T, Latif M, McAvaney B, Mitchell JFB, Stouffer RJ, Taylor KE (2007) The WCRP CMIP3 multi-model dataset: a new era in climate change research. Bull Am Meteorol Soc 88:1383–1394CrossRefGoogle Scholar
- Megann AP et al (2010) The sensitivity of a coupled climate model to its ocean component. J Clim 23:5126–5150. doi: 10.1175/2010JCLI3394.1 CrossRefGoogle Scholar
- Meijers AJS, Shuckburgh E, Bruneau N, Sallee JB, Bracegirdle TJ, Wang Z (2012) Representation of the Antarctic circumpolar current in the CMIP5 climate models and future changes under warming scenarios. J Geophys Res 117:C12008. doi: 10.1029/2012JC008412 CrossRefGoogle Scholar
- Meinen CS, Garzoli SL (2014) Attribution of deep western boundary current variability at 26.5 N. Deep Sea Res I 90:81–90. doi: 10.1016/j.dsr.2014.04.016
- Murphy JM, Sexton DMH, Jenkins GJ, Booth BBB, Brown CC, Clark RT, Collins M, Harris GR, Kendon EJ, Betts RA, Brown SJ, Humphrey KA, McCarthy MP, McDonald RE, Stephens A, Wallace C, Warren R, Wilby R, Wood R (2009) UK climate projections science report: climate change projections. Met Office Hadley Centre, Exeter. http://ukclimateprojections.defra.gov.uk/images/stories/projections_pdfs/UKCP09_Projections_V2
- Pope VD, Gallani ML, Rowntree PR, Stratton RA (2000) The impact of new physical parameterizations in the Hadley Centre climate model: HadAM3. Clim Dyn 16:123–146CrossRefGoogle Scholar
- Pukelsheim F (1994) The three sigma rule. Am Stat 48:88–91Google Scholar
- Randall DA et al (2007) Climate models and their evaluation. In: Solomon S et al (eds) Climate change 2007: the physical science basis. Cambridge University Press, Cambridge, pp 589–662Google Scholar
- Rougier JC (2007) Probabilistic inference for future climate using an ensemble of climate model evaluations. Clim Change 81:247–264CrossRefGoogle Scholar
- Rougier JC (2008) Efficient emulators for multivariate deterministic functions. J Comput Graph Stat 17:827–843CrossRefGoogle Scholar
- Rougier JC, Sexton DMH, Murphy JM, Stainforth D (2009) Emulating the sensitivity of the HadSM3 climate model using ensembles from different but related experiments. J Clim 22:3540–3557CrossRefGoogle Scholar
- Rougier JC (2013) ‘Intractable and unsolved’: some thoughts on statistical data assimilation with uncertain static parameters. Phil Trans R Soc A 371:20120297. doi: 10.1098/rsta.2012.0297 CrossRefGoogle Scholar
- Rowlands DJ, Frame DJ, Ackerley D, Aina T, Booth BBB, Christensen C, Collins M, Faull N, Forest CE, Grandey BS, Gryspeerdt E, Highwood EJ, Ingram WJ, Knight S, Lopez A, Massey N, McNamara F, Meinshausen N, Piani C, Rosier SM, Sanderson BJ, Smith LA, Stone DA, Thurston M, Yamazaki K, Yamazaki YH, Allen MR (2012) Broad range of 2050 warming from an observationally constrained large climate model ensemble. Nat Geosci doi: 10.1038/NGEO1430
- Russell JL, Stouffer RJ, Dixon KW (2006) Intercomparisons of the Southern Ocean circulations in IPCC coupled model control simulations. J Clim 19:4560–4575Google Scholar
- Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4:409–435CrossRefGoogle Scholar
- Santner TJ, Williams BJ, Notz WI (2003) The design and analysis of computer experiments. Springer, New YorkCrossRefGoogle Scholar
- Schmittner A, Urban NM, Shakun JD, Mahowald NM, Clark PU, Bartlein PJ, Mix AC, Rosell-Mele A (2011) Climate sensitivity estimated from temperature reconstructions of the last glacial maximum. Science 334:1385CrossRefGoogle Scholar
- Severijns CA, Hazeleger W (2005) Optimizing parameters in an atmospheric general circulation model. J Clim 18:3527–3535CrossRefGoogle Scholar
- Sexton DMH, Murphy JM, Collins M (2011) Multivariate probabilistic projections using imperfect climate models part 1: outline of methodology. Clim Dyn. doi: 10.1007/s00382-011-1208-9
- Shaffrey L et al (2009) UK-HiGEM: the new UK high resolution global environment model. Model description and basic evaluation. J Clim 22(8):1861–1896. doi: 10.1175/2008JCLI2508.1 CrossRefGoogle Scholar
- Solomon S, Qin D, Manning M, Chen Z, Marquis M, Averyt KB, Tignor M, Miller HL (eds) (2007) Contribution of working group I to the fourth assessment report of the intergovernmental panel on climate change, 2007. Cambridge University Press, CambridgeGoogle Scholar
- Uppala et al (2005) The ERA-40 re-analysis. Q J R Meteorol Soc 131:2961–3012CrossRefGoogle Scholar
- Vernon I, Goldstein M, Bower RG (2010) Galaxy formation: a Bayesian uncertainty analysis. Bayesian Anal 5(4):619–846 with DiscussionCrossRefGoogle Scholar
- Watanabe M, Suzuki T, O’Ishi R, Komuro Y, Watanabe S, Emori S, Takemura T, Chikira M, Ogura T, Sekiguchi M, Takata K, Yamazaki D, Yokohata T, Nozawa T, Hasumi H, Tatebe H, Kimoto M (2010) Improved climate simulation by MIROC5: mean states, variability and climate sensitivity. J Clim 23:6312–6335CrossRefGoogle Scholar
- Williamson D (2010) Policy making using computer simulators for complex physical systems: Bayesian decision support for the development of adaptive strategies, Ph.D. thesis, Durham UniversityGoogle Scholar
- Williamson D, Goldstein M, Blaker A (2012) Fast linked analyses for scenario based hierarchies. J R Stat Soc Ser C 61(5):665–692. doi: 10.1111/j.1467-9876.2012.01042.x
- Williamson D, Goldstein M, Allison L, Blaker A, Challenor P, Jackson L, Yamazaki K (2013) History matching for exploring and reducing climate model parameter space using observations and a large perturbed physics ensemble. Clim Dyn 41:1703-1729. doi: 10.1007/s00382-013-1896-4
- Williamson D, Blaker AT (2014) Evolving Bayesian emulators for structurally chaotic time series with application to large climate models. SIAM/ASA J Uncertain Quantif 2(1):1–28. doi: 10.1137/120900915
- Williamson D, Vernon IR (2013) Implausibility driven evolutionary Monte Carlo for efficient generation of uniform and optimal designs for multi-wave computer experiments. arXiv:1309.3520 (in revision)Google Scholar
- Xie P, Arkin PA (1997) Global precipitation: a 17-year monthly analysis based on gauge observations, satellite estimates, and numerical model outputs. Bull Am Meteorol Soc 78:2539–2558CrossRefGoogle Scholar
- Yamazaki K, Rowlands DJ, Aina T, Blaker A, Bowery A, Massey N, Miller J, Rye C, Tett SFB, Williamson D, Yamazaki YH, Allen MR (2012) Obtaining diverse behaviours in a climate model without the use of flux adjustments. J Geophys Res Atmos 118:2781-2793. doi: 10.1002/jgrd.50304