Emulated Multivariate Global Sensitivity Analysis for Complex Computer Models Applied to Agricultural Simulators

  • Daniel W. GladishEmail author
  • Ross Darnell
  • Peter J. Thorburn
  • Bhakti Haldankar


Complex mechanistic computer models often produce functional or multivariate output. Sensitivity analysis can be used to determine what input parameters are responsible for uncertainty in the output. Much of the literature around sensitivity analysis has focused on univariate output. Recent advances have been made in sensitivity analysis for multivariate output. However, these methods often depend on a significant number of model runs and may still be computationally intensive for practical purposes. Emulators have been a proven method for reducing the required number of model runs for univariate sensitivity analysis, with some recent development for multivariate computer models. We propose the use of generalized additive models and random forests combined with a principal component analysis for emulation for a multivariate sensitivity analysis. We demonstrate our method using a complex agricultural simulators. Supplementary materials accompanying this paper appear online.


Surrogate model Uncertainty quantification Variance-based sensitivity Sobol indices Generalized sensitivity 



We thank two anonymous referees and the Associated Editor, whose comments have greatly improved this manuscript.


Funding was provided by Science and Industry Endowment Fund (Grant No. PF13-053).

Supplementary material

13253_2018_346_MOESM1_ESM.rar (4 kb)
Supplementary material 1 (rar 4 KB)
13253_2018_346_MOESM2_ESM.rar (20 mb)
Supplementary material 2 (rar 20458 KB)


  1. Adams, B., Ebeida, M., Eldred, M., Jakeman, J., Swiler, L., Stephens, J., Vigil, D., Wildey, T., Bohnhoff, W., Eddy, J., Hu, K., Bauman, L., and Hough, P. (2014). Dakota, a multilevel parallel object-oriented framework for design optimization, parameter estimation, uncertainty quantification, and sensitivity analysis. Technical report, Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States).Google Scholar
  2. Asseng, S., Ewert, F., Rosenzweig, C., Jones, J., Hatfield, J., Ruane, A., Boote, K., Thorburn, P., Rtter, R., Cammarano, D., Brisson, N., Basso, B., Martre, P., Aggarwal, P., Angulo, C., Bertuzzi, P., Biernath, C., Challinor, A., Doltra, J., Gayler, S., Goldberg, R., Grant, R., Heng, L., Hooker, J., Hunt, L., Ingwersen, J., Izaurralde, R., Kersebaum, K., Mller, C., Naresh Kumar, S., Nendel, C., O’Leary, G., Olesen, J., Osborne, T., Palosuo, T., Priesack, E., Ripoche, D., Semenov, M., Shcherbak, I., Steduto, P., Stckle, C., Stratonovitch, P., Streck, T., Supit, I., Tao, F., Travasso, M., Waha, K., Wallach, D., White, J., Williams, J., and Wolf, J. (2013). Uncertainty in simulating wheat yields under climate change. Nature Climate Change, 3(9):827–832.Google Scholar
  3. Bastos, L. and O’Hagan, A. (2009). Diagnostics for gaussian process emulators. Technometrics, 51(4):425–438.MathSciNetGoogle Scholar
  4. Bayarri, M., Berger, J., Cafeo, J., Garcia-Donato, G., Liu, F., Palomo, J., Parthasarathy, R., Paulo, R., Sacks, J., and Walsh, D. (2007). Computer model validation with functional output. Annals of Statistics, 35(5):1874–1906.MathSciNetzbMATHGoogle Scholar
  5. Breiman, L. (2001). Random forests. Machine Learning, 45(1):5–32.zbMATHGoogle Scholar
  6. Campbell, K., McKay, M., and Williams, B. (2006). Sensitivity analysis when model outputs are functions. Reliability Engineering and System Safety, 91(10-11):1468–1472.Google Scholar
  7. Carnell, R. (2016). lhs: Latin Hypercube Samples. R package version 0.14.Google Scholar
  8. Cressie, N. and Wikle, C. K. (2011). Statistics for spatio-temporal data. John Wiley & Sons.Google Scholar
  9. Da Veiga, S. (2015). Global sensitivity analysis with dependence measures. Journal of Statistical Computation and Simulation, 85(7):1283–1305.MathSciNetGoogle Scholar
  10. De Lozzo, M. and Marrel, A. (2017). Sensitivity analysis with dependence and variance-based measures for spatio-temporal numerical simulators. Stochastic Environmental Research and Risk Assessment, 31(6):1437–1453.Google Scholar
  11. Duchon, J. (1977). Splines minimizing rotation-invariant semi-norms in sobolev spaces. In Schempp, W. and Zeller, K., editors, Constructive Theory of Functions of Several Variables, pages 85–100, Berlin, Heidelberg. Springer Berlin Heidelberg.Google Scholar
  12. Gamboa, F., Janon, A., Klein, T., and Lagnoux, A. (2013). Sensitivity indices for multivariate outputs. Comptes Rendus Mathematique, 351(7-8):307–310.MathSciNetzbMATHGoogle Scholar
  13. Gamboa, F., Janon, A., Klein, T., and Lagnoux, A. (2014). Sensitivity analysis for multidimensional and functional outputs. Electronic Journal of Statistics, 8(1):575–603.MathSciNetzbMATHGoogle Scholar
  14. Garcia-Cabrejo, O. and Valocchi, A. (2014). Global sensitivity analysis for multivariate output using polynomial chaos expansion. Reliability Engineering and System Safety, 126:25–36.Google Scholar
  15. Gladish, D., Pagendam, D., Peeters, L., Kuhnert, P., and Vaze, J. (2018). Emulation engines: Choice and quantification of uncertainty for complex hydrological models. Journal of Agricultural, Biological, and Environmental Statistics, 23(1):39–62.MathSciNetzbMATHGoogle Scholar
  16. Gretton, A., Bousquet, O., Smola, A., and Schölkopf, B. (2005). Measuring statistical dependence with hilbert-schmidt norms. In Jain, S., Simon, H., and Tomita, E., editors, Algorithmic Learning Theory, pages 63–77, Berlin, Heidelberg. Springer Berlin Heidelberg.Google Scholar
  17. Gupta, H. and Razavi, S. (2016). Challenges and Future Outlook of Sensitivity Analysis.Google Scholar
  18. Hamby, D. (1994). A review of techniques for parameter sensitivity analysis of environmental models. Environmental Monitoring and Assessment, 32(2):135–154.Google Scholar
  19. Higdon, D., Gattiker, J., Williams, B., and Rightley, M. (2008). Computer model calibration using high-dimensional output. Journal of the American Statistical Association, 103(482):570–583.MathSciNetzbMATHGoogle Scholar
  20. Hochman, Z., Prestwidge, D., and Carberry, P. (2014). Crop sequences in australia’s northern grain zone are less agronomically efficient than implied by the sum of their parts. Agricultural Systems, 129:124–132.Google Scholar
  21. Holzworth, D., Huth, N., deVoil, P., Zurcher, E., Herrmann, N., McLean, G., Chenu, K., van Oosterom, E., Snow, V., Murphy, C., Moore, A., Brown, H., Whish, J., Verrall, S., Fainges, J., Bell, L., Peake, A., Poulton, P., Hochman, Z., Thorburn, P., Gaydon, D., Dalgliesh, N., Rodriguez, D., Cox, H., Chapman, S., Doherty, A., Teixeira, E., Sharp, J., Cichota, R., Vogeler, I., Li, F., Wang, E., Hammer, G., Robertson, M., Dimes, J., Whitbread, A., Hunt, J., van Rees, H., McClelland, T., Carberry, P., Hargreaves, J., MacLeod, N., McDonald, C., Harsdorf, J., Wedgwood, S., and Keating, B. (2014). Apsim - evolution towards a new generation of agricultural systems simulation. Environmental Modelling and Software, 62:327–350.Google Scholar
  22. Holzworth, D., Snow, V., Janssen, S., Athanasiadis, I., Donatelli, M., Hoogenboom, G., White, J., and Thorburn, P. (2015). Agricultural production systems modelling and software: Current status and future prospects. Environmental Modelling and Software, 72:276–286.Google Scholar
  23. Hooten, M., Leeds, W., Fiechter, J., and Wikle, C. (2011). Assessing first-order emulator inference for physical parameters in nonlinear mechanistic models. Journal of Agricultural, Biological, and Environmental Statistics, 16(4):475–494.MathSciNetzbMATHGoogle Scholar
  24. Jansen, M. (1999). Analysis of variance designs for model output. Computer Physics Communications, 117(1):35–43.zbMATHGoogle Scholar
  25. Keating, B., Carberry, P., Hammer, G., Probert, M., Robertson, M., Holzworth, D., Huth, N., Hargreaves, J., Meinke, H., Hochman, Z., McLean, G., Verburg, K., Snow, V., Dimes, J., Silburn, M., Wang, E., Brown, S., Bristow, K., Asseng, S., Chapman, S., McCown, R., Freebairn, D., and Smith, C. (2003). An overview of apsim, a model designed for farming systems simulation. European Journal of Agronomy, 18(3-4):267–288.Google Scholar
  26. Lamboni, M., Makowski, D., Lehuger, S., Gabrielle, B., and Monod, H. (2009). Multivariate global sensitivity analysis for dynamic crop models. Field Crops Research, 113(3):312–320.Google Scholar
  27. Lamboni, M., Monod, H., and Makowski, D. (2011). Multivariate sensitivity analysis to measure global contribution of input factors in dynamic models. Reliability Engineering and System Safety, 96(4):450–459.Google Scholar
  28. Leeds, W., Wikle, C., Fiechter, J., Brown, J., and Milliff, R. (2013). Modeling 3-d spatio-temporal biogeochemical processes with a forest of 1-d statistical emulators. Environmetrics, 24(1):1–12.MathSciNetGoogle Scholar
  29. Li, L., Lu, Z., and Wu, D. (2016). A new kind of sensitivity index for multivariate output. Reliability Engineering and System Safety, 147:123–131.Google Scholar
  30. Liaw, A. and Wiener, M. (2002). Classification and regression by randomforest. R News, 2(3):18–22.Google Scholar
  31. Makowski, D., Naud, C., Jeuffroy, M.-H., Barbottin, A., and Monod, H. (2006). Global sensitivity analysis for calculating the contribution of genetic parameters to the variance of crop model prediction. Reliability Engineering and System Safety, 91(10-11):1142–1147.Google Scholar
  32. Mara, T. and Joseph, O. (2008). Comparison of some efficient methods to evaluate the main effect of computer model factors. Journal of Statistical Computation and Simulation, 78(2):167–178.MathSciNetzbMATHGoogle Scholar
  33. Marrel, A., Iooss, B., Jullien, M., Laurent, B., and Volkova, E. (2011). Global sensitivity analysis for models with spatially dependent outputs. Environmetrics, 22(3):383–397.MathSciNetGoogle Scholar
  34. Morris, M. (1991). Factorial sampling plans for preliminary computational experiments. Technometrics, 33(2):161–174.Google Scholar
  35. Oakley, J. and O’Hagan, A. (2004). Probabilistic sensitivity analysis of complex models: A Bayesian approach. Journal of the Royal Statistical Society. Series B: Statistical Methodology, 66(3):751–769.MathSciNetzbMATHGoogle Scholar
  36. Probert, M., Dimes, J., Keating, B., Dalal, R., and Strong, W. (1998). Apsim’s water and nitrogen modules and simulation of the dynamics of water and nitrogen in fallow systems. Agricultural Systems, 56(1):1–28.Google Scholar
  37. R Core Team (2017). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.Google Scholar
  38. Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., and Tarantola, S. (2010). Variance based sensitivity analysis of model output. design and estimator for the total sensitivity index. Computer Physics Communications, 181(2):259–270.MathSciNetzbMATHGoogle Scholar
  39. Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., and Tarantola, S. (2008). Global Sensitivity Analysis. The Primer.Google Scholar
  40. Saltelli, A., Tarantola, S., Campolongo, F., and Ratto, M. (2004). Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. Halsted Press, New York, NY, USA.zbMATHGoogle Scholar
  41. Shi, J., Wang, B., Murray-Smith, R., and Titterington, D. (2007). Gaussian process functional regression modeling for batch data. Biometrics, 63(3):714–723.MathSciNetzbMATHGoogle Scholar
  42. Sobol, I. (2001). Global sensitivity indices for nonlinear mathematical models and their monte carlo estimates. Mathematics and Computers in Simulation, 55(1-3):271–280.MathSciNetzbMATHGoogle Scholar
  43. Stanfill, B., Mielenz, H., Clifford, D., and Thorburn, P. (2015). Simple approach to emulating complex computer models for global sensitivity analysis. Environmental Modelling and Software, 74:140–155.Google Scholar
  44. Stein, M. (1987). Large sample properties of simulations using latin hypercube sampling. Technometrics, 29(2):143–151.MathSciNetzbMATHGoogle Scholar
  45. Storlie, C. and Helton, J. (2008). Multiple predictor smoothing methods for sensitivity analysis: Description of techniques. Reliability Engineering and System Safety, 93(1):28–54.Google Scholar
  46. Storlie, C., Swiler, L., Helton, J., and Sallaberry, C. (2009). Implementation and evaluation of nonparametric regression procedures for sensitivity analysis of computationally demanding models. Reliability Engineering and System Safety, 94(11):1735–1763.Google Scholar
  47. Strong, M., Oakley, J., and Brennan, A. (2014). Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: A nonparametric regression approach. Medical Decision Making, 34(3):311–326.Google Scholar
  48. Sudret, B. (2008). Global sensitivity analysis using polynomial chaos expansions. Reliability Engineering and System Safety, 93(7):964–979.Google Scholar
  49. Wallach, D., Goffinet, B., Bergez, J.-E., Debaeke, P., Leenhardt, D., and Aubertot, J.-N. (2001). Parameter estimation for crop models: A new approach and application to a corn model. Agronomy Journal, 93(4):757–766.Google Scholar
  50. Wood, S. N. (2006). Generalized additive models: an introduction with R. CRC press.Google Scholar
  51. Xiao, S., Lu, Z., and Wang, P. (2017). Multivariate global sensitivity analysis for dynamic models based on energy distance. Structural and Multidisciplinary Optimization.Google Scholar
  52. Xiao, S., Lu, Z., and Wang, P. (2018). Multivariate global sensitivity analysis for dynamic models based on wavelet analysis. Reliability Engineering and System Safety, 170:20–30.Google Scholar
  53. Zhao, G., Bryan, B. A., and Song, X. (2014). Sensitivity and uncertainty analysis of the apsim-wheat model: Interactions between cultivar, environmental, and management parameters. Ecological Modelling, 279:1 – 11.Google Scholar

Copyright information

© International Biometric Society 2018

Authors and Affiliations

  1. 1.CSIRO Data61, EcoSciences PrecinctBrisbaneAustralia
  2. 2.CSIRO Agriculture and FoodBrisbaneAustralia
  3. 3.University of SydneySydneyAustralia

Personalised recommendations