Advertisement

Methods for the Invalidation/Validation of Simulation Models

  • David J. Murray-Smith
Part of the Simulation Foundations, Methods and Applications book series (SFMA)

Abstract

In the context of simulation model development, the word “validation” is used to describe the process of establishing the extent to which a given model is an accurate representation of the corresponding real-world system. In practice, the emphasis should always be on “invalidation” since a model can never be proved to be, in any general sense, “correct”. This chapter provides an introduction to methods to be used for this process and includes quantitative methods, such as those based on system identification and parameter-estimation principles, barrier certificate methods, techniques based on concepts of model distortion and methods based on parameter sensitivity analysis. Methods of face validation, involving a more subjective approach and the opinion of experts, familiar with the real system, are also considered. That type of validation process is illustrated using an example of the development and testing of a complex simulation of a hydro-turbine generating system. Further sections of this chapter are devoted to approaches based on comparison with other models, the choice of data sets for model testing and validation, the validation of sub-models and generic models, issues relating to the validation of distributed parameter models and the validation of discrete-event and hybrid models. Before the final discussion section there is also a short review of issues arising in the acceptance or upgrading of models.

Keywords

Real System Parameter Sensitivity Analysis Gaussian Process Model Distribute Parameter Model Parameter Estimation Process 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Oberkampf WL (2007) Predictive capabilities in computational science and engineering. Presented at OASCR Applied Mathematics PI Meeting, Lawrence Livermore National Laboratory, 22–24 May 2007. http://science.energy.gov/~/media/ascr/pdf/workshops-conferences/mathtalks/Oberkampf.pdf. Accessed 10 June 2015
  2. 2.
    Anderson J, Papachristodoulou A (2009) On validation and invalidation of biological models. BMC Bioinforma 10:132. doi: 10.1186/1471-2105-10-132, http://www.biomedcentral.com/1471-2105/10/132. Accessed 10th June 2015CrossRefGoogle Scholar
  3. 3.
    Gore R, Diallo S (2013) The need for usable formal methods in verification and validation. In: Pasupathy R, Kim S-H, Tolk A et al (eds) Proceedings of the 2013 winter simulation conference. IEEE, Washington, DC, pp 1257–1268. doi: 10.1109/WSC.2013.6721513 Google Scholar
  4. 4.
    Heitmeyer CL (2007) Formal methods for specifying, validating and verifying requirements. J Univ Comput Sci 13(5):607–618Google Scholar
  5. 5.
    Gustavsson I (1972) Comparison of different methods for identification of industrial processes. Automatica 8(2):127–142CrossRefzbMATHGoogle Scholar
  6. 6.
    Prajna S (2003) Barrier certificates for nonlinear model validation. In: Proceedings 42nd IEEE conference on decision and control 9–12 Dec 2003, vol 3. IEEE, Washington, DC. pp 2884–2889, doi: 10.1109/CDC.2003.1273063
  7. 7.
    Tischler MB, Remple RK (2012) Aircraft and rotorcraft system identification, 2nd edn. AIAA, Reston, VAGoogle Scholar
  8. 8.
    Knudsen M (2006) Experimental modelling of dynamic systems: an educational approach. IEEE Trans Educ 49(1):29–38MathSciNetCrossRefGoogle Scholar
  9. 9.
    Gong M, Murray-Smith DJ (1993) Model reduction by an extended complex curve-fitting approach. Trans Inst Meas Control 15(4):188–198CrossRefGoogle Scholar
  10. 10.
    Balci O, Sargent R (1984) Validation of simulation models via simultaneous confidence intervals. Am J Math Manag Sci 4:375–406zbMATHGoogle Scholar
  11. 11.
    McFarland J, Mahadevan S (2008) Multivariate significance tests and model calibration under uncertainty. Comput Methods Appl Mech Eng 197(29–32):2407–2479Google Scholar
  12. 12.
    Huynh DPB, Knezevic DJ, Patera AT (2012) Certified reduced basis model characterization: a frequentistic uncertainty framework. Comput Methods Appl Mech Eng 201:13–24MathSciNetCrossRefGoogle Scholar
  13. 13.
    Rosenberg JR, Murray-Smith DJ, Rigas A (1982) An introduction to the application of system identification techniques to elements of the neuromuscular system. Trans Inst Meas Control 4(4):187–201CrossRefGoogle Scholar
  14. 14.
    Butterfield MH, Thomas PJ (1986) Methods of quantitative validation for dynamic system models-part 1: theory. Trans Inst Meas Control 8:182–200CrossRefGoogle Scholar
  15. 15.
    Cameron RG (1998) Model validation by the distortion method: linear state space systems control theory and applications. IEE Proc D 139(3):296–300CrossRefGoogle Scholar
  16. 16.
    Thomas PJ (1999) Simulation of industrial processes for control engineers. Butterworth-Heinemann, OxfordGoogle Scholar
  17. 17.
    Cameron RG, Marcos RL, De Prada C (1998) Model validation of discrete transfer functions using the distortion method. Math Comput Model Dyn Syst 4(1):58–72CrossRefzbMATHGoogle Scholar
  18. 18.
    Gray GJ, Voon LK, Murray-Smith (1997) Application of the distortion method for model validation. In: Troch I, Breitenecker F (eds) Proceedings 2nd MATHMOS VIENNA IMACS symposium on mathematical modelling February 1997. Argesim, Vienna, pp 1033–1038Google Scholar
  19. 19.
    Butterfield MH, Thomas PJ (1987) Quantitative validation of dynamic models for use in fast reactor safety assessments. In: Proceedings international conference on science and technology of fast reactor safety, Guernsey (UK), 12–16 May 1986. British Nuclear Energy Society, London, pp 145–152Google Scholar
  20. 20.
    Butterfield MH, Thomas PJ (1986) Methods of quantitative validation for dynamic system models-part 2: applications. Trans Inst Meas Control 8:201–219CrossRefGoogle Scholar
  21. 21.
    Butterfield MH (1990) A method of quantitative validation based on model distortion. Trans Inst Meas Control 12:167–173CrossRefGoogle Scholar
  22. 22.
    Kleijnen JPC (1995) Verification and validation of simulation models. Eur J Oper Res 82:145–162CrossRefzbMATHGoogle Scholar
  23. 23.
    Chattee E, Saam NJ, Möhring M (2000) Sensitivity analysis in the social sciences: problems and prospects. In: Suleiman et al (eds) Tools and techniques for social science simulation, chapter 3. Physica Verlag, HeidelbergGoogle Scholar
  24. 24.
    Bryce GW, Foord TR, Murray-Smith DJ, Agnew P (1976) Hybrid simulation of water turbine governors. In: Crosbie RE, Hay JL (eds) Simulations councils proceedings series 6(1), Simulation Councils, La Jolla, pp 35–44Google Scholar
  25. 25.
    Bryce GW, Agnew PW, Foord TR et al (1977) On-site investigation of electrohydraulic governors for water turbines. Proc IEE 124(2):147–153Google Scholar
  26. 26.
    The Mitre Corporation (2014) Verification and validation of simulation models. In: Mitre systems engineering guide, pp 461–469. www.mitre.org/publications/technical-papers/the-mitre-systems-engineering-guide. Accessed 10 June 2015
  27. 27.
    Hemez FM (2004) The myth of science-based predictive modelling. In: Proceedings foundations’04 workshop for verification, validation and accreditation (VV&A) in the 21st century, Arizona State University, Tempe, Arizona, 13–15 October 2004. Report LA-UR-04-6829, Los Alamos National Laboratory, USAGoogle Scholar
  28. 28.
    Kocijan J, Girard A, Banko B et al (2005) Dynamic system identification with dynamic processes. Math Comput Model Dyn Syst 11(4):411–424MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Tischler MB (1996) System identification for aircraft flight control development and validation. In: Tischler MB (ed) Advances in aircraft flight control. Taylor and Francis, London, pp 35–69Google Scholar
  30. 30.
    Tischler MB, Remple RK (2006) Aircraft and rotorcraft system identification. AIAA, RestonGoogle Scholar
  31. 31.
    Smith MI, Murray-Smith DJ, Hickman D (2007) Verification and validation issues in a generic model of electro-optic sensor systems. Def Model J Simul 4(1):17–17Google Scholar
  32. 32.
    Smith MI, Murray-Smith DJ, Hickman D (2007) Mathematical and computer modeling of electro-optic systems using a generic modeling approach. J Def Model Simul 4(1):3–16Google Scholar
  33. 33.
    Walmsley CW, McCurry MR, Clausen PD et al (2013) Beware the black box: investigating the sensitivity of FEA simulation to modelling factors in comparative biomechanics. PeerJ 1:e204, http://dx.doi.org/ 10.7717/peerj.204}. Accessed 10 June 2015CrossRefGoogle Scholar
  34. 34.
    Rizzi A, Vos J (1998) Towards establishing credibility in computational fluid dynamics simulations. AIAA J 36(5):668–675CrossRefGoogle Scholar
  35. 35.
    Thompson, DE (2005) Verification, validation and solution quality in computational physics: CFD methods applied to ice sheets, NASA/TM-2005-213453, NASA Technical Reports Server, 37 ppGoogle Scholar
  36. 36.
    Padfield GP, Du Val RW (1991) Application areas for rotorcraft system identification: simulation model validation. In: AGARD Lecture Series 178, Rotorcraft System Identification, 12.1-12.30, AGARD, Neuilly-sur-Seine, FranceGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • David J. Murray-Smith
    • 1
  1. 1.School of EngineeringUniversity of GlasgowGlasgowUK

Personalised recommendations