Skip to main content
SpringerLink
Log in
Book cover

Concepts and Methodologies for Modeling and Simulation pp 217–242Cite as

Verification, Validation, and Replication Methods for Agent-Based Modeling and Simulation: Lessons Learned the Hard Way!

  • Chapter

Part of the book series: Simulation Foundations, Methods and Applications ((SFMA))

Abstract

Reproducible modeling and simulation research has been identified as one of the Modeling and Simulation (M&S) Grand Challenge activities. Recently, uncertainty quantification has seen a renewed emphasis. While methods for verification and validation (V&V) have been widely developed for discrete-event simulations, newer simulation approaches such as the agent-based, agent-directed, and multi-agent simulation approaches introduce new V&V challenges. The active elements in these newer approaches have greater heterogeneity, e.g., every agent can be unique, with complex attributes and behaviors. Those behaviors can result in actions based on interaction with other agents, the environment, and even the outcome of simulated or artificial intelligence. The simulation spaces are often less constrained, e.g., rather than a network of servers and queues, the space can be continuous 2D Euclidian space with multiple associated geographic information systems (GIS) layers influencing the behavior of the actors. Over the last decade, a multitude of techniques has been used in agent-based modeling and simulation (ABMS) to perform V&V as well as replication and reproducibility (R&R) of the models. In this chapter, we present an overview of contributions in V&V, quality assurance (QA), and R&R of simulation studies, with special focus on ABMS. We also discuss the lessons learnt in V&V and replication from a series of simulation experiments using agent-based models (ABMs).

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  • An Y, Xiong G, Xiao T, Wang X (1989) IPSOS – an integrated simulation system for industrial processes. In: Proceedings of the 1989 Beijing simulation conference. Pergamon Press, London, p 500

    Google Scholar 

  • Arifin SMN, Davis GJ, Zhou Y (2010a) Verification & validation by docking: a case study of agent-based models of Anopheles gambiae. In: Summer computer simulation conference (SCSC), Ottawa, ON, Canada, July 2010

    Google Scholar 

  • Arifin SMN, Davis GJ, Kurtz SJ, Gentile JE, Zhou Y (2010b) Divide and conquer: a four-fold docking experience of agent-based models. In: Winter simulation conference (WSC), Baltimore, MD, USA, Dec 2010

    Google Scholar 

  • Arifin SMN, Kennedy RC, Lane KE, Fuentes A, Hollocher H, Madey GR (2010c) P-SAM: a post-simulation analysis module for agent-based models. In: Summer computer simulation conference (SCSC), Ottawa, ON, Canada, July 2010

    Google Scholar 

  • Arifin SMN, Davis GJ, Zhou Y (2012) A spatial agent-based model of malaria: model verification and effects of spatial heterogeneity. In: Zhang Y (ed) Theoretical and practical frameworks for agent-based systems. IGI Global, Hershey, PA, USA, pp 221–240

    Google Scholar 

  • Arifin SMN, Madey GR, Collins FH (2013) Examining the impact of larval source management and insecticide-treated nets: replicating recent models and exploring the combined impact using a spatial agent-based model of Anopheles gambiae and a landscape generator tool. Malar J 12:290

    Article  Google Scholar 

  • Arifin SMN, Zhou Y, Davis GJ, Gentile JE, Madey GR, Collins FH (2014) An agent-based model of the population dynamics of Anopheles gambiae. Malar J 13:424

    Article  Google Scholar 

  • Axtell R, Axelrod R, Epstien JM, Cohen MD (1996) Aligning simulation models: a case study and results. Comput Math Organ Theory 1:123–141

    Article  Google Scholar 

  • Balci O (1989) How to assess the acceptability and credibility of simulation results. In: Proceedings of the 21st conference on winter simulation, Washington, DC. ACM, New York, pp 62–71

    Google Scholar 

  • Balci O (1998a) Verification, validation and accreditation. In: Proceedings of the 1998 winter simulation conference, Washington, DC, 13–16 Dec 1998. IEEE, Piscataway, pp 41–48

    Google Scholar 

  • Balci O (1998b) Verification, validation, and testing. In: Banks J (ed) Handbook of simulation. Engineering & Management Press, pp 335–393

    Google Scholar 

  • Balci O (2003) Verification, validation, and certification of modeling and simulation applications. In: Proceedings of the 35th conference on winter simulation: driving innovation, New Orleans, pp 150–158

    Google Scholar 

  • Balci O (2004) Quality assessment, verification, and validation of modeling and simulation applications. In: Proceedings of the 2004 winter simulation conference, Washington, D.C., USA, vol 1. IEEE

    Google Scholar 

  • Biostatistics (2014) http://biostatistics.oxfordjournals.org/. Accessed 2 Dec 2014

  • Burton RM (2003) Computational laboratories for organization science: questions, validity and docking. Comput Math Organ Theory 9(2):91–108

    Article  Google Scholar 

  • Carley KM (2002) Computational organizational science and organizational engineering. Simul Model Pract Theory 10(5):253–269

    Article  MATH  Google Scholar 

  • Chitnis N, Schapira A, Smith T, Steketee R (2010) Comparing the effectiveness of malaria vector-control interventions through a mathematical model. Am J Trop Med Hyg 83:230–240

    Article  Google Scholar 

  • Eckhoff P (2011) A malaria transmission-directed model of mosquito life cycle and ecology. Malar J 10:303

    Article  MathSciNet  Google Scholar 

  • Edmonds B, Hales D (2003) Replication, replication and replication: some hard lessons from model alignment. J Artif Soc Soc Simul 6:4

    Google Scholar 

  • Elzas MS (1988) Expert simulation systems in practice. In: Vichnevetsky R et al (eds) Proceedings of the 12th World congress on scientific computation, vol 5, Paris, 18–22 July 1988, pp 47–50

    Google Scholar 

  • Fomel S, Hennenfent G (2009) Reproducible computational experiments using Scons. In: IEEE international conference on acoustics, speech, and signal processing, 4, Taipei, Taiwan, pp 1257–1260

    Google Scholar 

  • Garzia RF (ed) (1979) Simulation week report. Modelling (IEEE Computer Society Simulation Technical Committee Newsletter) Issue 5, May 1979, p 5

    Google Scholar 

  • Gu W, Novak RJ (2009a) Agent-based modelling of mosquito foraging behaviour for malaria control. Trans R Soc Trop Med Hyg 103:1105–1112

    Article  Google Scholar 

  • Gu W, Novak RJ (2009b) Predicting the impact of insecticide-treated bed nets on malaria transmission: the devil is in the detail. Malar J 8:256

    Article  Google Scholar 

  • Huang Y, Xiang X, Madey G, Cabaniss S (2005) Agent-based scientific simulation. Comput Sci Eng 7(1):22–29. doi:10.1109/MCSE.2005.7. http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1377072&isnumber=30051

  • Institute for Disease Modeling (IDM) (2014) http://idmod.org. Accessed 2 Dec 2014

  • Jasny BR, Chin G, Chong L, Vignieri S (2011) Again, and again, and again. Science 334:1225

    Article  Google Scholar 

  • Karplus WJ (1979) Review of the article by Ören TI, Zeigler BP (1979) Concepts for advanced simulation methodologies. Simulation 32(3):69–82

    Google Scholar 

  • Kennedy RC, Xiang X, Madey GR, Cosimano TF (2005) Verification and validation of scientific and economic models. In: Proc. Agent, Chicago, pp 177–192

    Google Scholar 

  • Kennedy RC, Xiang X, Cosimano TF, Arthurs LA, Maurice PA, Madey GR, Cabaniss SE (2006) Verification and validation of agent-based and equation-based simulations: a comparison. In: 2006 agent-directed simulation (ADS), Huntsville

    Google Scholar 

  • Ketcham MG (1986) Expert systems and user decisions in simulation studies. In: Cellier FE (ed) Languages for continuous system simulation. SCS, San Diego, pp 44–49

    Google Scholar 

  • Ketcham MG, Fowler JW, Phillips DT (1984) New directions for the design of advanced simulation systems. In: Sheppard S, Pooch U, Pegden D (eds) Proceedings of the 1984 winter simulation conference, Dallas. SCSI, San Diego, pp 563–568

    Google Scholar 

  • Kettenis DL (2001) Ethical issues in modeling and simulation. Guest Editor’s Introduction to the special issue. Simul-T Soc Mod Sim 17(4):162

    Google Scholar 

  • Lin M-J (1990) Automatic simulation model design from a situation theory based manufacturing system description. Ph.D. Dissertation, Texas A&M University, College Station, Texas

    Google Scholar 

  • Nance RE, Mezaache AL, Overstreet CM (1981) Simulation model management: resolving the technological gaps. In: Proceedings of the 1981 winter simulation conference, Atlanta, GA, USA, pp 173–179

    Google Scholar 

  • National Research Council (2012) Assessing the reliability of complex models: mathematical and statistical foundations of verification, validation, and uncertainty quantification. The National Academies Press, Washington

    Google Scholar 

  • North MJ, Macal CM (2002) The Beer Dock: three and a half implementations of the beer distribution game. In: Sixth annual swarm users meeting (SwarmFest), University of Washington, Seattle, 29–31 Mar 2002

    Google Scholar 

  • Olaru D, Purchase S, Denize S (2009) Using docking/replication to verify and validate computational models. In: 18th World IMACS/MODSIM Congress 2009, Cairns, Australia

    Google Scholar 

  • OpenMalaria (2014) A simulator of malaria epidemiology and control. https://code.google.com/p/openmalaria. Accessed 2 Dec 2014

  • Ören TI (1981) Concepts and criteria to assess acceptability of simulation studies: a frame of reference. Commun ACM 24(4):180–189

    Article  Google Scholar 

  • Ören TI (1983) Quality assurance of system design and model management for complex problems. In: Adequate modeling of systems. Springer, Berlin/Heidelberg, pp 205–219

    Chapter  Google Scholar 

  • Ören TI (1984) Quality assurance in modelling and simulation: a taxonomy. Simulation and model-based methodologies: an integrative view. Springer, Berlin/Heidelberg, pp 477–517

    Book  Google Scholar 

  • Ören TI (2000) Responsibility, ethics, and simulation. Transactions 17(4)

    Google Scholar 

  • Ören TI (1980) Assessing acceptability of simulation studies. In: Proceedings of 1980 winter simulation conference, vol 2, Orlando, FL, USA

    Google Scholar 

  • Ören TI, Yilmaz L (2009) Failure avoidance in agent-directed simulation: beyond conventional V&V and QA. In: Agent-directed simulation and systems engineering, Systems engineering series. Wiley, Berlin

    Google Scholar 

  • Ören TI, Zeigler BP (1979) Concepts for advanced simulation methodologies. Simulation 32(3):69–82

    Article  Google Scholar 

  • Ören TI, Elzas MS, Sheng G (1985) Model reliability and software quality assurance in simulation of nuclear fuel waste management systems. ACM SIGSIM Simulation Digest 16(4):4–19

    Article  Google Scholar 

  • Pavón J, Arroyo M, Hassan H, Sansores C (2008) Agent-based modelling and simulation for the analysis of social patterns. Pattern Recogn Lett 29(8):1039–1048

    Article  Google Scholar 

  • Pegden CD, Shannon RE, Sadowski RP (1995) Introduction to simulation using SIMAN. McGraw-Hill, New York

    Google Scholar 

  • Peng RD (2011) Reproducible research in computational science. Science 334:1226–1227

    Article  Google Scholar 

  • PLoS Computational Biology Guidelines for Authors (2014) http://www.ploscompbiol.org/static/guidelines.action#software. Accessed 2 Dec 2014

  • Rand W, Wilensky U (2006) Verification and validation through replication: a case study using Axelrod and Hammond’s ethnocentrism model. In: 14th annual conference of the North American Association for Computational Social and Organization Sciences (NAACSOS), Notre Dame, 22–23 June 2006

    Google Scholar 

  • Rouchier J, Cioffi-Revilla C, Polhill JG, Takadama K (2008) Progress in model-to-model analysis. J Artif Soc Soc Simul 11(2):8

    Google Scholar 

  • Santer BD, Wigley TML, Taylor KE (2011) The reproducibility of observational estimates of surface and atmospheric temperature change. Science 334:1232–1233

    Article  Google Scholar 

  • Sargent RG (2001) Verification and validation: some approaches and paradigms for verifying and validating simulation models. In: Proceedings of the 33rd conference on winter simulation. IEEE Computer Society, Washington, DC, pp 106–114

    Google Scholar 

  • Sargent RG (2004) Validation and verification of simulation models. In: Proceedings of the 2004 winter simulation conference, Washington, D.C., USA, vol 1. IEEE

    Google Scholar 

  • Shannon RE (1986) Intelligent simulation environments. In: PA Luker, Adelsberger HH (eds) Proceedings of the conference on intelligent simulation environments, Society for Computer Simulation, San Diego, 23–25 Jan 1986

    Google Scholar 

  • Smit W (1999) A question of ethics. In: The book edited to honor Prof. Ir. M.S. Elzas, University of Wageningen, Dept. of Informatics, Wageningen, pp 30–33

    Google Scholar 

  • Standridge CR, Pritsker AAB (1982) Using data base capabilities in simulation. In: Cellier FE (ed) Progress in modelling and simulation. Academic, London, pp 347–365

    Google Scholar 

  • Taylor SJE, Khan A, Morse KL, Tolk A, Yilmaz L, Zander J (2013) Grand challenges on the theory of modeling and simulation. In: Proceedings of the symposium on theory of modeling & simulation 2013, San Diego, California, USA, 34:1–8

    Google Scholar 

  • Thiele JC, Kurth W, Grimm V (2014) Facilitating parameter estimation and sensitivity analysis of agent-based models: a cookbook using NetLogo and ‘R’. J Artif Soc Soc Simul 17(3)

    Google Scholar 

  • Troncale LR (1985) The future of general systems research: obstacles, potentials, case studies. Syst Res 2(1):43–84

    Article  Google Scholar 

  • Vector-Borne Disease Network (VecNet) (2014) https://www.vecnet.org. Accessed 2 Dec 2014

  • Walker WE, Harremoës P, Rotmans J, van der Sluijs JP, van Asselt MBA, Janssen P, Krayer von Krauss MP (2003) Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support. Integr Assess 4(1):5–17

    Article  Google Scholar 

  • Wang Z-Z, Li B-H (1991) Computer simulation: the past, present and future. In: Proc. of the 1991 summer computer simulation conference, 22–24 July 1991, Baltimore. The Society for Computer Simulation International, San Diego, pp 1059–1065

    Google Scholar 

  • Wilensky U, Rand W (2007) Making models match: replicating an agent-based model. J Artif Soc Soc Simul 10(4):2

    Google Scholar 

  • Will O (2009) Resolving a replication that failed: news on the Macy & Sato Model. J Artif Soc Soc Simul 12:4

    Google Scholar 

  • Will O, Hegselmann R (2008) A replication that failed: on the computational model in ‘Michael W. Macy and Yoshimichi Sato: Trust, Cooperation and Market Formation in the U.S. and Japan. In: Proceedings of the National Academy of Sciences, May 2002’. J Artif Soc Soc Simulat 11(3):3

    Google Scholar 

  • Wittmann J (1992) Model and experiment – a new approach to definitions. In: Luker P (ed) Proceedings of the 1992 summer computer simulation conference, 27–30 July 1992, Reno. The Society for Computer Simulation International, San Diego, pp 115–119

    Google Scholar 

  • Xiang X, Kennedy R, Madey G (2005) Verification and validation of agent-based scientific simulation models. In: Agent-directed simulation conference, San Diego

    Google Scholar 

  • Xu J, Gao Y, Madey G (2003) A docking experiment: swarm and repast for social network modeling. In: Seventh annual Swarm researchers meeting (Swarm2003), Notre Dame, Indiana USA

    Google Scholar 

  • Yakob L, Yan G (2009) Modeling the effects of integrating larval habitat source reduction and insecticide treated nets for malaria control. PLoS One 4:e6921

    Article  Google Scholar 

  • Yilmaz L (2006) Validation and verification of social processes within agent-based computational organization models. Comput Math Organ Theory 12:283–312

    Article  MATH  Google Scholar 

  • Yilmaz L (2011) Reproducibility in modeling & simulation research. Simulation, Editorial 87(1):3–4

    Article  Google Scholar 

  • Yilmaz L (2012a) Reproducibility in M&S research: issues, strategies, and implications for model development environment. J Exp Theor Artif Intell 24(4):457–474

    Article  Google Scholar 

  • Yilmaz L (2012b) Scholarly communication of reproducible modeling and simulation research using e Portfolios. In: Proceedings of the 2012 international summer computer simulation conference, Genoa, 8–11 July 2012, pp 241–248

    Google Scholar 

  • Yilmaz L, Ören T (2013) Toward replicability-aware modeling and simulation: changing the conduct of M&S in the information age. In: Ontology, epistemology, and teleology for modeling and simulation. Springer, Berlin/Heidelberg, pp 207–226

    Chapter  Google Scholar 

  • Zeigler BP (1982) Subject plan for an encyclopedia: a taxonomy for modelling and simulation. ACM Simuletter 13:1–4, pp 55–62

    Google Scholar 

  • Zhong W, Kim Y (2010) Using model replication to improve the reliability of agent-based models. In: Chai S-K, Salerno JJ, Mabry PL (eds) Proceedings of the third international conference on social computing, behavioral modeling, and prediction (SBP’10). Springer, Berlin/Heidelberg, pp 118–127

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN, USA

    S. M. Niaz Arifin & Gregory R. Madey

Authors
  1. S. M. Niaz Arifin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gregory R. Madey
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. M. Niaz Arifin .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Software Engineering, Department of Industrial and Systems Engineering, Auburn University, Auburn, Alabama, USA

    Levent Yilmaz

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Arifin, S.M.N., Madey, G.R. (2015). Verification, Validation, and Replication Methods for Agent-Based Modeling and Simulation: Lessons Learned the Hard Way!. In: Yilmaz, L. (eds) Concepts and Methodologies for Modeling and Simulation. Simulation Foundations, Methods and Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-15096-3_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-15096-3_10

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-15095-6

  • Online ISBN: 978-3-319-15096-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation