Abstract
A basis for the taxonomy of concepts related with quality assurance in modelling and simulation is provided. It has three parts:
-
1)
Basic elements of modelling and simulation which could and should be assessed
-
2)
Criteria which can be used for assessment, and
-
3)
Types of assessments.
The comprehensive categorized list of related terms given can be the starting point
-
1)
for a systematic study of the possibilities
-
2)
for their systematic refinements
-
3)
for the development of systematic studies of quality assurance in modelling and simulation, and
-
4)
for compiling a glossary of terms of quality assurance in modelling and simulation.
This study has been sponsored by the Natural Sciences and Engineering Research Council of Canada (NSERC) by the Operating Grant A8117.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References and Bibliography
Allen, B., Clema, J.K. (1976). Independent Verification/Validation Support Software. In: Proceedings of the Aerospace and Electronics Conference, Dayton, Ohio, pp. 276–281.
Annino, J.S., Russell, E.C. (1979). The ten Most Frequent Causes of Simulation Analysis Failure–And How to Avoid Them. Simulation Today, Vol. 32, No. 60, pp. 137–140.
ANSI. Quality Assurance Terms and Definitions. Publication No. N45.2.10–1973. American National Standards Committee N45, American Society of Mechanical Engineers, New York.
ANSI/ASQC. Quality Systems Terminology. Publication No. A3–1078. American Society for Quality Control, Milwaukee, Wisconsin.
Bahr, D (1975). A Note on Figures of Merit. ACM Performance Evaluation Review, Vol. 4, No. 3, pp. 1–3.
Balci, O. (1981). Statistical Validation of Multivariate Response Simulation Models. Ph.D. Dissertation, Graduate School of Syracuse University, Syracuse, NY.
Balci, O., Sargent, R.G. 1980a. A Methodology for Cost-Risk Analysis in Validation of Simulation Models. Working Paper No. 80–006, Department of Industrial Engineering and Operations Research, Syracuse University, Syracuse, NY.
Balci, O., Sargent, R.G. 1980. Bibliography on Validation of Simulation Models. Newsletter, TIMS College on Simulation and Gaming, Vol. 4, No. 4, Spring 1980, pp. 11–15 (125 entries).
Balci, O. Sargent, R.G. 1981a. A Methodology for Validating Multivariate Response Simulation Models by Using Simultaneous Confidence Intervals. Technical Report CS81019-R, Computer Science Department, Virginia Polytechnic Institute and State University, Blacksburg, VA.
Balci, O., Sargent 1981b. Validation of Multivariate Response Simulation Models By Using Hotelling’s Two-Sample t2 Test. Technical Report CS81016-R, Computer Science Department, Virginia Polytechnic Institute and State University, Blacksburg, VA.
Balci, O., Sargent, R.G. (1984). A Bibliography on the Credibility Assessment and Validation of Simulation and Mathematical Models. Simuletter, Vol. 15, No. 3.
Belady, L.A., Leavenworth, B. (1980). Program Modifiability. Report RC 8147, IBM T.J. Watson Research Center, Yorktown Heights, NY.
Berman, M.B. (1972). Notes on Validating/Verifying Computer Simulation Models. P-4891, Rand, Santa Monica, CA.
Britting, K.R. (1976). Backward Integration of System Dynamics Models. In: World Modeling: A Dialogue, Churchman, C.W., Mason, R.O. (Eds.), North Holland, Amsterdam, p. 141–149.
Cellier, F.E. (1979). Combined Continuous/Discrete System Simulation by Use of Digital Computers: Techniques and Tools. Dissertation, Doctor of Technical Sciences, ETH, Zurich, Switzerland.
Cellier, F.E., Moebius, P.J. (1979). Towards Robust General Purpose Simulation Software. ACM SIGNUM.
Chandy, K.M., Misra, J. (1980). On the Correctness of Simulation. (Pèrsonal Communication).
Cutler, M.M. (1979) Proving Properties of Simulation Programs for System Verification and Validation. In: Proceedings of the 1979 Summer Computer Simulation Conference, Toronto, Ont., Canada, pp. 610–616.
Cutler, M.M. (1980). A Formal Program Model for Discrete Event Simulation and Its Use in the Verification and Validation of System Models and Implementations. Ph.D. Dissertation, University of California, Los Angeles, CA
Deeter, C.R., Hoffman, A.A.J. (1978). An Annotated Bibliography of Mathematical Models Classified by Validation and Error Analysis Methods. Texas, Christian University, Fort Worth, Texas, 112 p.
Dickhoven, S. (1979). Software Requirements for an Improvement of Transfer and Adaptibility of Models. US National Bureau of Standards Workshop: ‘Utility and Use of Large-Scale Mathematical Models,’ Gaithersburg, MD.
Drake, J.W. (1973). Why Models Fail in Transportation. In: Proceedings of 1973 Summer Computer Simulation Conference, pp. 1140–1144.
Elzas, M.S. 1978a. Whither Simulation? In: Proceedings of the 1978 UKSC Conference on Computer Simulation, Chester, England 4–6 April, pp. 494–501.
Elzas, M.S. 1978b. Results of a Simulation: Can They Be “Trusted”? In: Proceedings of the Seminar “Modelling in Business”, Amsterdam, Holland, 12–13 January, pp. 170–183.
Elzas, M.S. (1979). What is Needed for Robust Simulation? In: Methodology in Systems Modelling and Simulation, B.P. Zeigler, M.S. Elzas, G.J. Klir, T.I. Oren (Eds.). North-Holland, Amsterdam, pp. 57–91.
Ferrell, W.R., McGoey, P.J. (1980). A Model of Calibration for Subjective Probabilities. Organizational Behavior and Human Performance, Vol. 26, pp. 32–53.
Gass, S.I. 1977a. Evaluation of Complex Models. Computers and Operations Research, Vol. 4, No. 1, pp. 27–35.
Gass, S.I. 1977b. A Procedure for the Evaluation of Complex Models. In: Proceedings of the First International Conference in Mathematical Models. University of Missouri.
Gass, S.I. 1979a. Validation and Assessment Issues of Energy Models. Prepared for NATO Advanced Research Institute on “The Application of Systems Science to Energy Policy Planning,” November 12–16, Upton, NY.
Gass, S.I. 1979b. Computer Science & Technology: Computer Model Documentation: A Review and an Approach. NBS Special Publication No. 500–39, National Bureau of Standards, Washington, DC.
Gass, S.I. (Ed.) 1979c. Utility and Use of Large-Scale Mathematical Models. NBS Special Publication, No. 534, National Bureau of Standards, Washington, DC, 223 p.
Gass, S.I. (Ed.) 1979d. Bibliography: Model Evaluation and Assessment. In: Utility and Use of Large-Scale Mathematical Models-Proceedings of a Workshop held at the National Bureau of Standards, Gaithersburg, Maryland, April 28–29, 1977, pp. 215–217.
Gass, S.I. (Ed.) (1980). Validation and Assessment Issues of Energy Models. Proceedings of a Workshop Held at the National Bureau of Standards, Gaithersburg, Maryland, January 10–11. NBS Special Publication No. 569, National Bureau of Standards, Washington, DC, 570 p.
Gass, S.I. (Ed.) 1981a. Validation and Assessment of Energy Models. NBS Special Publication No. 616, National Bureau of Standards, Washington, DC, 255 p.
Gass, S.I. 1981b. Documentation for a Model: A Hierarchical Approach. Communications of the ACM, Vol. 24, No. 11, pp. 728–733.
Hailpern, B., Owicki, S. (1981). Modular Verification of Computer Communication Protocols. Report RC 8726, IBM T.J. Watson Research Center, Yorktown Heights, NY.
Henize, J. (1979). Evaluating Policy Models. In: Proceedings of the Summer Computer Simulation Conference. Toronto, Ontario, Canada, pp 550–556.
Herman, C. (1967). Validation Problems in Games and Simulations. Behavioral Science, Vol. 12, pp. 212–230.
Holling, C.S. (Ed.) (1978). Adaptive Environmental Assessment and Management. Wiley, New York, 377 pp.
House, P.W. (1974). Diogenes Revisited — The Search for a Valid Model. Simulation, Vol. 23, No. 4, pp. 117–125.
Howden, W.E. (1980). Applicability of Software Validation Techniques to Scientific Programs. ACM Transactions on Programming Languages and Systems, Vol. 2, No. 3, pp. 307–320.
Hunt, B.R. (1976). Large Scale System Theories: Some Paradigms from Software Systems Engineering. In: Cybernetics and Modelling and Simulation of Large Scale Systems, T.I. Ören (Ed.), International Association for Cybernetics, Namur, Belgium.
Kahne, S.E. (1976). Model Credibility for Large-Scale Systems. IEEE Transactions on Systems, Man, and Cybernetics, August 1976, pp. 586–590.
Karr, M, Loveman T, (1978). Incorporation of Units into Programming Languages. Communications of the ACM, Vol. 21, No. 5, pp. 385–391.
Kheir, H.A., Holmes, W.M. (1978). On Validating Simulation Models of Missile Systems. Simulation, Vol. 30, No. 4, pp. 117–128.
Kleijnen, J.P.C. (1981). Cross-Validation Using the t Statistic. Research Memorandum, Tilburg University, Tilburg, Netherlands.
Kobayashi, H. (1975). System Design and Performance Analysis Using Analytic Models. Report RA 75, IBM T.J. Watson Research Center, 92 pp.
Landry, M., Malouin, J-L., Oral, M. (1983). Model Validation in Operations Research. European J. of Operational research, Vol. 14, pp. 207–220.
Lee jr., (1973). Requiem for Large-Scale Models, J. of American Institute of Planners, Vol. 39, No. 3.
Lewandowski, A. (1982). Issues in Model Validation. RR-82–37, IIASA, Laxenburg, Austria. Reprinted from Angewandte Systemanalyse, Vol. 3, No. 1, 1982.
Marmor-Squires, A.B. (1977). Software Validation: An Annotated Bibliography. In: Current Trends in Programming Methodology. Vol. II. Program Validation, R.T. Yeh (Ed.). Prentice-Hall, Englewood Cliffs, NY, pp. 269–319.
McLeod, J. (1977). Terms and Definitions — Play it Again, Stew. Simulation Today, Vol. 29, No. 4, pp. vii-ix.
McLeod, J. (1982). Computer Modeling and Simulation: Principles of Good Practice. Simulation Series, Vol. 10, No. 2, The Society for Computer Simulation, La Jolla, CA.
Nance, R.E. (1977). The Feasibility of and Methodology for Developing Federal Documentation Standards for Simulation Models. Report, Dept. of Computer Science, VPI and State University Blacksburg, VA.
Newton, O.L., Whetherbee, J.E. (1980). Guidelines for Documenting Simulation Models. In: Simulation with Discrete Models: A State-of-the-Art View, T.I. Ören, C.M. Shub, P.F. Roth (Eds.), IEEE, New York, pp. 243–258.
Oren, T.I. (1979). Concepts for Advanced Computer-Assisted Modelling. In: Methodology in Systems Modelling and Simulation, B.P. Zeigler et al. (Eds.). North-Holland, pp. 29–55. (Opening Paper in the Symposium on Modelling and Simulation Methodology, Rehovot, Israel, 1978 August 13–18).
Ören, T.I. 1981a. Foundations for an Information Technology. In: Proceedings of 1981 Winter Simulation Conference, T.I. Ören, C.M. Delfosse, C.M. Shub (Etls.). IEEE, New York, pp. 201–208.
Ören, T.I. 1981b. Concepts and Criteria to Assess Acceptability of Simulation Studies: A Frame of Reference. CACM, Vol. 24, No. 4, pp. 180–189.
Ören, T.I. 1982a. Computer-Aided Modelling Systems. (Keynote paper — SIMULATION ‘80, Interlaken, Switzerland, 1980 June 25–27), In: Progress in Modelling and Simulation, F.E. Cellier (Ed.). Academic Press, London, England, pp. 189–203.
Ören, T.I. 1982b. Synergies of Simulation Model Management, Software Engineering, Artificial Intelligence, and General System Theories. In: General Systems, Vol. 26, R.K. Ragade (Ed.). pp. 203–209. Also published in: Proceedings of the 26th Annual Conference of the Society for General Systems Research, Vol. 1, pp. 197–204.
Ören, T.I. (1983). Quality Assurance of System Design and Model Management for Complex Problems. In: Adequate Modeling of Systems, H. Wedde (Ed.). Springer-Verlag, Heidelberg, W. Germany, pp. 205–219.
Ören T.I. 1984a. Directions for the Future of Modelling and Simulation: A Normative View. International Journal of Systems Analysis, Modelling, Simulation, Vol. 1, No. 4, pp. 293–306.
Ören, T.I. (1984b — In Press). Model-Based Information Technology: Computer and System Theoretic Foundation. Behavioral Science, July 1984.
Ören, T.I. (1984c — In Press). Simulation Methodology: A Top-Down View. In: Encyclopedia of Systems and Control, M. Singh (Ed.). Pergamon Press, Oxford, England.
Ören, T.I. (1984d — In Press). Simulation: A Taxonomy. In: Encyclopedia of Systems and Control, M. Singh (Ed.). Pergamon Press, Oxford, England.
Ören, T.I. (1984e — In Press). Simulation Models: A Taxonomy. In: Encyclopedia of Systems and Control, M. Singh (Ed.). Pergamon Press, Oxford, England.
Ören, T.I. (I984, — In Press). Model Behavior: A Taxonomy of Types and Generation and Processing Techniques. In: Encyclopedia of Systems and Control, M. Singh (EH.). Pergamon Press, Oxford, England.
Pugh, R.E. (1977). Evaluation of Policy Simulation Models. Information Resources Press, Washington, DC.
Roth, P.F., Gass, S.I., Lemoine, A.J. (1978). Some Considerations for Improving Federal Modelling. In: Proceedings of 1978 Winter Simulation Conference, pp. 213–217.
Sargent, R.G. (1978). Validation of Discrete-Event Simulation Models. In: Proceedings of NSF Workshop on Validation of Mathematical Models in Energy Related Research and Development, Forth Worth, TX.
Sargent, R.G. (1982). Verification and Validation of Simulation Models. In: Progress in Modelling and Simulation, F.E. Cellier (Ed.). Academic Press London, England.
Schellenberger, R.E. (1974). Criteria for Assessing Model Validity for Managerial Purposes. Decision Sciences, Vol. 5, No. 5.
Schlesinger, S. et al. (1979). Terminology for Model Credibility. Simulation, Vol. 32, No. 3, pp. 103–104.
Schruben, L.W. (1980). Establishing the Credibility of Simulations. Simulation, Vol. 34, No. 3, pp. 101–105.
Theorey, T.J. (1975). Validation Criteria for Computer System Simulation. Simuletter, Vol. 6, No. 4, pp. 9–20.
Thissen, W. 1978a. Investigations into the Club of Rome’s Wold 3 Model — Lessons for Understanding Complicated Models. Ph.D. Thesis, Dept. of Technical Physics, Eindhoven University of Technology, The Netherlands.
Thissen, W. 1978b. Investigations into the World 3 Model: Lessons for Understanding Complicated Models. IEEE Trans, on Systems, Man, and Cybernetyics, SMC-8, No. 3, pp. 183–193.
Van Horn, R. (1969). Validation. In: The Decision of Computer Experiments, T.H. Naylor (Ed.). Duke University Press, Durham, NC.
Van Horn, R. (1971). Validation of Simulation Results. Management Science, Vol. 17, No. 5.
Watt, K.F. (1977). Why Won’t Anyone Believe Us? Simulation, Vol. 28, No. 1, pp. 1–3.
Wierzbicki, A. (1977). Models and Sensitivity of Control Systems. (Tn Polish). Wydawn-ictwa Naukowo-Techniczne, Warsaw, Poland
Zeigler, B.P. (1976). Theory of Modelling and Simulation. Wiley-Interscience, New York.
Zucker, P.A. (1983). Some Pitfalls in the Validation of Multidimensional Models. In: Proceedings of Summer Computer Simulation Conference, Vancouver, BC, Canada, pp. 187–189.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1984 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ören, T.I. (1984). Quality Assurance in Modelling and Simulation: A Taxonomy. In: Ören, T.I., Zeigler, B.P., Elzas, M.S. (eds) Simulation and Model-Based Methodologies: An Integrative View. NATO ASI Series, vol 10. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-82144-8_17
Download citation
DOI: https://doi.org/10.1007/978-3-642-82144-8_17
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-82146-2
Online ISBN: 978-3-642-82144-8
eBook Packages: Springer Book Archive