Tests of Model Behavior that can Detect Structural Flaws: Demonstration with Simulation Experiments

Abstract

Validation of System Dynamics (SD) models involves two general types of tests: Tests of model structure and tests of model behavior. Yet, since SD models are causal models, the essence of model validity lies always in structural validity: “Right behavior for the right reasons”. (The nature of SD model validity has been discussed in various SD literature. For example, Forrester 1961 (Chapter 13); Forrester 1968; Forrester and Senge 1980; Bell and Senge 1980; Richardson and Pugh 1981 (Chapter 5 and 6) and Barlas 1985). Standard behavior tests, which compare the model-generated behavior to the observed reference behavior are generally “weak” in SD context, since they can not separate spurious behavior accuracy (“right behavior for the wrong reasons”) from true behavior validity. Such tests provide no structural information. Structure tests, on the other hand, are “strong” tests, since they evaluate the model structure directly. But their their major drawback is that they are informal, qualitative, hence difficult to communicate. (See Forrester and Senge 1980, and Richardson and Pugh 1981 (chapter 5) for some specific behavior and structure tests). Thus, it seems like a “third type of test” would be most appropriate for SD validation purposes: Quantitative/formal behavior tests that can provide some structural information (“structurally-oriented” behavior tests). Interestingly, such tests already exist in the SD validation “repertoire”. (For example, “Extreme Condition” simulations, “Behavior Sensitivity” testing, in Forrester and Senge 1980 and Richardson and Pugh 1981). But these tests are usually listed along with all other behavior tests, which undermines the major difference that exists between the former and the latter.