Abstract
Described is a program of research that uses rigorous methods to evaluate models of user cognition and action based on the construction-integration architecture of comprehension (Doane and Sohn, 2000; Kintsch, 1988; 1998). The models interrelate user environmental information, background knowledge, and current goals, and then spread activation throughout the interrelated information to simulate UNIX user command productions, aviation pilot eye fixations and control movements during flight, and army personnel intelligence planning. Models of individuals in the complex interactive environments are tested for descriptive as well as predictive validity. Comparisons of model and human empirical data have resulted in a high degree of agreement, validating the ability of the comprehension-based architecture to support models that can predict user performance. Evaluation methods are detailed and the importance of evaluative rigor is discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Anderson, D. A.: 1989, Artificial Intelligence and Intelligent Systems. New York, NY: Wiley and Sons.
Anderson, J. R.: 1993, Rules of the Mind. Hillsdale, NJ: Erlbaum.
Anderson, J. R. and Lebiere, C.: 1998, Atomic components of thought.Mahwah, NJ: Erlbaum.
Broadbent, D. E.: 1993, Comparison with human experiments. In: D. E. Broadbent (ed.), The Ssimulation of Human Intelligence. Cambridge, MA: Blackwell, pp. 198–217.
Dell, G. S., Schwartz, M. F., Martin, N., Saffran, E. M. and Gagnon, D. A.: 2000, The role of computational models in neuropsychological investigations of language: Reply to Ruml and Caramazza (2000). Psychological Review 107, 635–645.
Doane, S. M., Mannes, S. M., Kintsch, W. and Polson, P. G.: 1992, Modeling User Command Production: A Comprehension-Based Approach. User Modeling and User Adapted Interaction 2, 249–285.
Doane, S. M., McNamara, D. S., Kintsch, W., Polson, P. G. and Clawson, D.: 1992, Prompt comprehension in UNIX command production. Memory and Cognition 20(4), 327–343.
Doane, S. M., Pellegrino, J. W., and Klatzky, R. L.: 1990, Expertise in a computer operating system: Conceptualization and performance. Human–Computer Interaction 5, 267–304.
Doane, S. M. and Sohn, Y.W.: 2000, ADAPT: A predictive cognitive model of user visual attention and action planning. User Modeling and User Adapted Interaction 10, 1–45.
Doane, S. M., Sohn, Y. W., Adams, D. and McNamara, D. S.: 1994, Learning from instruction: A comprehension-based approach. Proceedings of the 16th Annual Conference of the Cognitive Science Society, Atlanta, GA: Erlbaum, pp. 254–259.
Doane, S. M., Sohn, Y. W. and Barnes, M. J.: 2000, Modeling the intuitive commander. ARL Technical Report. U.S. Army Research Laboratory. Ft. Huachuca, AZ.
Doane, S. M., Sohn, Y. W., McNamara, D. S. and Adams, D.: 2000, Comprehension-based skill acquisition. Cognitive Science 24, 1–52.
Dreyfus, H. L.: 1992, What computer still can't do: A critique of artificial reason. Cambridge, MA, MIT Press.
Dreyfus, H. L.: 1996, Response to my critics. Artificial Intelligence 80, 171–191.
Durso, F. T. and Gronlund, S. D.: 1999, Situation awareness. In: F. T. Durso, R. Nickerson, R. Schvaneveldt, S. Dumais, S. Lindsay and M. Chi (eds.), The Handbook of Applied Cognition, New York: Wiley, pp. 283–314.
Holyoak, K. J.: 1991, Symbolic connectionism: toward third-generation theories of expertise. In: K. A. Ericsson and J. Smith (eds.), Toward a General Theory of Expertise. Cambridge University Press, pp. 301–336.
Holyoak, K. J. and Thagard, P.: 1989, Analogical mapping by constraint satisfaction. Cognitive Science 13, 295–355.
Howson, C. and Urbach, P.: 1993, Scientific reasoning: The Bayesian approach. La Salle, IL: Open Court Press.
Hurvich, C. M.: 1997, Mean square over degrees of freedom: New perspectives on a model selection treasure. In D. R. Brillinger, L. T. Fernholz and S. Morgenthaler (eds.), The practice of data analysis: Essays in honor of John W. Tukey. Princeton, NJ: Princeton University Press, pp. 203–215.
Kaplan, C. A.: 1987, Computer Simulation: Separating Fact from Fiction. Pittsburgh: Carnegie-Mellon University.
Kintsch, W.: 1998, Comprehension: A Paradigm for Cognition. Cambridge, MA: Cambridge University Press.
Kintsch, W.: 1988, The use of knowledge in discourse processing: A construction-integration model. Psychological Review 95, 163–182.
Kintsch, W.: 1994, The psychology of discourse processing. In: M. A. Gernsbacher (ed.), Handbook of Psycholinguistics. San Diego: Academic Press, pp. 721–739.
Lovett, M. C. and Anderson, J. R.: 1996, History of success and current context in problem solving. Cognitive Psychology 31, 168–217.
Mannes, S. M. and Doane, S. M.: 1991, A hybrid model of script generation: Or getting the best of both worlds. Connection Science 3(1), 61–87.
Mannes, S. M. and Kintsch, W.: 1991, Routine computing tasks: Planning as understanding. Cognitive Science 15, 305–342.
Massaro, D. W. (1988). Some criticisms of connectionist models of human performance. Journal of Memory and Language 27, 213–234.
Meehl, P. E.: 1997, The problem is epistemology, not statistics: Replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. In: L. L. Harlow, S. A. Mulaik and J. H. Steiger (eds.), What if there were no significance tests? Mahwah, NJ: Erlbaum, pp. 393–425.
Miller, J. R. and Kintsch, W.: 1980, Readability and recall of short prose passages: A theoretical analysis. Journal of Experimental Psychology: Human Learning and Memory, 6, 335–354.
Newell, A.: 1990, Unified theories of cognition (The 1987 William James Lectures). Cambridge, MA: Harvard University Press.
Recker, M. M. and Pirolli, P.: 1995, Modeling individual differences in students' learning strategies. The Journal of the Learning Sciences 4, 1–38.
Roberts, S. and Pashler, H.: 2000, How persuasive is a good fit? A comment on theory testing. Psychological Review 107, 358–367.
Rosenbloom, P. S., Laird, J. E. and Newell, A.: 1991, Toward the knowledge level in SOAR: The role of architecture in the use of knowledge. In: K. VanLehn (ed.), Architectures for intelligence. Hillsdale, NJ: Erlbaum, pp. 75–112.
Simon, H. A. and Kaplan, C. A.: 1991, Foundations of cognitive science. In: M. I. Posner (ed.), Foundations of cognitive science. Cambridge, MA: MIT Press, pp. 1–47.
Sohn, Y. W. and Doane, S. M.: 2001, The roles of expertise, working memory capacity, and long-term memory retrieval in complex cognitive performance. Manuscript submitted for publication.
Thagard, P.: 1989, Explanatory coherence. Brain and Behavioral Sciences 12, 435–467.
van Dijk, T. A. and Kintsch, W.: 1983, Strategies of discourse comprehension. New York: Academic Press.
VanLehn, K.: 1988, Student modeling. In: M. C. Polson and J. J. Richardson (eds.), Foundations of intelligent tutoring systems. Hillsdale, NJ: Erlbaum, pp. 55–76.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Sohn, Y.W., Doane, S.M. Evaluating Comprehension-Based User Models: Predicting Individual User Planning and Action. User Modeling and User-Adapted Interaction 12, 171–205 (2002). https://doi.org/10.1023/A:1015072424236
Issue Date:
DOI: https://doi.org/10.1023/A:1015072424236