Evaluating a Simulated Student Using Real Students Data for Training and Testing\(^{\thanks{The research presented in this paper is supported by National Science Foundation Award No. REC-0537198.}}\)

  • Noboru Matsuda
  • William W. Cohen
  • Jonathan Sewall
  • Gustavo Lacerda
  • Kenneth R. Koedinger
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4511)


SimStudent is a machine-learning agent that learns cognitive skills by demonstration. It was originally developed as a building block of the Cognitive Tutor Authoring Tools (CTAT), so that the authors do not have to build a cognitive model by hand, but instead simply demonstrate solutions for SimStudent to automatically generate a cognitive model. The SimStudent technology could then be used to model human students’ performance as well. To evaluate the applicability of SimStudent as a tool for modeling real students, we applied SimStudent to a genuine learning log gathered from classroom experiments with the Algebra I Cognitive Tutor. Such data can be seen as the human students’ “demonstrations” of how to solve problems. The results from an empirical study show that SimStudent can indeed model human students’ performance. After training on 20 problems solved by a group of human students, a cognitive model generated by SimStudent explained 82% of the problem-solving steps performed correctly by another group of human students.


Cognitive Skill Production Rule Operator Sequence Inductive Logic Programming Intelligent Tutor System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Greer, J.E., McCalla, G.: Student modelling: the key to individualized knowledge-based instruction, vol. x, p. 383. Springer-Verlag, Berlin, New York (1994)zbMATHGoogle Scholar
  2. 2.
    Anderson, J.R., et al.: Cognitive tutors: Lessons learned. Journal of the Learning Sciences 4(2), 167–207 (1995)CrossRefGoogle Scholar
  3. 3.
    Aleven, V., et al.: The Cognitive Tutor Authoring Tools (CTAT): Preliminary evaluation of efficiency gains. In: Ikeda, M., Ashley, K.D., Chan, T.W. (eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, pp. 61–70. Springer Verlag, Berlin (2006)Google Scholar
  4. 4.
    Matsuda, N., et al.: Applying Machine Learning to Cognitive Modeling for Cognitive Tutors, in Machine Learning Department Technical Report (CMU-ML-06-105), School of Computer Science, Carnegie Mellon University: Pittsburgh, PA (2006)Google Scholar
  5. 5.
    Muggleton, S., de Raedt, L.: Inductive Logic Programming: Theory and methods. Journal of Logic Programming 19-20(Supplement 1), 629–679 (1994)CrossRefGoogle Scholar
  6. 6.
    Lau, T.A., Weld, D.S.: Programming by demonstration: an inductive learning formulation. In: Proceedings of the 4th international conference on Intelligent user interfaces, pp. 145–152. ACM Press, New York (1998)Google Scholar
  7. 7.
    Johnson, W.L., et al.: Integrating pedagogical agents into virtual environments. Presence 7(6), 523–546 (1998)CrossRefGoogle Scholar
  8. 8.
    Baffes, P., Mooney, R.: Refinement-Based Student Modeling and Automated Bug Library Construction. Journal of Artificial Intelligence in Education 7(1), 75–116 (1996)Google Scholar
  9. 9.
    Merceron, A., Yacef, K.: A web-based tutoring tool with mining facilities to improve learning and teaching. In: Proceedings of the 11th International Conference on Artificial Intelligence in Education, Hoppe, U., Verdejo, F., Kay, J.(eds.), pp. 201–208 (2003)Google Scholar
  10. 10.
    Mertz, J.S.: Using A Simulated Student for Instructional Design. International Journal of Artificial Intelligence in Education 8, 116–141 (1997)Google Scholar
  11. 11.
    Koedinger, K.R., Corbett, A.: Cognitive Tutors: Technology Bringing Learning Sciences to the Classroom. In: Sawyer, R.K. (ed.) The Cambridge Handbook of the Learning Sciences, pp. 61–78. Cambridge University Press, New York, NY (2006)Google Scholar
  12. 12.
    Matsuda, N., Cohen, W.W., Koedinger, K.R.: Applying Programming by Demonstration in an Intelligent Authoring Tool for Cognitive Tutors. In: AAAI Workshop on Human Comprehensible Machine Learning (Technical Report WS-05-04), AAAI association: Menlo Park, CA, pp. 1–8 (2005)Google Scholar
  13. 13.
    Friedman-Hill, E.: Jess in Action: Java Rule-based Systems, Greenwich, CT: Manning (2003)Google Scholar
  14. 14.
    Quinlan, J.R.: Learning Logical Definitions from Relations. Machine Learning 5(3), 239–266 (1990)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Noboru Matsuda
    • 1
  • William W. Cohen
    • 2
  • Jonathan Sewall
    • 1
  • Gustavo Lacerda
    • 2
  • Kenneth R. Koedinger
    • 1
  1. 1.Human-Computer Interaction Institute 
  2. 2.Machine Learning Department, Carnegie Mellon University, 5000 Forbes Ave., Pittsburgh PA, 15217 

Personalised recommendations