An analysis of students’ gaming behaviors in an intelligent tutoring system: predictors and impacts

  • Kasia Muldner
  • Winslow Burleson
  • Brett Van de Sande
  • Kurt VanLehn
Original Paper

Abstract

Students who exploit properties of an instructional system to make progress while avoiding learning are said to be “gaming” the system. In order to investigate what causes gaming and how it impacts students, we analyzed log data from two Intelligent Tutoring Systems (ITS). The primary analyses focused on six college physics classes using the Andes ITS for homework and test preparation, starting with the research question: What is a better predictor of gaming, problem or student? To address this question, we developed a computational gaming detector for automatically labeling the Andes data, and applied several data mining techniques, including machine learning of Bayesian network parameters. Contrary to some prior findings, the analyses indicated that student was a better predictor of gaming than problem. This result was surprising, so we tested and confirmed it with log data from a second ITS (the Algebra Cognitive Tutor) and population (high school students). Given that student was more predictive of gaming than problem, subsequent analyses focused on how students gamed and in turn benefited (or not) from instructional features of the environment, as well as how gaming in general influenced problem solving and learning outcomes.

Keywords

Educational data mining Gaming Utility of hints Bayesian network parameter learning 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aleven, V.: Helping students to become better help seekers: towards supporting metacognition in a cognitive tutor. In: Paper Presented at German-USA Early Career Research Exchange Program: Research on Learning Technologies and Technology-Supported Education (2001)Google Scholar
  2. Aleven V., McLaren B., Roll I., Koedinger K.: Toward meta-cognitive tutoring: a model of help seeking with a cognitive tutor. Int. J. Artif. Intell. Educ. 16, 101–128 (2006)Google Scholar
  3. Arroyo I., Woolf  B.: Inferring learning and attitudes from a Bayesian network of log file data. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED’05), Amsterdam, Netherlands, pp. 33–40 (2005)Google Scholar
  4. Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A., Mahadevan, S., Woolf, B.: Repairing disengagement with non-invasive interventions. In: Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED’07), Los Angeles, United States, pp. 195–202 (2007)Google Scholar
  5. Baker, R.: Is gaming the system state-or-trait? educational data mining through the multi-contextual application of a validated behavioral model. In: Proceedings of the Workshop on Data Mining for User Modeling, Corfu, Greece, pp. 76–80 (2007)Google Scholar
  6. Baker, R., Corbett, A., Koedinger, K.: Detecting student misuse of intelligent tutoring systems. In: Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS’04), Maceio, Brazil, pp. 531–540 (2004a)Google Scholar
  7. Baker, R., Corbett, A., Koedinger, K., Wagner, A.: Off-task behavior in the cognitive tutor classroom: when students “game the system”. In: Proceedings of the ACM CHI 2004: Computer-Human Interaction (CHI’04), Vienna, Austria, pp. 383–390 (2004b)Google Scholar
  8. Baker, R.S., Roll, I., Corbett, A., Koedinger, K.: Do performance goals lead students to game the system. In: Proceedings of the 12th International Conference on Artificial Intelligence and Education (AIED2005), Amsterdam, Netherlands, pp. 57–64 (2005)Google Scholar
  9. Baker, R., Corbett, A., Koedinger, K., Evenson, E., Roll, I., Wagner, A., Naim, M., Raspat, J., Baker, D., Beck, J.: Adapting to when students game an intelligent tutoring system. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 392–401 (2006a)Google Scholar
  10. Baker, R.S.J.d., Corbett, A.T., Koedinger, K., Roll, I.: Generalizing detection of gaming the system across a tutoring curriculum. In: Proceedings of the 11th International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 402–411 (2006b)Google Scholar
  11. Baker R., Corbett A., Roll I., Koedinger K.: Developing a generalizable detector of when students game the system. User Model. User-Adap. Inter. 18(3), 287–314 (2008a)CrossRefGoogle Scholar
  12. Baker R., Walonoski J., Heffernan N., Roll I., Corbett A., Koedinger K.: Why students engage in “gaming the system”. Behavior in interactive learning environments. J. Interact. Learn. Res. 19(2), 185–224 (2008b)Google Scholar
  13. Baker, R., Corbett, A., Koedinger, K.: Educational software features that encourage and discourage “gaming the system”. In: Proceedings of the 14th international conference on artificial intelligence in education (AIED’09), Brighton, UK, pp. 475–482 (2009)Google Scholar
  14. Baker R.: Data mining for education. In: McGaw, B., Peterson, P., Baker, E., (eds) International Encyclopedia of Education, vol. 7, 3rd edn, pp. 112–118. Elsevier Oxford, UK (2010)CrossRefGoogle Scholar
  15. Beck J.E., Jia P., Mostow J.: Using automated questions to assess reading comprehension, vocabulary, and effects of tutorial interventions. Technol. Instr. Cogn. Learn. (TICL) 2, 97–134 (2004)Google Scholar
  16. Beck, J., Chang, K., Mostow, J., Corbett, A.: Does help? Introducing the Bayesian evaluation and assessment methodology. In: Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS’08), Montreal, Canada, pp. 383–394 (2008)Google Scholar
  17. Bunt, A., Conati, C., Muldner, K.: Scaffolding self-explanation to improve learning in exploratory learning environments. In: Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS’04), Maceio, Brazil, pp. 656–667 (2004)Google Scholar
  18. Carnegie Learning Inc.: Retrieved April 1, 2010, from http://www.carnegielearning.com/ (2010)
  19. Chang, K., Beck, J., Mostow, J., Corbett, A.: A Bayes net toolkit for student modeling in intelligent tutoring systems. In: Proceedings of the 11th International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 104–113 (2006)Google Scholar
  20. Chi, M.T.H.: How adaptive is an expert human tutor? In: Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS’10), Pittsburgh, United States, pp. 401–113 (2010)Google Scholar
  21. Chi, M., VanLehn, K.: The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In: Proceedings of the 29th annual conference of the cognitive science society, Nashville, Tennessee, pp. 167–172 (2007a)Google Scholar
  22. Chi, M., VanLehn, K.: Accelerated future learning via explicit instruction of a problem solving strategy. In: Proceedings of the 13th international conference on artificial intelligence in education (AIED’07), Los Angeles, United States, pp. 409–416 (2007b)Google Scholar
  23. Chi M.T.H., Bassok M., Lewis M., Reimann P., Glaser R.: Self-explanations: how students study and use examples in learning to solve problems. Cogn. Sci. 15, 145–182 (1989)Google Scholar
  24. Chi M.T.H., Siler S.A., Jeong H., Yamauchi T., Hausmann R.G.: Learning from human tutoring. Cogn. Sci. 25, 471–533 (2001)CrossRefGoogle Scholar
  25. Chi M.T.H., Roy M., Hausmann R.: Observing tutoring collaboratively: insights about tutoring effectiveness from vicarious learning. Cogn. Sci. 32(2), 301–341 (2008)CrossRefGoogle Scholar
  26. Chi, M., VanLehn, K., Litman, D.: Do micro-level tutorial decisions matter: applying reinforcement learning to induce pedagogical tutorial tactics. In: Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS’10), Pittsburgh, United States, pp. 184–193 (2010a)Google Scholar
  27. Chi, M., VanLehn, K., Litman, D., Jordan, P.: Inducing effective pedagogical strategies using learning context features. In: Proceedings of the 18th International Conference on User Modeling, Adaptation and Personalization (UMAP’10), Kona, Hawaii, pp. 147–158 (2010b)Google Scholar
  28. Chi, M., VanLehn, K., Litman, D., Jordan P.: Empirically evaluating the Application of reinforcement learning to the induction of effective and adaptive pedagogical strategies. User model User-Adap. Inter. (to appear). Special issue on educational data mining for personalized educational systemsGoogle Scholar
  29. Cohen P., Beal C.: Temporal data mining for educational applications. Int. J. Softw. Inform. 3(1), 31–46 (2009)Google Scholar
  30. Conati C., Gertner A., VanLehn K.: Using Bayesian networks to manage uncertainty in student modeling. User Model. User-Adap. Inter. 12(4), 371–417 (2002)CrossRefMATHGoogle Scholar
  31. Conati C., Maclaren H.: Empirically building and evaluating a probabilistic model of user affect. User Model. User-Adap. Inter. 19(3), 267–303 (2009)CrossRefGoogle Scholar
  32. Corbett A.T., Anderson J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-Adap. Inter. 4(4), 253–278 (1995)CrossRefGoogle Scholar
  33. D’Mello S., Craig S., Sullins J., Graesser A.: Predicting affective states expressed through an emote-aloud procedure from autotutor’s mixed-initiative dialogue. Int. J. Artif. Intell. Educ. 16(1), 3–28 (2006)Google Scholar
  34. Davis W., Carson C., Ammeter A., Treadway D.: The interactive effects of goal orientation and feedback specificity on task performance. Human Perform. 18, 409–426 (2005)CrossRefGoogle Scholar
  35. Dean T., Kanazawa K.: A model for reasoning about persistence and causation. Comput. Intell. 5(3), 142–150 (1989)CrossRefGoogle Scholar
  36. Dempster L., Laird N., Rubin D.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. 39(1), 1–38 (1977)MATHMathSciNetGoogle Scholar
  37. Feng, M., Beck, J., Heffernan, N.T.: Using learning decomposition and bootstrapping with randomization to compare the impact of different educational interventions on learning. In: Proceedings of the 2nd International Conference on Educational Data Mining (EDM’09), Cordoba, Spain, pp. 51–60 (2009)Google Scholar
  38. Getoor L., Rhee J., Koller D., Small P.: Understanding tuberculosis epidemiology using probabilistic relational models. J. Artif. Intell. Med. 30, 233–256 (2004)CrossRefGoogle Scholar
  39. Hays, M., Lane, C., Auerbach, D., Core, M., Gomboc, D., Rosenberg, M.: Feedback specificity and the learning of intercultural communication skills. In: Proceedings of the 14th International Conference on Artificial Intelligence in Education (AIED’09), Brighton, UK, pp. 391–398 (2009)Google Scholar
  40. Heckerman D.: Bayesian networks for data mining. Data Min. Knowl. Discov. 1, 79–119 (1997)CrossRefGoogle Scholar
  41. Koedinger, K.R., Anderson, Hadley, W.H., Mark, M.A.: Intelligent tutoring goes to school in the big city. In: Proceedings of the 3rd International Conference on Artificial Intelligence and Education (AIED’95), Washington, DC, United States, pp. 421–428 (1995)Google Scholar
  42. Mayo M., Mitrovic A.: Optimizing ITS behaviour with Bayesian networks and decision theory. Int. J. Artif. Intell. Educ. 12, 124–153 (2001)Google Scholar
  43. Muldner, K., Conati, C.: Using similarity to infer meta-cognitive behaviors during analogical problem solving. In: Proceedings of the 10th International Conference on User Modeling (UM’05), Edinborough, Scottland, pp. 134–143 (2005)Google Scholar
  44. Muldner, K., Burleson, W., de Sande, B., VanLehn, K.: An analysis of gaming behaviors in an intelligent tutoring system. In: Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS’10), Pittsburgh, United States, pp. 195–205 (2010)Google Scholar
  45. Muldner K., Conati C.: Scaffolding meta-cognitive skills for effective analogical problem solving via tailored example selection. Int. J. Artif. Intell. Educ. 20(2), 99–136 (2010)Google Scholar
  46. Murphy, K.: Dynamic Bayesian Networks: Representation, Inference and Learning. Ph.D. Thesis, UC, Berkeley (2002)Google Scholar
  47. Murphy, K.: Bayes Net Toolbox for Matlab. http://bnt.sourceforge.net. Accessed 2010 April 1 (2004)
  48. Murray, R.C., VanLehn, K.: Effects of dissuading unnecessary help requests while providing proactive help. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED’05), Amsterdam, Netherlands, pp. 887–889 (2005)Google Scholar
  49. Nelson-Le Gall S.: Help-seeking behavior in learning review of research in education. Rev. Res. Educ. 12, 55–90 (1985)Google Scholar
  50. Netica Reference Manual. Retrieved March 1, 2010, from www.norsys.com/netica-j/docs/NeticaJ_Man.pdf (2010)
  51. Phye G.D., Sanders C.: Advice and feedback: elements of practice for problem solving. Contemp. Educ. Psychol. 19(3), 286–301 (1994)CrossRefGoogle Scholar
  52. Rai, D., Gong, Y., Beck, J.: Using Dirichlet priors to improve model parameter plausibility. In: Proceedings of the international conference on educational data mining (EDM’09), pp. 141–150 (2009)Google Scholar
  53. Razzaq, L., Heffernan, N.: To tutor or not to tutor: that is the question. In: Proceedings of th 2nd International Conference on Artificial Intelligence in Education (AIED’09), Cordoba, Spain, pp. 457–464 (2009)Google Scholar
  54. Renkl A.: Learning from worked-examples: a study on individual differences. Cogn. Sci. 21(1), 1–30 (1997)CrossRefGoogle Scholar
  55. Reye J.: Student modeling based on belief networks. Int. J. Artif. Intell. Educ. 14, 1–33 (2004)Google Scholar
  56. Ritter, S., Harris, T., Nixon, T., Dickison, D., Murray, R.C., Towle, B.: Reducing the knowledge tracing space. In: Proceedings of the 1st International Conference On Educational Data Mining (EDM’08), Montreal, Canada, pp. 151–160 (2008)Google Scholar
  57. Rodin A., Mosley T.H., Clark A.G., Sing C.F., Boerwinkle E.: Mining genetic epidemiology data with Bayesian networks application to APOE gene variation and plasma lipid levels. J. Comput. Biol. 12(1), 1–11 (2005)CrossRefGoogle Scholar
  58. Rodrigo, M., Baker, R., d’Mello, S., Gonzalez, M.C.T., Lagud, M., Lim, S., Macapanpan, A., Pascua, S., Santillano, J., Sugay, J., Tep, S., Viehland, N.: Comparing learners affect while using an intelligent tutoring systems and a simulation problem solving game. In: Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS’08), Montreal, Canada, pp. 40–49 (2008)Google Scholar
  59. Roll, I., Aleven, V., McLaren, B., Ryu, E., Baker, R.S.J.d., Koedinger, K.: The help tutor: does metacognitive feedback improve students’ help-seeking actions, skills and learning? In: Proceedings of the Eight International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 360–369 (2006)Google Scholar
  60. Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach. Los Altos, CA, Morgan-Kaufman (2009)Google Scholar
  61. Shih, B., Koedinger, K., Scheines, R.: A response time model for bottom-out hints as worked examples. In: Proceedings of the 1st International Conference on Educational Data Mining (EDM’08), Montreal Canada, pp. 117–126 (2008)Google Scholar
  62. Shute V.: Focus on formative feedback. Rev. Educ. Res. 78(1), 153–189 (2008)CrossRefGoogle Scholar
  63. VanLehn K.: Analogy events: how examples are used during problem solving. Cogn. Sci. 22(3), 347–388 (1998)CrossRefGoogle Scholar
  64. VanLehn K., Lynch C., Schulze K., Shapiro J.A., Shelby R., Taylor L., Treacy D., Weinstein A., WintersgillM.TheAndesphysics tutoring system: Lessons learned. Int. J. Artif. Intell. Educ. 15(3), 1–47 (2005)Google Scholar
  65. VassarStats: Website for Statistical Computation. Retrieved April 1, 2010, from http://faculty.vassar.edu/lowry/VassarStats.html
  66. Walonoski, J.A., Heffernan, N.T.: Detection and analysis of off-task gaming behavior in intelligent tutoring systems. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 382–391 (2006a)Google Scholar
  67. Walonoski, J.A., Heffernan, N.T.: Prevention of off-task gaming behavior in intelligent tutoring systems. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 722–724 (2006b)Google Scholar
  68. Zhang, X., Mostow, J., Beck, J.: A case study empirical comparison of three methods to evaluate tutorial behaviors. In: Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS’08), Montral, Canada, pp. 122–131 (2008)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  • Kasia Muldner
    • 1
  • Winslow Burleson
    • 2
  • Brett Van de Sande
    • 2
  • Kurt VanLehn
    • 2
  1. 1.Department of PsychologyArizona State UniversityTempeUSA
  2. 2.School of Computing, Informatics and Decision Systems EngineeringArizona State UniversityTempeUSA

Personalised recommendations