Skip to main content
SpringerLink
Log in

Log file analysis for disengagement detection in e-Learning environments

  • Original Paper
  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

Most e-Learning systems store data about the learner’s actions in log files, which give us detailed information about learner behaviour. Data mining and machine learning techniques can give meaning to these data and provide valuable information for learning improvement. One area that is of particular importance in the design of e-Learning systems is learner motivation as it is a key factor in the quality of learning and in the prevention of attrition. One aspect of motivation is engagement, a necessary condition for effective learning. Using data mining techniques for log file analysis, our research investigates the possibility of predicting users’ level of engagement, with a focus on disengaged learners. As demonstrated previously across two different e-Learning systems, HTML-Tutor and iHelp, disengagement can be predicted by monitoring the learners’ actions (e.g. reading pages and taking test/quizzes). In this paper we present the findings of three studies that refine this prediction approach. Results from the first study show that two additional reading speed attributes can increase the accuracy of prediction. The second study suggests that distinguishing between two different patterns of disengagement (spending a long time on a page/test and browsing quickly through pages/tests) may improve prediction in some cases. The third study demonstrates the influence of exploratory behaviour on prediction, as most users at the first login familiarize themselves with the system before starting to learn.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Arroyo I., Woolf B.P.: Inferring learning and attitudes from a Bayesian Network of log file data. In: Looi, C.K., McCalla, G., Bredeweg, B., Breuker, J. (eds) Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology, pp. 33–40. IOS Press, Amsterdam (2005)

    Google Scholar 

  • Baker, R., Corbett, A., Koedinger, K.: Detecting student misuse of intelligent tutoring systems. In: Proceedings of the 7th International Conference on Intelligent Tutoring Systems, pp. 531–540 (2004)

  • Bandura A.: Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, Prentice Hall (1986)

    Google Scholar 

  • Beal, C.R., Lee, H.: Creating a pedagogical model that uses student self reports of motivation and mood to adapt ITS instruction. In: Proceedings of the Workshop on Emotion and Motivation in Educational Software (EMES). IOS Press, Amsterdam. Retrieved on 10 June 2006 from http://k12.usc.edu/Pubs/ (2005)

  • Beal, C.R., Qu, L., Lee, H.: Classifying learner engagement through integration of multiple data sources. In: Proceedings of the 21st National Conference on Artificial Intelligence. AAAI Press, Menlo Park. Retrieved on 10 June 2006 from http://k12.usc.edu/Pubs/ (2006)

  • Beck J.: Engagement tracing: using response times to model student disengagement. In: Looi, C., McCalla, G., Bredeweg, B., Breuker, J. (eds) Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology, pp. 88–95. IOS Press, Amsterdam (2005)

    Google Scholar 

  • Becta, U.K.: The benefits of an interactive whiteboard. Retrieved on 11 June 2006 from http://schools.becta.org.uk/index.php?section=tl&catcode=ss_tl_use_02&rid=86 (2006)

  • Breiman L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)

    MATH  MathSciNet  Google Scholar 

  • Chen, G.D., Shen, G.Y., Ou, K.L., Liu, B.: Promoting motivation and eliminating disorientation for web based courses by a multi-user game. Paper presented at The ED-MEDIA/ED-TELECOM 98 World Conference on Educational Multimedia and Hypermedia and World conference on Educational Telecommunications, June 20–25, Germany (1998)

  • Cocea M.: Assessment of motivation in online learning environments. In: Wade, V., Ashman, H., Smith, B. (eds) Proceedings of the 4th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, pp. 414–418. Springer, Berlin (2006)

    Chapter  Google Scholar 

  • Cocea, M., Weibelzahl, S.: Can log files analysis estimate learners’ level of motivation? In: Proceedings of ABIS Workshop, ABIS 2006—14th Workshop on Adaptivity and User Modeling in Interactive Systems, pp. 32–35. Hildesheim (2006)

  • Cocea M., Weibelzahl S.: Eliciting motivation knowledge from log files towards motivation diagnosis for Adaptive Systems. In: Conati, C., McCoy, K., Paliouras, G. (eds) Proceedings of the 11th International Conference on User Modelling, Lecture Notes in Artificial Intelligence (LNAI), vol. 4511, pp. 197–206. Springer, Berlin (2007a)

    Google Scholar 

  • Cocea M., Weibelzahl S.: Cross-system validation of engagement prediction from log files. In: Duval, E., Klamma, R., Wolpers, M. (eds) Creating New Learning Experiences on a Global Scale, 2nd European Conference on Technology Enhanced Learning, EC-TEL 2007, Lecture Notes in Computer Science (LNCS), vol. 4753, pp. 14–25. Springer, Berlin (2007b)

    Google Scholar 

  • Connolly T., Stansfield M.: Using games-based e-Learning technologies in overcoming difficulties in teaching information systems. J Inf Technol Educ 5, 459–476 (2006)

    Google Scholar 

  • Csikszentmihalyi M.: Finding Flow: The Psychology of Engagement with Everyday Life. BasicBooks, New York (1997)

    Google Scholar 

  • De Vicente A., Pain H.: Informing the detection of the students’ motivational state: an empirical study. In: Cerri, S.A., Gouarderes, G., Paraguau, F. (eds) Intelligent Tutoring Systems, 6th International Conference, pp. 933–943. Springer, Berlin (2002)

    Google Scholar 

  • Farzan, R., Brusilovsky, P.: Social navigation support in e-Learning: what are real footprints. In: Proceedings of IJCAI’05 Workshop on Intelligent Techniques for Web Personalization, pp. 49–56. Edinburgh (2005)

  • Gussak D., Baylor A.L.: Constructing agents for self-learning: animated agents as expressive vehicles. In: Lassner, D., McNaught, C. (eds) Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications, pp. 477–478. AACE, Chesapeake (2003)

    Google Scholar 

  • Hurley T., Weibelzahl S.: Eliciting adaptation knowledge from on-line tutors to increase motivation. In: Conati, C., McCoy, K., Paliouras, G. (eds) Proceedings of The 11th International Conference on User Modelling, Lecture Notes in Artificial Intelligence (LNAI), vol. 4511, pp. 370–374. Springer, Berlin (2007)

    Google Scholar 

  • Ishii, T., Saitou, M., Hiramoto, S.: An instructional design for developing learning contents by subject matter experts. In: Proceedings of the 9th World Conference on Continuing Engineering Education, pp. 469–474 (2004)

  • Johns, J., Woolf, B.: A dynamic mixture model to detect student motivation and proficiency. In: Proceedings of the 21st National Conference on Artificial Intelligence (AAAI-06). Boston, MA, pp. 163–168 (2006)

  • Keller J.M.: Development and use of the ARCS model of instructional design. J Instr Dev 10(3), 2–10 (1987)

    Article  Google Scholar 

  • Lombard, M., Snyder-Duch, J., Campanella Bracken, C.: Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research. Retrieved on 11 June 2007 from http://www.temple.edu/mmc/reliability (2003)

  • Machado, I., Martins, A., Paiva, A.: One for all and all for one: a learner modelling server in a multi-agent platform. In: Proceedings of the 7th International Conference on User Modelling, pp. 211–221. Springer, Wien (1999)

  • Martyn M.: Clickers in the classroom: an active learning approach. EDUCAUSE Q 30(2), 71–74 (2007)

    Google Scholar 

  • Mitchell T.M.: Machine Learning. McGraw-Hill, New York (1997)

    MATH  Google Scholar 

  • Pintrich P., Schunk D.: Motivation in Education: Theory, Research and Applications. 2nd edn. Prentice-Hall, Englewood Cliffs (2002)

    Google Scholar 

  • Quinlan R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Mateo (1993)

    Google Scholar 

  • Qu L., Wang N., Johnson W.L.: Detecting the learner’s motivational states in an interactive learning environment. In: Looi, C.-K., McCalla, G., Bredeweg, B., Breuker, J. (eds) Proceedings of the 12th International Conference on Artificial Intelligence in Education., pp. 547–554. IOS Press, Amsterdam (2005)

    Google Scholar 

  • Shneiderman B., Alavi M., Norman K., Borkowski E.: Windows of opportunity in electronic classrooms. Commun ACM 38(11), 19–24 (1995)

    Article  Google Scholar 

  • Speier C., Vessey I., Valacich J.S.: Effects of interruptions, task complexity, and information presentation on computer-supported decision-making performance. Decis Sci 34(4), 771–797 (2003)

    Article  Google Scholar 

  • Walonoski J., Heffernan N.T.: Detection and analysis of off-task gaming behavior in intelligent tutoring systems. In: Ikeda, Ashley, Chan (eds) Proceedings of the 8th International Conference in Intelligent Tutoring Systems, pp. 382–391. Springer, Berlin (2006a)

    Google Scholar 

  • Walonoski J., Heffernan N.T.: Prevention of off-task gaming behaviour within intelligent tutoring systems. In: Ikeda, Ashley, Chan (eds) Proceedings of the 8th International Conference in Intelligent Tutoring Systems, pp. 722–724. Springer, Berlin (2006b)

    Google Scholar 

  • Weber, G., Kuhl, H.-C., Weibelzahl, S.: Developing adaptive internet based courses with the authoring system NetCoach. In: Hypermedia: Openness, Structural Awareness, and Adaptivity (LNAI 2266), pp. 226–238. Springer, Berlin (2001)

  • Witten I.H., Frank E.: Data mining. Practical Machine Learning Tools and Techniques. 2nd edn. Morgan Kaufmann Publishers/Elsevier, San Mateo (2005)

    MATH  Google Scholar 

  • Zhang, G., Cheng, Z., He, A., Huang, T.: A WWW-based learner’s learning motivation detecting system. In: Proceedings of International Workshop on Research Directions and Challenge Problems in Advanced Information Systems Engineering, Honjo City, Japan, September 16–19, http://www.akita-pu.ac.jp/system/KEST2003/ (2003)

Download references

Author information

Authors and Affiliations

  1. School of Informatics, National College of Ireland, Mayor Street, Dublin 1, Ireland

    Mihaela Cocea & Stephan Weibelzahl

  2. London Knowledge Lab, Birkbeck College, University of London, 23-29 Emerald Street, London, WC1N 3Q, UK

    Mihaela Cocea

Authors
  1. Mihaela Cocea
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephan Weibelzahl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mihaela Cocea.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cocea, M., Weibelzahl, S. Log file analysis for disengagement detection in e-Learning environments. User Model User-Adap Inter 19, 341–385 (2009). https://doi.org/10.1007/s11257-009-9065-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-009-9065-5

Keywords

Search

Navigation