Skip to main content
SpringerLink
Log in

Predicting Learners’ Emotions in Mobile MOOC Learning via a Multimodal Intelligent Tutor

  • Conference paper
  • First Online:
Intelligent Tutoring Systems (ITS 2018)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 10858))

Included in the following conference series:

Abstract

Massive Open Online Courses (MOOCs) are a promising approach for scalable knowledge dissemination. However, they also face major challenges such as low engagement, low retention rate, and lack of personalization. We propose AttentiveLearner2, a multimodal intelligent tutor running on unmodified smartphones, to supplement today’s clickstream-based learning analytics for MOOCs. AttentiveLearner2 uses both the front and back cameras of a smartphone as two complementary and fine-grained feedback channels in real time: the back camera monitors learners’ photoplethysmography (PPG) signals and the front camera tracks their facial expressions during MOOC learning. AttentiveLearner2 implicitly infers learners’ affective and cognitive states during learning from their PPG signals and facial expressions. Through a 26-participant user study, we found that: (1) AttentiveLearner2 can detect 6 emotions in mobile MOOC learning reliably with high accuracy (average accuracy = 84.4%); (2) the detected emotions can predict learning outcomes (best R2 = 50.6%); and (3) it is feasible to track both PPG signals and facial expressions in real time in a scalable manner on today’s unmodified smartphones.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Chuang, I., Ho, A.D.: HarvardX and MITx: four years of open online courses–Fall 2012-Summer 2016 (2016)

    Google Scholar 

  2. Coetzee, D., Fox, A., Hearst, M.A., Hartmann, B.: Chatrooms in MOOCs: all talk and no action. In: ACM Conference on Learning@ Scale, pp. 127–136. ACM (2014)

    Google Scholar 

  3. D’Mello, S.K., Dowell, N., Graesser, A.: Unimodal and multimodal human perception of naturalistic non-basic affective states during human-computer interactions. IEEE Trans. Affect. Comput. 4(4), 452–465 (2013)

    Article  Google Scholar 

  4. D’Mello, S.K., Graesser, A.: Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Model. User Adapt. Interact. 20(2), 147–187 (2010)

    Article  Google Scholar 

  5. Grafsgaard, J., Wiggins, J.B., Boyer, K.E., Wiebe, E.N., Lester, J.: Automatically recognizing facial expression: Predicting engagement and frustration. In: Educational Data Mining 2013 (2013)

    Google Scholar 

  6. Guo, P.J., Kim, J., Rubin, R.: How video production affects student engagement: An empirical study of MOOC videos. In: ACM Conference on Learning@ Scale, pp. 41–50. ACM (2014)

    Google Scholar 

  7. Han, T., Xiao, X., Shi, L., Canny, J., Wang, J.: Balancing accuracy and fun: designing engaging camera based mobile games for implicit heart rate monitoring. In: ACM Conference on Human Factors in Computing Systems, pp. 847–856. ACM (2015)

    Google Scholar 

  8. Hjortskov, N., Rissén, D., Blangsted, A.K., Fallentin, N., Lundberg, U., Søgaard, K.: The effect of mental stress on heart rate variability and blood pressure during computer work. Eur. J. Appl. Physiol. 92(1–2), 84–89 (2004)

    Article  Google Scholar 

  9. Jeni, L.A., Cohn, J.F., De La Torre, F.: Facing imbalanced data–recommendations for the use of performance metrics. In: Humaine Association Conference on Affective Computing and Intelligent Interaction, pp. 245–251. IEEE (2013)

    Google Scholar 

  10. Kim, J., Guo, P.J., Seaton, D.T., Mitros, P., Gajos, K.Z., Miller, R.C.: Understanding in-video dropouts and interaction peaks in online lecture videos. In: ACM Conference on Learning@ Scale, pp. 31–40. ACM (2014)

    Google Scholar 

  11. Krause, M., Mogalle, M., Pohl, H., Williams, J.J.: A playful game changer: Fostering student retention in online education with social gamification. In: ACM Conference on Learning@ Scale, pp. 95–102. ACM (2015)

    Google Scholar 

  12. McDuff, D., Mahmoud, A., Mavadati, M., Amr, M., Turcot, J., Kaliouby, R.e.: Affdex SDK: a cross-platform real-time multi-face expression recognition toolkit. In: ACM Conference on Human Factors in Computing Systems, pp. 3723–3726. ACM (2016)

    Google Scholar 

  13. Monkaresi, H., Bosch, N., Calvo, R.A., D’Mello, S.K.: Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Trans. Affect. Comput. 8(1), 15–28 (2017)

    Article  Google Scholar 

  14. Oviatt, S.: The Design of Future Educational Interfaces. Routledge, London (2013)

    Google Scholar 

  15. Pham, P., Wang, J.: Adaptive review for mobile MOOC learning via implicit physiological signal sensing. In: ACM International Conference on Multimodal Interaction, pp. 37–44. ACM (2016)

    Google Scholar 

  16. Pham, P., Wang, J.: AttentiveLearner: improving mobile MOOC learning via implicit heart rate tracking. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M. (eds.) AIED 2015. LNCS (LNAI), vol. 9112, pp. 367–376. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19773-9_37

    Chapter  Google Scholar 

  17. Pham, P., Wang, J.: AttentiveLearner2: a multimodal approach for improving MOOC learning on mobile devices. In: André, E., Baker, R., Hu, X., Rodrigo, M., du Boulay, B. (eds.) AIED 2017. LNCS (LNAI), vol. 10331, pp. 561–564. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61425-0_64

    Chapter  Google Scholar 

  18. Pham, P., Wang, J.: Understanding emotional responses to mobile video advertisements via physiological signal sensing and facial expression analysis. In: The 22nd International Conference on Intelligent User Interfaces, pp. 67–78. ACM (2017)

    Google Scholar 

  19. Van der Sluis, F., Ginn, J., Van der Zee, T.: Explaining student behavior at scale: the influence of video complexity on student dwelling time. In: ACM Conference on Learning@ Scale, pp. 51–60. ACM (2016)

    Google Scholar 

  20. Xiao, X., Han, T., Wang, J.: LensGesture: augmenting mobile interactions with back-of-device finger gestures. In: ACM on International Conference on Multimodal Interaction, pp. 287–294. ACM (2013)

    Google Scholar 

  21. Xiao, X., Wang, J.: Towards attentive, bi-directional MOOC learning on mobile devices. In: ACM on International Conference on Multimodal Interaction, pp. 163–170. ACM (2015)

    Google Scholar 

  22. Xiao, X., Wang, J.: Understanding and detecting divided attention in mobile MOOC learning. In: ACM Conference on Human Factors in Computing Systems, pp. 2411–2415. ACM (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science and LRDC, University of Pittsburgh, Pittsburgh, PA, USA

    Phuong Pham & Jingtao Wang

Authors
  1. Phuong Pham
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingtao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jingtao Wang .

Editor information

Editors and Affiliations

  1. Université du Québec, Montreal, Québec, Canada

    Roger Nkambou

  2. NCSU, Raleigh, North Carolina, USA

    Roger Azevedo

  3. University of Saskatchewan, Saskatoon, Saskatchewan, Canada

    Julita Vassileva

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pham, P., Wang, J. (2018). Predicting Learners’ Emotions in Mobile MOOC Learning via a Multimodal Intelligent Tutor. In: Nkambou, R., Azevedo, R., Vassileva, J. (eds) Intelligent Tutoring Systems. ITS 2018. Lecture Notes in Computer Science(), vol 10858. Springer, Cham. https://doi.org/10.1007/978-3-319-91464-0_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-91464-0_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-91463-3

  • Online ISBN: 978-3-319-91464-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation