Multimedia Tools and Applications

, Volume 74, Issue 9, pp 3183–3206 | Cite as

Affective-aware tutoring platform for interactive digital television

  • Sandra Baldassarri
  • Isabelle Hupont
  • David Abadía
  • Eva Cerezo


Interactive Digital TeleVision (IDTV) is emerging as a potentially important medium for learning at home. This paper presents a novel affective-aware tutoring platform for IDTV which makes use of automatic facial emotion recognition to improve the tutor-student relationship. The system goes further than simply broadcasting an interactive educational application by allowing the personalization of the course content. The tutor can easily access academic information relating to the students and also emotional information captured from learners’ facial expressions. In this way, depending on their academic and affective progress, the tutor can send personal messages or extra educational contents for improving students’ learning. In order to include these features it was necessary to address some important technical challenges derived from IDTV hardware and software restrictions. The system has been successfully tested with real students and tutors in a non-laboratory environment. Our system tries to advance in the challenge of providing to distance learning systems with the perceptual abilities of human teachers with the final aim of improving students learning experience and outcome. Nevertheless, there is still relatively little understanding of the impact of affect on students’ behaviour and learning and of the dynamics of affect during learning with software. Systems like ours would make it possible to attack these relevant open questions.


Affective computing Learning technologies Facial recognition Facial expressions Emotions Interactive digital tv 



This work has been partly financed by the Spanish “Dirección General de Investigación”, contract number TIN2011-24660, by the CYTED, contract number 512RT0461, by the Mechatronics and Systems Group (SISTRONIC) of the Aragon Institute of Technology and by the Spanish “Ministerio de Ciencia e Innovación” in the context of the QuEEN project, contract number IPT-2011-1235-430000.


  1. 1.
    Abadía D, Navamuel JJ, Vea-Murguia J, Alvarez F, Menendez JM, Sanchez FA, Fernandez G, Rovira M, Domech A (2009) i-LAB: Distributed laboratory network for interactive TV set-top box and services testing, IEEE 13th International Symposium on Consumer Electronics 773–776Google Scholar
  2. 2.
    Affdex (2013) Available:
  3. 3.
    Alexander S, Sarrafzadeh A (2004) Interfaces that adapt like humans, computer human interaction 641–645Google Scholar
  4. 4.
    An KH, Chung MJ (2009) Cognitive face analysis system for future interactive TV. IEEE Transactions on Consumer Eectronics 55(4):2271–2279CrossRefGoogle Scholar
  5. 5.
    Baker RR, D’Mello SK, Rodrigo MMT, Graesser AC (2010) Better to be frustrated than bored: The incidence, persistence,a nd impact of learners’cognitive–affective states during interactions with three different computer-based learning environments. Int J Human-Computer Studies 68:223–241CrossRefGoogle Scholar
  6. 6.
    Bates PJ (2005) Learning through iDTV–results of t-learning study. Europ Confer Interact Telev 137–138Google Scholar
  7. 7.
    Bellotti F, Pellegrino M, Tsampoulatidis I, Vrochidis S, Lhoas P, Bo G, De Gloria A, Kompatsiaris I (2010) An integrated framework for personalized T-Learning in Cases on transnational learning and technologically enabled environments 118Google Scholar
  8. 8.
    Brusilovsky P, Schwarz E, Weber G (1996) ELM–ART: An intelligent tutoring system on World Wide Web. In Proceedings 3rd international conference on intelligent tutoring systems, ITS 96, pp. 261–269.Google Scholar
  9. 9.
    Brusilovsky P, Vassileva J (2003) Course sequencing techniques for large-scale web-based education. International Journal of Continuing Engineering Education and Lifelong Learning 13(1–2):75–94CrossRefGoogle Scholar
  10. 10.
    Burleson W (2006) Affective learning companions: Strategies for empathetic agents with real-time multimodal affective sensing to foster meta-cognitive and meta-affective approaches to learning, motivation, and perseverance. Doctoral Thesis, Massachusetts In-stitute of Technology.Google Scholar
  11. 11.
    Caridakis G, Malatesta L, Kessous L, Amir N, Paouzaiou A, Karpouzis K (2006) Modeling naturalistic affective states via facial and vocal expression recognition. Intern Confer Multim Interf 146–154Google Scholar
  12. 12.
    Chen C (2008) Intelligent web-based learning system with personalized learning path guidance. Computers & Education vol 51:787–814CrossRefGoogle Scholar
  13. 13.
    Clark D (2013) MOOCs: Taxonomy of 8 types of MOOC. Donald Clark Paln B.
  14. 14.
    Cmolik L, Mikovec Z, Slavik P, Mannova B (2007) Personalized e-learning in interactive digital television environment, in IADIS International Conference WWW/Internet, 35–39Google Scholar
  15. 15.
    Conati C (2002) Probabilistic assessment of user’s emotions in educational games. Journal of Applied Artificial Intelligence 16:555–575CrossRefGoogle Scholar
  16. 16.
    Conati C, Zhao X (2004) Building and evaluating an intelligent pedagogical agent to improve the effectiveness of an educational game, Proceedings of the Ninth International Conference on Intelligent User Interface, pp. 6–13Google Scholar
  17. 17.
    D’Mello S, Jackson T, Craig S, Morgan B, Chipman P, White H, El Kaliouby R, Picard RW, Graesser A (2008) Auto tutor detects and responds to learners affective and cognitive states. In: Proceedings of the Workshop on Emotional and Cognitive Issues at the International Conference of Intelligent Tutoring Systems, pp. 23–27Google Scholar
  18. 18.
    D’Mello SK, Lehman B, Graesser A (2011) A motivationally supportive affect-sensitive autotutor. In New Perspectives on Affect and Learning Technologies, 113–126. New York: SpringerGoogle Scholar
  19. 19.
    Damásio M, Quico C (2004) T-learning and interactive television edutainment: The Portuguese case study. ED-Media 4511–4518Google Scholar
  20. 20.
    IPEA: Instituto de Pesquisa Economica Aplicada (2013) Panorama da Comunicacao e das telecomunicacoes no Brasil, (in portuguesse):
  21. 21.
    de Vicente A, Pain H (2002) Informing the detection of the students’ motivational state: An empirical study, Proceedings of the Sixth International Conference on Intelligent Tutoring Systems, pp. 933–943Google Scholar
  22. 22.
    Desmarais MC, d Baker RS (2012) A review of recent advances in learner and skill modeling in intelligent learning environments. User Model User-Adap Inter 22(1–2):9–38CrossRefGoogle Scholar
  23. 23.
    dos Santos DT, do Vale DT, Meloni LG (2007) Digital TV and distance learning: Potentials and limitations. In: Proceedings of the 36 th Annual Conference on Frontiers in Education pp. 1–6Google Scholar
  24. 24.
    Dosi A, Prario B (2004) New frontiers of T-learning: The emergence of interactive digital broadcasting learning services in Europe. ED-Media, pp. 4831–4836Google Scholar
  25. 25.
    Ekman P, Friesen WV, Hager JC (2002) Facial action coding system, Research Nexus eBookGoogle Scholar
  26. 26.
    eMotion©: emotion recognition software, 2013. Available:
  27. 27.
    Face API technical specifications brochure. Available:
  28. 28.
    Fragopanagos N, Taylor JG (2005) Emotion recognition in human–computer interaction. Neural Netw 18:389–405CrossRefGoogle Scholar
  29. 29.
    Gee JP (2004) Situated language and learning: A critique of traditional schooling. Routledge, Taylor & Francis, LondonGoogle Scholar
  30. 30.
    Graesser AC, D’Mello S, Person NK (2009) Metaknowledge in tutoring. In: Hacker D, Donlosky J, Graesser AC (eds) Handbook of metacognition in education. Taylor & Francis, MahwahGoogle Scholar
  31. 31.
    Hammal Z, Couvreur L, Caplier A, Rombaut M (2007) Facial expression classification: An approach based on the fusion of facial deformations using the transferable belief model. Int J Approx Reason 46:542–567CrossRefGoogle Scholar
  32. 32.
    Henze N, Nejdl W (2001) Adaptation in open corpus hypermedia. Int J Artif Intell Educ 12(4):325–350Google Scholar
  33. 33.
    Hone K (2006) Empathic agents to reduce user frustration: the effects of varying agent characteristics. Interact Comput 18:227–245CrossRefGoogle Scholar
  34. 34.
    Hupont I, Baldassarri S, Cerezo E (2013) Facial emotional classification: From a discrete perspective to a continuous emotional space. Pattern Anal Applic 16(1):41–54CrossRefMathSciNetGoogle Scholar
  35. 35.
    Hupont I, Ballano S, Baldassarri S, Cerezo E (2011) Scalable multimodal fusion for continuous affect sensing, Proc. IEEE Workshop on Affective Computational Intelligence (WACI 2011), Symposium Series on Computational Intelligence, ISBN 978-1-61284-082-6, pp. 68-75Google Scholar
  36. 36.
    Hupont I, Ballano S, Cerezo E, Baldassarri S (2013) From a discrete perspective of emotions to continuous, dynamic and multimodal affect sensing, Advances in Emotion Recognition, ISBN 978-1118130667, Wiley-BlackwellGoogle Scholar
  37. 37.
    Isen AM (2000) Positive affect and decision making. In: Lewis M, Haviland J (eds) Handbook of emotions. Guilford, New YorkGoogle Scholar
  38. 38.
    Kalman RE (1960) A new approach to linear filtering and prediction problems. Transactions of the ASME - Journal of Basic Engineering, Series D 82:34–45Google Scholar
  39. 39.
    Kapoor A, Burleson W, Picard RW (2007) Automatic prediction of frustration. International Journal of Human-Computer Studies 65:724–736CrossRefGoogle Scholar
  40. 40.
    Kim J, Kang S (2011) An ontology-based personalized target advertisement system on interactive TV. IEEE International Conference on Consumer Electronics, pp. 895–896Google Scholar
  41. 41.
    Kort B, Reilly R, Picard R (2001) An affective model of interplay between emotions and learning: reengineering educational pedagogy—building a learning companion, Proceedings IEEE International Conference on Advanced Learning Technology: Issues, Achievements and Challenges. IEEE Computer Society, pp.43–48.Google Scholar
  42. 42.
    Kumar P, Yildirim EA (2005) Minimum-volume enclosing ellipsoids and core sets. J Optim Theory Appl 126:1–21CrossRefMATHMathSciNetGoogle Scholar
  43. 43.
    López-Nores M, Blanco-Fernández Y, Pazos-Arias JJ, García-Duque J, Ramos-Cabrer M, Gil-Solla A, Díaz-Redondo RP, Fernández-Vilas A (2009) Receiver-side semantic reasoning for digital TV personalization in the absence of return channels. Multimed Tools Appl 41:407–436CrossRefGoogle Scholar
  44. 44.
    Mandryk RL, Atkins MS (2007) A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies 65(4):329–347CrossRefGoogle Scholar
  45. 45.
    McDuff D, Kaliouby RE, Picard RW (2012) Crowdsourcing facial responses to online videos. IEEE Trans on Affective Computing 3(4):456–468CrossRefGoogle Scholar
  46. 46.
    Merrill DC, Reiser BJ, Ranney M, Trafton JG (1992) Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. J Learn Sci 2(3):277–305CrossRefGoogle Scholar
  47. 47.
    Montpetit M, Klym N, Mirlacher T (2011) The future of IPTV. Multimedia Tools and Applications 53:519–532CrossRefGoogle Scholar
  48. 48.
    Morrell DR, Stirling WC (2003) An extended set-valued Kalman filter, In Proceedings of ISIPTA, 396–407Google Scholar
  49. 49.
    Nicolaou MA, Gunes H, Pantic M (2011) Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Trans Affect Comput 2(2):92–105CrossRefGoogle Scholar
  50. 50.
    Papanikolaou KA, Grigoriadou M (2002) Towards new forms of knowledge communication: The adaptive dimension of a webbased learning environment. Computers and Education 39:333–360CrossRefGoogle Scholar
  51. 51.
    Pazos-Arias JJ, López-Nores M, García-Duque J, Díaz-Redondo RP, Blanco-Fernández Y, Ramos-Cabrer M, Gil-Solla A, Fernández-Vilas A (2008) Provision of distance learning services over Interactive Digital TV with MHP. Comput Educ 50:927–949CrossRefGoogle Scholar
  52. 52.
    Petridis S, Gunes H, Kaltwang S, Pantic M, Static vs. dynamic modeling of human nonverbal behavior from multiple cues and modalities, Proceedings of the International Conference on Multimodal Interfaces, 2009, pp. 23–30Google Scholar
  53. 53.
    Picard RW (2000) Affective computing, The MIT Press Google Scholar
  54. 54.
    Picard RW, Papert S, Bender W, Blumberg B, Breazeal C, Cavallo D, Machover T, Resnick M, Roy D, Strohecker C (2004) Affective learning-a manifesto. BT Technology Journal 22(4)Google Scholar
  55. 55.
    Plutchik R (1980) Emotion: A psychoevolutionary synthesis. Harper & Row, New YorkGoogle Scholar
  56. 56.
    Porayska-Pomsta K, Mavrikis M, Pain H (2008) Diagnosing and acting on student affect: The tutor’s perspective. User Model User-Adap Inter 18(1–2):125–173CrossRefGoogle Scholar
  57. 57.
    Rey-López M, Díaz-Redondo RP, Fernández-Vilas A, Pazos-Arias JJ, López-Nores M, García-Duque J, Gil-Solla A, Ramos-Cabrer M (2008) T-MAESTRO and its authoring tool: Using adaptation to integrate entertainment into personalized t-learning. Multimedia Tools and Applications 40:409–451CrossRefGoogle Scholar
  58. 58.
    Rey-López M, Fernández-Vilas A, Díaz-Redondo RP (2006) A model for personalized learning through IDTV. In: Wade V, Ashman H, Smyth B (eds.) AH 2006, LNCS 4018, pp. 457–461Google Scholar
  59. 59.
    Roland H (2000) Logically optimal curriculum sequences for adaptive hypermedia systems. In International conference on adaptive hypermedia and adaptive web-based system, pp. 121–132.Google Scholar
  60. 60.
    Russell JA (1980) A circumplex model of affect. Journal of personality and social psychology 39(6):1161–1178CrossRefGoogle Scholar
  61. 61.
    Russell T (1997) Technology wars: Winners and losers. Educom Review 32(2):44–46Google Scholar
  62. 62.
    Sarrafzadeh A, Alexander S, Dadgostar F, Fan C, Bigdeli A (2008) How do you know that I don’t understand? A look at the future of intelligent tutoring systems, Computers in Human Behavior vol 24:1342–1363Google Scholar
  63. 63.
    Schiaffino S, Garcia P, Amandi A (2008) eTeacher: Providing personalized assistance to e-learning students. Computers & Education vol 51:1744–1754CrossRefGoogle Scholar
  64. 64.
    Smith ASG, Blandford A (2003) MLTutor: An application of machine learning algorithms for an adaptive web-based information system. Int J Artif Intell Educ 13(2–4):233–260Google Scholar
  65. 65.
    Snow R, Farr M (1987) Cognitive-conative-affective processes in aptitude, learning, and instruction: An introduction. Conative and affective process analysis 3:1–10Google Scholar
  66. 66.
    Soleymani M, Pantic M (2013) Emotional Aware TV, Proceedings of TVUX-2013: Workshop on Exploring and Enhancing the User Experience for TV at ACM CHI 2013 Google Scholar
  67. 67.
    Soyel H, Demirel H (2007) Facial expression recognition using 3D facial feature distances. Lect Notes Comput Sci 4633:831–883CrossRefGoogle Scholar
  68. 68.
    Suraweera P, Mitrovic A (2002) KERMIT: A constraint-based tutor for database modeling. In Intelligent Tutoring Systems, 377–387. Springer Berlin HeidelbergGoogle Scholar
  69. 69.
    VanLehn K (2006) The behavior of tutoring systems. International journal of artificial intelligence in education 16(3):227–265Google Scholar
  70. 70.
  71. 71.
    Wallhoff F (2006) Facial expressions and emotion database, Technische Universität München Available:
  72. 72.
    Whissell CM (1989) The dictionary of affect in language, Emotion: Theory, research and experience, vol 4. The Measurement of Emotions, New YorkGoogle Scholar
  73. 73.
    Whissell C, Whissell’s dictionary of affect in language technical manual and user’s guide. Available:
  74. 74.
    Witten I, Frank E (2005) Data mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San FranciscoGoogle Scholar
  75. 75.
    Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurement of levels of interest from video. IEEE Transactions on Multimedia 8:500–508CrossRefGoogle Scholar
  76. 76.
    Zapata-Ros M, El diseño instruccional de los MOOCs y el de los nuevos cursos online abiertos personalizados (POOCs), 2013 (In Spanish) [Preprint]

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Sandra Baldassarri
    • 1
  • Isabelle Hupont
    • 2
  • David Abadía
    • 2
  • Eva Cerezo
    • 1
  1. 1.GIGA-AffectiveLab, Computer Science Department, Engineering Research Institute of Aragon (I3A)Universidad of ZaragozaZaragozaSpain
  2. 2.Multimedia Technologies DivisionAragon Institute of TechnologyZaragozaSpain

Personalised recommendations