Advertisement

Towards emotion recognition from contextual information using machine learning

  • Martín G. Salido Ortega
  • Luis-Felipe RodríguezEmail author
  • J. Octavio Gutierrez-Garcia
Original Research
  • 17 Downloads

Abstract

Emotions influence cognitive processes that underlie human behavior. Whereas experiencing negative emotions may lead to develop psychological disorders, experiencing positive emotions may improve creative thinking and promote cooperative behavior. The importance of human emotions has led to the development of automatic emotion recognition systems based on analysis of speech waveforms, facial expressions, and physiological signals as well as text data mining. However, emotions are associated with a context (in which emotions are actually experienced), hence, this work focuses on emotion recognition from contextual information. In this paper, we present a study aimed to assess the feasibility of automatically recognizing emotions from individuals’ contexts. In this study, 32 participants provided information using a mobile application about their emotions and the context (e.g., companions, activities, and locations) in which these emotions were experienced. We used machine learning techniques to build individual models, general models, and gender-specific models to automatically recognize emotions of participants. The empirical results show that individuals’ emotions are highly related to their context and that automatic recognition of emotions in real-world situations is feasible by using contextual data.

Keywords

Emotion recognition Context-aware applications Smart devices Affective computing 

Notes

Acknowledgements

J. O. Gutierrez-Garcia gratefully acknowledges the financial support from the Asociación Mexicana de Cultura, A.C. This work was supported by PFCE 2019.

Compliance with ethical standards

Conflicts of interest

The authors declare that they have no conflict of interest.

References

  1. Ahn H, Picard RW (2005) Affective-cognitive learning and decision making: a motivational reward framework for affective agents. In: Tao J, Tan T, Picard RW (eds) Affective computing and intelligent Interaction. ACII 2005. Lecture notes in computer science, vol 3784. Springer, Berlin, Heidelberg, pp 866–873Google Scholar
  2. Alegre U, Augusto JC, Clark T (2016) Engineering context-aware systems and applications: a survey. J Syst Softw 117:55–83Google Scholar
  3. Ashkanasy NM, Daus CS (2002) Emotion in the workplace: the new challenge for managers. Acad Manage Exec 16(1):76–86Google Scholar
  4. Becerra R, Preece D, Campitelli G, Scott-Pillow G (2019) The assessment of emotional reactivity across negative and positive emotions: development and validation of the perth emotional reactivity scale (pers). Assessment 26(5):867–879Google Scholar
  5. Bechara A (2004) The role of emotion in decision-making: evidence from neurological patients with orbitofrontal damage. Brain Cogn 55(1):30–40Google Scholar
  6. Bechara A, Damasio H, Damasio AR (2000) Emotion, decision making and the orbitofrontal cortex. Cereb Cortex 10(3):295–307Google Scholar
  7. Bellavista P, Corradi A, Fanelli M, Foschini L (2012) A survey of context data distribution for mobile ubiquitous systems. ACM Comput Surv (CSUR) 44(4):1–45Google Scholar
  8. Brave S, Nass C (2007) Emotion in human-computer interaction. In: Sears A, Jacko JA (eds) The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications. CRC Press, Boca Raton, pp 103–118Google Scholar
  9. Breazeal C (2003) Emotion and sociable humanoid robots. Int J Hum Comput Stud 59(1):119–155Google Scholar
  10. Breiman L (2001) Random forests. Mach Learn 45(1):5–32zbMATHGoogle Scholar
  11. Cabanac M (1981) Physiological signals for thermal comfort. In: Cena K, Clark JA (eds) Bioengineering, thermal physiology and comfort. Elsevier, Amsterdam, pp 181–192Google Scholar
  12. Calvo RA, D’Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1(1):18–37Google Scholar
  13. Conati C (2002) Probabilistic assessment of user’s emotions in educational games. Appl Artif Intell 16(7–8):555–575Google Scholar
  14. Costa H, Macedo L (2012) Affective computing. Tech Report, University of Coimbra, Coimbra, Portugal. https://eden.dei.uc.pt/~hpcosta/docs/papers/techReports/TAMC-stateOfTheArt.pdf
  15. Croy I, Olgun S, Joraschky P (2011) Basic emotions elicited by odors and pictures. Emotion 11(6):1331–1335Google Scholar
  16. Deng J, Xu X, Zhang Z, Frühholz S, Schuller B (2017) Universum autoencoder-based domain adaptation for speech emotion recognition. IEEE Signal Process Lett 24(4):500–504Google Scholar
  17. Dimotakis N, Scott BA, Koopman J (2011) An experience sampling investigation of workplace interactions, affective states, and employee well-being. J Organ Behav 32(4):572–588Google Scholar
  18. Ekman P (2000) Basic emotions. In: Dalgleish T, Power M (eds) Handbook of cognition and emotion. Wiley, pp 45–60Google Scholar
  19. Ekman PE, Davidson RJ (1994) The nature of emotion: fundamental questions. Oxford University Press, OxfordGoogle Scholar
  20. El Ayadi M, Kamel MS, Karray F (2011) Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recogn 44(3):572–587zbMATHGoogle Scholar
  21. Farmer R, Sundberg ND (1986) Boredom proneness-the development and correlates of a new scale. J Pers Assess 50(1):4–17Google Scholar
  22. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36(1):259–275zbMATHGoogle Scholar
  23. Fawcett T (2006) An introduction to roc analysis. Pattern Recogn Lett 27(8):861–874MathSciNetGoogle Scholar
  24. Forman G, Scholz M (2010) Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement. ACM SIGKDD Explor Newslett 12(1):49–57Google Scholar
  25. Fredrickson BL (2001) The role of positive emotions in positive psychology: the broaden-and-build theory of positive emotions. Am Psychol 56(3):218–226Google Scholar
  26. Giatsoglou M, Vozalis MG, Diamantaras K, Vakali A, Sarigiannidis G, Chatzisavvas KC (2017) Sentiment analysis leveraging emotions and word embeddings. Expert Syst Appl 69:214–224Google Scholar
  27. Granat A, Gadassi R, Gilboa-Schechtman E, Feldman R (2017) Maternal depression and anxiety, social synchrony, and infant regulation of negative and positive emotions. Emotion 17(1):11–27Google Scholar
  28. Grünerbl A, Muaremi A, Osmani V, Bahle G, Oehler S, Tröster G, Mayora O, Haring C, Lukowicz P (2015) Smartphone-based recognition of states and state changes in bipolar disorder patients. IEEE J Biomed Health Inform 19(1):140–148Google Scholar
  29. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. ACM SIGKDD Explor Newslett 11(1):10–18Google Scholar
  30. Hutchings CV, Shum KW, Gawkrodger DJ (2001) Occupational contact dermatitis has an appreciable impact on quality of life. Contact Dermat 45(1):17–20Google Scholar
  31. John GH, Langley P (1995) Estimating continuous distributions in Bayesian classifiers. In: Proceedings of the eleventh conference on uncertainty in artificial intelligence. Morgan Kaufmann Publishers Inc., San Francisco, pp 338–345Google Scholar
  32. Junot A, Paquet Y, Martin-Krumm C (2017) Passion for outdoor activities and environmental behaviors: a look at emotions related to passionate activities. J Environ Psychol 53:177–184Google Scholar
  33. Kanjo E, Al-Husain L, Chamberlain A (2015) Emotions in context: examining pervasive affective sensing systems, applications, and analyses. Pers Ubiquit Comput 19(7):1197–1212Google Scholar
  34. Keller MC, Fredrickson BL, Ybarra O, Côté S, Johnson K, Mikels J, Conway A, Wager T (2005) A warm heart and a clear head: the contingent effects of weather on mood and cognition. Psychol Sci 16(9):724–731Google Scholar
  35. Kim HJ, Choi YS (2011) Emosens: affective entity scoring, a novel service recommendation framework for mobile platform. In: Proceedings of the 5th ACM conference on recommender system, pp 1–4Google Scholar
  36. Kim M, Chong SC, Chun C, Choi Y (2017) Effect of thermal sensation on emotional responses as measured through brain waves. Build Environ 118:32–39Google Scholar
  37. Kołakowska A (2018) Usefulness of keystroke dynamics features in user authentication and emotion recognition. In: Hippe Z, Kulikowski J, Mroczek T (eds) Human-computer systems interaction. Advances in intelligent systems and computing, vol 551. Springer, Cham, pp 42–52Google Scholar
  38. Kreibig SD (2010) Autonomic nervous system activity in emotion: a review. Biol Psychol 84(3):394–421Google Scholar
  39. Le Cessie S, Van Houwelingen JC (1992) Ridge estimators in logistic regression. Appl Stat 41(1):191–201zbMATHGoogle Scholar
  40. Lee H, Choi YS, Lee S, Park I (2012) Towards unobtrusive emotion recognition for affective social communication. In: IEEE consumer communications and networking conference, pp 260–264Google Scholar
  41. Lewis M (2008) The emergence of human emotions. In: Lewis M, Haviland-Jones JM, Feldman Barrett L (eds) Handbook of emotions, 3rd edn. Guilford Press, New York, pp 304–319Google Scholar
  42. LiKamWa R, Liu Y, Lane ND, Zhong L (2013) Moodscope: building a mood sensor from smartphone usage patterns. In: Proceedings of the 11th annual international conference on Mobile systems, applications, and services, Taipei, Taiwan, pp 389–402Google Scholar
  43. Liu KH, Huang DS, Li B (2007) Improving the performance of ICA based microarray data prediction models with genetic algorithm. In: 2007 IEEE congress on evolutionary computation, Singapore, pp 606–611Google Scholar
  44. Loewenstein G, Lerner JS (2003) The role of affect in decision making. In: Davidson RJ, Sherer KR, Goldsmith HH (eds) Handbook of affective science, Oxford University Press, pp 619–642Google Scholar
  45. Martin-Krumm C, Fenouillet F, Csillik A, Kern L, Besançon M, Heutte J, Paquet Y, Delas Y, Trousselard M, Lecorre B et al (2018) Changes in emotions from childhood to young adulthood. Child Indic Res 11(2):541–561Google Scholar
  46. Mesquita B, Boiger M, De Leersnyder J (2017) Doing emotions: the role of culture in everyday emotions. Eur Rev Soc Psychol 28(1):95–133Google Scholar
  47. Morrison AS, Mateen MA, Brozovich FA, Zaki J, Goldin PR, Heimberg RG, Gross JJ (2016) Empathy for positive and negative emotions in social anxiety disorder. Behav Res Ther 87:232–242Google Scholar
  48. Nalepa GJ, Kutt K, Bobek S (2019) Mobile platform for affective context-aware systems. Future Gener Comput Syst 92:490–503Google Scholar
  49. Nass C, Takayama L, Brave S (2006) Socializing consistency: from technical homogeneity to human epitome. In: Zhang P, Galletta DF (eds) Human-computer interaction and management information systems: foundations. M. E. Sharpe, Armonk, NY, pp 373–391Google Scholar
  50. Oh K, Park HS, Cho SB (2010) A mobile context sharing system using activity and emotion recognition with bayesian networks. In: 7th international conference on ubiquitous intelligence & computing and 7th international conference on autonomic & trusted computing, Xian, Shaanxi, pp 244–249Google Scholar
  51. Ortony A, Clore GL, Collins A (1990) The cognitive structure of emotions. Cambridge University Press, CambridgeGoogle Scholar
  52. Panda R, Malheiro RM, Paiva RP (2019) Novel audio features for music emotion recognition. IEEE Trans Affect Comput 1:1–1Google Scholar
  53. Pekrun R, Vogl E, Muis KR, Sinatra GM (2017) Measuring emotions during epistemic activities: the epistemically-related emotion scales. Cogn Emot 31(6):1268–1276Google Scholar
  54. Phelps EA (2006) Emotion and cognition: insights from studies of the human amygdala. Annu Rev Psychol 57:27–53Google Scholar
  55. Politou E, Alepis E, Patsakis C (2017) A survey on mobile affective computing. Comput Sci Rev 25:79–100Google Scholar
  56. Ptaszynski M, Rzepka R, Araki K (2010) On the need for context processing in affective computing. Proc Fuzzy Syst Sympos Jpn Soc Fuzzy Theory Intell Inform 26:920–924Google Scholar
  57. Rodríguez PM, Del Pino DA, Alvaredo RB (2011) De lo psicológico a lo fisiológico en la relación entre emociones y salud. Revista Psicología Científica 13(19):34–39Google Scholar
  58. Royet JP, Zald D, Versace R, Costes N, Lavenne F, Koenig O, Gervais R (2000) Emotional responses to pleasant and unpleasant olfactory, visual, and auditory stimuli: a positron emission tomography study. J Neurosci 20(20):7752–7759Google Scholar
  59. Salido Ortega MG, Rodriguez LF, Gutierrez-Garcia JO (2018) Energy-aware data collection from the internet of things for building emotional profiles. In: Third international conference on fog and mobile edge computing (FMEC), Barcelona, pp 234–239Google Scholar
  60. Sandstrom GM, Lathia N, Mascolo C, Rentfrow PJ (2017) Putting mood in context: using smartphones to examine how people feel in different locations. J Res Pers 69:96–101Google Scholar
  61. Seo J, Laine TH, Sohn KA (2019) Machine learning approaches for boredom classification using eeg. J Ambient Intell Human Comput.  https://doi.org/10.1007/s12652-019-01196-3 Google Scholar
  62. Sheldon KM (1994) Emotionality differences between artists and scientists. J Res Pers 28(4):481–491Google Scholar
  63. Soleimaninejadian P, Zhang M, Liu Y, Ma S (2018) Mood detection and prediction based on user daily activities. In: First Asian conference on affective computing and intelligent interaction (ACII Asia), Beijing, pp 1–6Google Scholar
  64. Sprenkle DH, Piercy FP (2005) Pluralism, diversity, and sophistication in family therapy research. Res Methods Fam Therapy 2:3–18Google Scholar
  65. Stone AA, Schwartz JE, Schkade D, Schwarz N, Krueger A, Kahneman D (2006) A population approach to the study of emotion: diurnal rhythms of a working day examined with the day reconstruction method. Emotion 6(1):139Google Scholar
  66. Swets JA (1988) Measuring the accuracy of diagnostic systems. Science 240(4857):1285–1293MathSciNetzbMATHGoogle Scholar
  67. Tagar MR, Federico CM, Halperin E (2011) The positive effect of negative emotions in protracted conflict: the case of anger. J Exp Soc Psychol 47(1):157–164Google Scholar
  68. Tangney JP, Wagner P, Fletcher C, Gramzow R (1992) Shamed into anger? the relation of shame and guilt to anger and self-reported aggression. J Pers Soc Psychol 62(4):669Google Scholar
  69. Wegrzyn M, Vogt M, Kireclioglu B, Schneider J, Kissler J (2017) Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS One 12(5):1–15Google Scholar
  70. Wharton AS, Erickson RI (1993) Managing emotions on the job and at home: understanding the consequences of multiple emotional roles. Acad Manag Rev 18(3):457–486Google Scholar
  71. Wingenbach TS, Ashwin C, Brosnan M (2018) Sex differences in facial emotion recognition across varying expression intensity levels from videos. PLoS One 13(1):1–18Google Scholar
  72. Witten IH, Frank E, Hall MA, Pal CJ (2016) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, BurlingtonGoogle Scholar
  73. Yin Z, Zhao M, Wang Y, Yang J, Zhang J (2017) Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Comput Methods Programs Biomed 140:93–110Google Scholar
  74. Zhang Y, Tang J, Sun J, Chen Y, Rao J (2010) Moodcast: emotion prediction via dynamic continuous factor graph model. In: IEEE international conference on data mining, Sydney, NSW, pp 1193–1198Google Scholar
  75. Zhang X, Li W, Chen X, Lu S (2018) Moodexplorer: towards compound emotion detection via smartphone sensing. Proc ACM Interact Mobile Wear Ubiquitous Technol 1(4):176:1–176:30Google Scholar
  76. Zualkernan I, Aloul F, Shapsough S, Hesham A, El-Khorzaty Y (2017) Emotion recognition using mobile phones. Comput Electr Eng 60:1–13Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Instituto Tecnológico de SonoraCd. ObregónMexico
  2. 2.ITAMCiudad de MéxicoMexico

Personalised recommendations