Advertisement

Machine learning approaches for boredom classification using EEG

  • Jungryul Seo
  • Teemu H. Laine
  • Kyung-Ah Sohn
Original Research
  • 13 Downloads

Abstract

Recently, commercial physiological sensors and computing devices have become cheaper and more accessible, while computer systems have become increasingly aware of their contexts, including but not limited to users’ emotions. Consequently, many studies on emotion recognition have been conducted. However, boredom has received relatively little attention as a target emotion due to its diverse nature. Moreover, only a few researchers have tried classifying boredom using electroencephalogram (EEG). In this study, to perform this classification, we first reviewed studies that tried classifying emotions using EEG. Further, we designed and executed an experiment, which used a video stimulus to evoke boredom and non-boredom, and collected EEG data from 28 Korean adult participants. After collecting the data, we extracted its absolute band power, normalized absolute band power, differential entropy, differential asymmetry, and rational asymmetry using EEG, and trained these on three machine learning algorithms: support vector machine, random forest, and k-nearest neighbors (k-NN). We validated the performance of each training model with 10-fold cross validation. As a result, we achieved the highest accuracy of 86.73% using k-NN. The findings of this study can be of interest to researchers working on emotion recognition, physiological signal processing, machine learning, and emotion-aware system development.

Keywords

Boredom EEG Machine learning Classification Emotion Sensor 

Notes

Acknowledgements

This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2018-2018-0-01431) supervised by the IITP (Institute for Information & communications Technology Promotion). We would like to show our appreciation to the participants in our data collection. Special thanks to Ms. Wooryeon Go (Emily), Dr. Carolina Islas Sedano, Dr. Hana Vrzakova, Dr. Roman Bednarik, and Dr. Bednarik’s research group members at the University of Eastern Finland. Furthermore, we would like to thank Mr. Joochan Kim, at the Luleå University of Technology in Sweden.

References

  1. Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Mach Learn 6(1):37–66.  https://doi.org/10.1007/BF00153759 Google Scholar
  2. van der Akker J, Keursten P, Plomp T (1992) The integration of computer use in education. Int J Educ Res 17(1):65–76Google Scholar
  3. Aksoy S, Haralick RM (2001) Feature normalization and likelihood-based similarity measures for image retrieval. Pattern Recogn Lett 22(5):563–582zbMATHGoogle Scholar
  4. Alm CO, Roth D, Sproat R (2005) Emotions from text: machine learning for text-based emotion prediction. In: Proceedings of the conference on human language technology and Empirical methods in natural language processing, association for computational Linguistics, Stroudsburg, PA, USA, HLT ’05, pp 579–586,  https://doi.org/10.3115/1220575.1220648
  5. Altman NS (1992) An introduction to kernel and nearest-neighbor nonparametric regression. Am Stat 46(3):175–185.  https://doi.org/10.1080/00031305.1992.10475879 MathSciNetGoogle Scholar
  6. Baker R, D’Mello S, Rodrigo M, Graesser A (2010) Better to be frustrated than bored: the incidence and persistence of affect during interactions with three different computer-based learning environments. Int J Hum Comput Stud 68(4):223–241Google Scholar
  7. Barandiaran I (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20(8):56Google Scholar
  8. Baumgartner T, Esslen M, Jäncke L (2006) From emotion perception to emotion experience: emotions evoked by pictures and classical music. Int J Psychophysiol 60(1):34–43.  https://doi.org/10.1016/j.ijpsycho.2005.04.007 Google Scholar
  9. Bechara A, Damasio H, Damasio AR (2000) Emotion, decision making and the orbitofrontal cortex. Cerebral Cortex 10(3):295–307.  https://doi.org/10.1093/cercor/10.3.295 Google Scholar
  10. Bednarik R, Vrzakova H, Hradis M (2012) What do you want to do next: a novel approach for intent prediction in gaze-based interaction. In: Proceedings of the symposium on eye tracking research and applications, ACM, pp 83–90Google Scholar
  11. Bereiter C (2002) Education and mind in the knowledge age. Routledge, AbingdonGoogle Scholar
  12. Bruce BC, Rubin A (1993) Electronic quills a situated evaluation of using computers for writing in classrooms. Routledge, AbingdonGoogle Scholar
  13. Busso C, Deng Z, Yildirim S, Bulut M, Lee CM, Kazemzadeh A, Lee S, Neumann U, Narayanan S (2004) Analysis of emotion recognition using facial expressions, speech and multimodal information. In: Proceedings of the 6th international conference on Multimodal interfaces—ICMI ’04,  https://doi.org/10.1145/1027933.1027968,arXiv:1011.1669v3
  14. Castellano G, Kessous L, Caridakis G (2008) Emotion recognition through multiple modalities: face, body gesture, speech. Affect and emotion in human-computer interaction. Springer, Berlin, pp 92–103Google Scholar
  15. Chanel G, Kronegg J, Grandjean D, Pun T (2006) Emotion assessment: arousal evaluation using EEG’s and peripheral physiological signals. In: Gunsel B, Jain AK, Tekalp AM, Sankur B (eds) Multimedia content representation, classification and security. MRCS 2006. Lecture notes in computer science, vol 4105. Springer, Berlin, HeidelbergGoogle Scholar
  16. Chi TS, Yeh LY, Hsu CC (2012) Robust emotion recognition by spectro-temporal modulation statistic features. J Ambient Intell Hum Comput 3(1):47–60.  https://doi.org/10.1007/s12652-011-0088-5 Google Scholar
  17. Chuang ZJ, Wu Ch (2004) Multi-modal emotion recognition from speech and text. J Comput Linguist Chin Lang Process 9(2):45–62Google Scholar
  18. Cole M (1998) Cultural psychology: a once and future discipline. Harvard University Press, CambridgeGoogle Scholar
  19. Davidson RJ (1992) Anterior cerebral asymmetry and the nature of emotion. Brain Cogn 20(1):125–151Google Scholar
  20. Davidson RJ, Fox NA (1982) Asymmetrical brain activity discriminates between positive and negative affective stimuli in human infants. Science 218(4578):1235–1237Google Scholar
  21. Eastwood JD, Frischen A, Fenske MJ, Smilek D (2012) The unengaged mind defining boredom in terms of attention. Perspect Psychol Sci 7(5):482–495.  https://doi.org/10.1177/1745691612456044 Google Scholar
  22. Fagerberg P, Ståhl A, Höök K (2004) EMoto: emotionally engaging interaction. Pers Ubiquitous Comput 8(5):377–381Google Scholar
  23. Fahlman SA, Mercer-Lynn KB, Flora DB, Eastwood JD (2013) Development and validation of the multidimensional state boredom scale. Assessment 20(1):68–85.  https://doi.org/10.1177/1073191111421303 Google Scholar
  24. Feldman L (1995) Variations in the circumplex structure of mood. Personal Soc Psychol Bull 21(8):806–817Google Scholar
  25. Giakoumis D, Vogiannou A, Kosunen I, Moustakas K, Tzovaras D, Hassapis G (2010) Identifying psychophysiological correlates of boredom and negative mood induced during HCI. In: Bio-inspired human-machine interfaces and healthcare applications, January 2010, pp 3–12,  https://doi.org/10.5220/0002812600030012
  26. Giakoumis D, Tzovaras D, Moustakas K, Hassapis G (2011) Automatic recognition of boredom in video games using novel biosignal moment-based features. IEEE Trans Affect Comput 2(3):119–133.  https://doi.org/10.1109/T-AFFC.2011.4 Google Scholar
  27. Glowinski D, Camurri A, Volpe G, Dael N, Scherer K (2008) Technique for automatic emotion recognition by body gesture analysis. In: 2008 IEEE computer society conference on computer vision and pattern recognition workshops, CVPR Workshops pp 1–6,  https://doi.org/10.1109/CVPRW.2008.4563173
  28. Gruber S, Peyton JK, Bruce BC (1994) Collaborative writing in multiple discourse contexts. CSCW 3(3–4):247–269.  https://doi.org/10.1007/BF00750742 Google Scholar
  29. Gunes H, Piccardi M (2007) Bi-modal emotion recognition from expressive face and body gestures. J Netw Comput Appl 30(4):1334–1345.  https://doi.org/10.1016/j.jnca.2006.09.007 Google Scholar
  30. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. ACM SIGKDD Explor Newslett 11(1):10–18Google Scholar
  31. Ho TK (1995) Random decision forests. In: Document analysis and recognition, 1995, proceedings of the third international conference on, IEEE, vol 1, pp 278–282Google Scholar
  32. Horlings R, Datcu D, Rothkrantz LJM (2008) Emotion recognition using brain activity. In: Proceedings of the 9th international conference on computer systems and technologies and workshop for PhD students in computing, ACM, March, p 6Google Scholar
  33. Jang EH, Park BJ, Park MS, Kim SH, Sohn JH (2015) Analysis of physiological signals for recognition of boredom, pain, and surprise emotions. J Physiol Anthropol 34(1):1–12.  https://doi.org/10.1186/s40101-015-0063-5 Google Scholar
  34. Jaques N, Conati C, Harley JM, Azevedo R (2014) Predicting affect from gaze data during interaction with an intelligent tutoring system. Lect Notes Comput Sci 8474:29–38Google Scholar
  35. Jasper H (1958) Report of the committee on methods of clinical examination in electroencephalography: 1957. Electroencephalogr Clin Neurophysiol 10(2):370–375.  https://doi.org/10.1016/0013-4694(58)90053-1 Google Scholar
  36. Jiang R, Ho AT, Cheheb I, Al-Maadeed N, Al-Maadeed S, Bouridane A (2017) Emotion recognition from scrambled facial images via many graph embedding. Pattern Recogn 67:245–251.  https://doi.org/10.1016/j.patcog.2017.02.003 Google Scholar
  37. Kanevsky LS (1994) A comparative study of children’s learning in the zone of proximal development. Eur J High Abil 5(2):163–175.  https://doi.org/10.1080/0937445940050206 Google Scholar
  38. Katahira K, Yamazaki Y, Yamaoka C, Ozaki H, Nakagawa S, Nagata N (2018) Eeg correlates of the flow state: a combination of increased frontal theta and moderate frontocentral alpha rhythm in the mental arithmetic task. Front Psychol 9:300.  https://doi.org/10.3389/fpsyg.2018.00300 Google Scholar
  39. Khalili Z, Moradi MH (2008) Emotion detection using brain and peripheral signals. In: 2008 Cairo international biomedical engineering conference, pp 1–4,  https://doi.org/10.1109/CIBEC.2008.4786096
  40. Kim D, Kim Y, Kim S, Park Y, Park J, Bae K, Lee S, Lee J, Lim C, Jeon Y, Jin S, Chae J, Hwang H (2017) Understanding and application of EEG. Hakjisa, Seogyo-dongGoogle Scholar
  41. Kim DJ, Kim YS (2010) Pattern classification of four emotions using EEG. J Korea Inst Inf Electron Commun Technol 3:23–27Google Scholar
  42. Kim HD, Sim KB (2007) Brain-wave analysis using fMRI, TRS and EEG for human emotion recognition. In: Proceedings of KFIS autumn conference 2007, Korean Institute of Intelligent Systems, vol 17, pp 7–10Google Scholar
  43. Kim J, Seo J, Laine TH (2018) Detecting Boredom from Eye Gaze and EEG. Biomed Signal Process Control 46:302–313Google Scholar
  44. Lang PJ, Bradley MM, Cuthbert BN (2008) International affective picture system (IAPS): affective ratings of pictures and instruction manual, Technical Report. University of Florida, Gainesville, FloridaGoogle Scholar
  45. Lee CC, Shih CY, Lai WP, Lin PC (2012) An improved boosting algorithm and its application to facial emotion recognition. J Ambient Intell Hum Comput 3(1):11–17.  https://doi.org/10.1007/s12652-011-0085-8 Google Scholar
  46. Lee HJ, Shin DI, Shin DK (2013) A Study on an emotion-classification algorithm of users adapting Brainwave. In: Proceedings of symposium of the Korean Institute of communications and Information Sciences, Korea Institute Of Communication Sciences, pp 786–787Google Scholar
  47. Lee HJ, Shin DI, Shin DK (2014) The classification algorithm of users emotion using brain-wave. J Kor Inst Commun Inf Sci 39(02):122–129Google Scholar
  48. Lee S, Song K (2007) The effects of affective feedbacks according to the learner’s emotions in e-learning. J Kor Soc Comput Inf 10(4):56Google Scholar
  49. Li M, Lu BL (2009) Emotion classification based on gamma-band EEG. In: 2009 Annual International Conference of the IEEE Engineering in medicine and biology society pp 1223–1226,  https://doi.org/10.1109/IEMBS.2009.5334139
  50. Lin HCK, Hsieh MC, Loh LC, Wang CH (2012) An emotion recognition mechanism based on the combination of mutual information and semantic clues. J Ambient Intell Hum Comput 3(1):19–29.  https://doi.org/10.1007/s12652-011-0086-7 Google Scholar
  51. Lin YP, Wang CH, Wu TL, Jeng SK, Chen JH (2008) Support vector machine for EEG signal classification during listening to emotional music. In: 2008 IEEE 10th workshop on multimedia signal processing, pp 127–130,  https://doi.org/10.1109/MMSP.2008.4665061
  52. Lin YP, Wang CH, Wu TL, Jeng SK, Chen JH (2009) EEG-based emotion recognition in music listening: a comparison of schemes for multiclass support vector machine. In: 2009 IEEE international conference on Acoustics, speech and signal processing pp 489–492,  https://doi.org/10.1109/ICASSP.2009.4959627
  53. Lin YP, Wang CH, Jung TP, Wu TL, Jeng SK, Duann JR, Chen JH (2010) EEG-based emotion recognition in music listening. IEEE Trans Biomed Eng 57(7):1798–1806.  https://doi.org/10.1109/TBME.2010.2048568 Google Scholar
  54. Logeswaran N, Bhattacharya J (2009) Crossmodal transfer of emotion by music. Neurosci Lett 455(2):129–133.  https://doi.org/10.1016/j.neulet.2009.03.044 Google Scholar
  55. Mandryk RL, Atkins MS (2007) A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. Int J Hum Comput Stud 65(4):329–347Google Scholar
  56. Mao X, Li Z (2009) Implementing emotion-based user-aware e-learning. Int Conf Hum Factors Comput Syst 56:3787–3792Google Scholar
  57. Mistry K, Zhang L, Neoh SC, Lim CP, Fielding B (2017) A micro-GA embedded PSO feature selection approach to intelligent facial emotion recognition. IEEE Trans Cybern 47(6):1496–1509.  https://doi.org/10.1109/TCYB.2016.2549639 Google Scholar
  58. Mohammadpour M, Hashemi SMR, Houshmand N (2017) Classification of EEG-based emotion for BCI applications. In: 2017 artificial intelligence and robotics (IRANOPEN), pp 127–131,  https://doi.org/10.1109/RIOS.2017.7956455
  59. Murugappan M, Rizon M, Nagarajan R, Yaacob S, Zunaidi I, Hazry D (2008) Lifting scheme for human emotion recognition using EEG. In: 2008 International symposium on information technology 2:0–6,  https://doi.org/10.1109/ITSIM.2008.4631646
  60. Nicolopoulou A, Cole M (1996) Generation and transmission of shared knowledge in the culture of collaborative learning: the Fifth Dimension, its play-world and its institutional contexts. Oxford University Press, OxfordGoogle Scholar
  61. Nie D, Wang XW, Shi LC, Lu BL (2011) EEG-based emotion recognition during watching movies. In: 2011 5th International IEEE/EMBS conference on neural engineering, pp 667–670,  https://doi.org/10.1109/NER.2011.5910636
  62. Nwe TL, Foo SW, De Silva LC (2003) Speech emotion recognition using hidden Markov models. Speech Commun 41(4):603–623.  https://doi.org/10.1016/S0167-6393(03)00099-2 Google Scholar
  63. Oostenveld R, Praamstra P (2001) The five percent electrode system for high-resolution eeg and erp measurements. Clin Neurophysiol 112(4):713–719.  https://doi.org/10.1016/S1388-2457(00)00527-7 Google Scholar
  64. Oroujlou N, Vahedi M (2011) Motivation, attitude, and language learning. Procedia Soc Beh Sci 29:994–1000.  https://doi.org/10.1016/j.sbspro.2011.11.333 Google Scholar
  65. Petrantonakis PC, Hadjileontiadis LJ (2010) Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis. IEEE Trans Affect Comput 1(2):81–97.  https://doi.org/10.1109/T-AFFC.2010.7 Google Scholar
  66. Petrantonakis PC, Hadjileontiadis LJ (2011) A novel emotion elicitation index using frontal brain asymmetry for enhanced EEG-Based emotion recognition. IEEE Trans Inf Technol Biomed 15(5):737–746.  https://doi.org/10.1109/TITB.2011.2157933 Google Scholar
  67. Picard RW (1995) Affective computing. Tech Rep 5:6–12.  https://doi.org/10.1007/BF01238028 Google Scholar
  68. Rosenblatt F (1961) Principles of neurodynamics perceptrons and the theory of brain mechanisms. Cornell aeronautical lab, BuffalozbMATHGoogle Scholar
  69. Russell JA (1980) A circumplex model of affect. J Personal Soc Psychol 39(6):1161–1178.  https://doi.org/10.1037/h0077714 Google Scholar
  70. Sanei S, Chambers JA (2013) EEG signal processing. Wiley, HobokenGoogle Scholar
  71. Scardamalia M, Bereiter C (1994) Computer support for knowledge-building communities. J Learn Sci 3(3):265–283.  https://doi.org/10.1207/s15327809jls0303_3 Google Scholar
  72. Schmidt LA, Trainor LJ (2001) Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cogn Emot 15(4):487–500.  https://doi.org/10.1080/02699930126048 Google Scholar
  73. Schuller B, Rigoll G, Lang M (2003) Hidden markov model-based speech emotion recognition. In: 2003 IEEE international conference on Acoustics, speech, and signal processing, 2003. Proceedings. (ICASSP ’03)., vol 2, pp II–1,  https://doi.org/10.1109/ICASSP.2003.1202279
  74. Shen L, Leon E, Callaghan V, Shen R (2007) Exploratory research on an Affective e-Learning Model. In: Proceedings of workshop on blended learning, vol 267-278, 10.1007/978-3-540-78139-4, 9780201398298Google Scholar
  75. Shen L, Wang M, Shen R (2009) Affective e-learning: using emotional data to improve learning in pervasive learning environment related work and the pervasive e-learning platform. Educ Technol Soc 12(2):176–189Google Scholar
  76. Sidney KD, Craig SD, Gholson B, Franklin S, Picard R, Graesser AC (2005) Integrating affect sensors in an intelligent tutoring system. In: In affective interactions: the computer in the affective loop workshop at 2005 international conference on intelligent user interfaces, pp 7–13Google Scholar
  77. Singh G, Jati A, Khasnobish A, Bhattacharyya S, Konar A, Tibarewala DN, Janarthanan R (2012) Negative emotion recognition from stimulated EEG signals. In: Computing communication networking technologies (ICCCNT), 2012 third international conference on, pp 1–8,  https://doi.org/10.1109/ICCCNT.2012.6395891
  78. Sottilare R, Goldberg B (2012) Designing adaptive computer-based tutoring systems to accelerate learning and facilitate retention. J Cogn Technol 17(1):19–33Google Scholar
  79. Takahashi K (2005) Remarks on emotion recognition from bio-potential signals. In: in 2nd Int. Conf. on Autonomous Robots and AgentsGoogle Scholar
  80. Tan JW, Walter S, Scheck A, Hrabal D, Hoffmann H, Kessler H, Traue HC (2012) Repeatability of facial electromyography (EMG) activity over corrugator supercilii and zygomaticus major on differentiating various emotions. J Ambient Intell Hum Comput 3(1):3–10.  https://doi.org/10.1007/s12652-011-0084-9 Google Scholar
  81. Timmermann M, Jeung H, Schmitt R, Boll S, Freitag CM, Bertsch K, Herpertz SC (2017) Oxytocin improves facial emotion recognition in young adults with antisocial personality disorder. Psychoneuroendocrinology 85(July):158–164.  https://doi.org/10.1016/j.psyneuen.2017.07.483 Google Scholar
  82. Vijayan AE, Sen D, Sudheer A (2015) EEG-Based Emotion Recognition Using Statistical Measures and Auto-Regressive Modeling. In: 2015 IEEE international conference on computational intelligence and communication technology, pp 587–591,  https://doi.org/10.1109/CICT.2015.24
  83. Vogel-Walcutt JJ, Fiorella L, Carper T, Schatz S (2012) The definition, assessment, and mitigation of state boredom within educational settings: a comprehensive review. Educ Psychol Rev 24(1):89–111.  https://doi.org/10.1007/s10648-011-9182-7 Google Scholar
  84. Wu CH, Chuang ZJ, Lin YC (2006) Emotion recognition from text using semantic labels and separable mixture models. ACM Trans Asian Lang Inf Process 5(2):165–183.  https://doi.org/10.1145/1165255.1165259 Google Scholar
  85. Wundt W (1897) Outline of psychology (CH Judd, Trans.). Williams and Norgate, London, Great Britain (Wilhelm Engelmann, Leipzig, Germany) Google Scholar
  86. Yeager DS, Henderson MD, Paunesku D, Walton GM, D'Mello S, Spitzer BJ, Duckworth AL (2014) Boring but important: a self-transcendent purpose for learning fosters academic self-regulation. J Pers Soc Psychol 107(4):559Google Scholar
  87. Zhao Y, Wang X, Goubran M, Whalen T, Petriu EM (2013) Human emotion and cognition recognition from body language of the head using soft computing techniques. J Ambient Intell Hum Comput 4(1):121–140.  https://doi.org/10.1007/s12652-012-0107-1 Google Scholar
  88. Zheng WL, Lu BL (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7(3):162–175.  https://doi.org/10.1109/TAMD.2015.2431497 Google Scholar
  89. Zheng WL, Zhu JY, Lu BL (2017) Identifying stable patterns over time for emotion recognition from EEG. In: IEEE transactions on affective computing PP(99):1,  https://doi.org/10.1109/TAFFC.2017.2712143

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer EngineeringAjou UniversitySuwonRepublic of Korea
  2. 2.Department of Computer Science, Electrical and Space EngineeringLuleå University of TechnologySkellefteåSweden

Personalised recommendations