Emotion Understanding Using Multimodal Information Based on Autobiographical Memories for Alzheimer’s Patients

  • Juan Manuel Fernandez MontenegroEmail author
  • Athanasios Gkelias
  • Vasileios Argyriou
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10116)


Alzheimer Disease (AD) early detection is considered of high importance for improving the quality of life of patients and their families. Amongst all the different approaches for AD detection, significant work has been focused on emotion analysis through facial expressions, body language or speech. Many studies also use the electroencephalogram in order to capture emotions that patients cannot physically express. Our work introduces an emotion recognition approach using facial expression and EEG signal analysis. A novel dataset was created specifically to remark the autobiographical memory deficits of AD patients. This work uses novel EEG features based on quaternions, facial landmarks and the combination of them. Their performance was evaluated in a comparative study with a state of the art methods that demonstrates the proposed approach.


Support Vector Machine Facial Expression Emotion Recognition Autobiographical Memory Alzheimer Disease Patient 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Supplementary material

426013_1_En_17_MOESM1_ESM.avi (12.1 mb)
Supplementary material 1 (avi 12433 KB)


  1. 1.
    Alpher, A.: Advances in frobnication. J. Foo 12, 234–778 (2002)Google Scholar
  2. 2.
    Alpher, A., Fotheringham-Smythe, J.P.N.: Frobnication revisited. J. Foo 13, 234–778 (2003)CrossRefGoogle Scholar
  3. 3.
    Herman, S., Fotheringham-Smythe, J.P.N., Gamow, G.: Can a machine frobnicate? J. Foo 14, 234–778 (2004)Google Scholar
  4. 4.
    Smith, F.: The Frobnicatable Foo Filter. GreatBooks, Atown (2009)Google Scholar
  5. 5.
    Wills, H.: Frobnication tutorial. Technical report CS-1204, XYZ University, Btown (1999)Google Scholar
  6. 6.
    Rosler, A., Mapstone, M.E., Hays, A.K., Mesulam, M., Rademaker, A., Gitelman, D.R., Weintraub, S.: Alterations of visual search strategy in Alzheimer’s disease and aging. Neuropsychology 14(3), 398–408 (2000)CrossRefGoogle Scholar
  7. 7.
    Pereira, M.L., Camargo, M.V.Z.A., Aprahamian, I., Forlenza, O.V.: Eye movement analysis and cognitive processing: detecting indicators of conversion to Alzheimer’s disease. Neuropsychiatric Dis. Treat. 10, 1273–1285 (2014)CrossRefGoogle Scholar
  8. 8.
    Alpher, A.: Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015)CrossRefGoogle Scholar
  9. 9.
    Koelstra, S., Muehl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012)CrossRefGoogle Scholar
  10. 10.
    Sapey-Triomphe, L.A., Heckemann, R.A., Boublay, N., Dorey, J.M., Hnaff, M.A., Rouch, I., Padovan, C.: Neuroanatomical correlates of recognizing face expressions in mild stages of Alzheimers disease. PLoS ONE 10(12) (2015)Google Scholar
  11. 11.
    Van den Stock, J., De Winter, F.L., de Gelder, B., Rangarajan, J.R., Cypers, G., Maes, F., Sunaert, S., Goffin, K., Vandenberghe, R., Vandenbulcke, M.: Impaired recognition of body expressions in the behavioral variant of frontotemporal dementia. Neuropsychologia 75, 496–504 (2015)CrossRefGoogle Scholar
  12. 12.
    Han, K.H., Zaytseva, Y., Bao, Y., Pppel, E., Chung, S.Y., Kim, J.W., Kim, H.T.: Impairment of vocal expression of negative emotions in patients with Alzheimers disease. Front. Aging Neurosci. 6(101), 1–6 (2014)Google Scholar
  13. 13.
    Irish, M., Hornberger, M., Lah, S., Miller, L., Pengas, G., Nestor, P.J., Hodges, J.R., Piguet, O.: Profiles of recent autobiographical memory retrieval in semantic dementia, behavioural-variant frontotemporal dementia, and Alzheimers disease. Neuropsychologia 49(9), 2694–2702 (2011)CrossRefGoogle Scholar
  14. 14.
    Seidl, U., Lueken, U., Thomann, P.A., Kruse, A., Schrder, J.: Facial expression in Alzheimers disease impact of cognitive deficits and neuropsychiatric symptoms. Am. J. Alzheimer’s Dis. Other Dementias 27(2), 100–106 (2012)CrossRefGoogle Scholar
  15. 15.
    American Psychiatric Association: Diagnostic and statistical manual of mental disorders (DSM-5). American Psychiatric Association Publishing (2013)Google Scholar
  16. 16.
    Bettadapura, V.: Face expression recognition and analysis: the state of the art. Tech. Report, pp. 1–27. arXiv:1203.6722 (2012)
  17. 17.
    Ekman, P., Friesen, W.V.: The Facial Action Coding System: A Technique for The Measurement of Facial Movement. Consulting Psychologists Press, San Francisco (1978)Google Scholar
  18. 18.
    Weninger, F., Wllmer, M., Schuller, B.: Emotion recognition in naturalistic speech and language a survey. In: Emotion Recognition: A Pattern Analysis Approach, pp. 237–267 (2015)Google Scholar
  19. 19.
    Chowdhuri, M.A.D., Bojewar, S.: Emotion detection analysis through tone of user: a survey. Emotion 5(5), 859–861 (2016)Google Scholar
  20. 20.
    Soleymani, M., Asghari-Esfeden, S., Fu, Y., Pantic, M.: Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans. Affect. Comput. 7(1), 17–28 (2016)CrossRefGoogle Scholar
  21. 21.
    Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012)CrossRefGoogle Scholar
  22. 22.
    Nicolaou, M.A., Gunes, H., Pantic, M.: Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Trans. Affect. Comput. 2(2), 92–105 (2011)CrossRefGoogle Scholar
  23. 23.
    McKeown, G., Valstar, M., Cowie, R., Pantic, M., Schroder, M.: The semaine database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Trans. Affect. Comput. 3(1), 5–17 (2012)CrossRefGoogle Scholar
  24. 24.
    Yan, W.J., Li, X., Wang, S.J., Zhao, G., Liu, Y.J., Chen, Y.H., Fu, X.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), e86041 (2014)Google Scholar
  25. 25.
    Lokannavar, S., Lahane, P., Gangurde, A., Chidre, P.: Emotion recognition using EEG signals. Emotion 4(5), 54–56 (2015)Google Scholar
  26. 26.
    Vijayan, A.E., Sen, D., Sudheer, A.P.: EEG-based emotion recognition using statistical measures and auto-regressive modeling. In: IEEE International Conference on Computational Intelligence and Communication Technology (CICT), vol. 14(1), pp. 587–591 (2015)Google Scholar
  27. 27.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRefGoogle Scholar
  28. 28.
    Baltru, T., Robinson, P., Morency, L.P.: OpenFace: an open source facial behavior analysis toolkit. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10 (2016)Google Scholar
  29. 29.
    Sohaib, A.T., Qureshi, S., Hagelbäck, J., Hilborn, O., Jerčić, P.: Evaluating classifiers for emotion recognition using EEG. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2013. LNCS (LNAI), vol. 8027, pp. 492–501. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-39454-6_53 CrossRefGoogle Scholar
  30. 30.
    Müller-Putz, G.R., Riedl, R., Wriessnegger, S.C.: Electroencephalography (EEG) as a research tool in the information systems discipline: foundations, measurement, and applications. Commun. Assoc. Inform. Syst. 37(46), 911–948 (2015)Google Scholar
  31. 31.
    Petrantonakis, P.C., Hadjileontiadis, L.J.: Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis. IEEE Trans. Affect. Comput. 1, 81–97 (2010)CrossRefGoogle Scholar
  32. 32.
    Valstar, M.F., Patras, I., Pantic, M.: Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 3, pp. 76–84 (2005)Google Scholar
  33. 33.
    Michel, P., El Kaliouby, R.: Real time facial expression recognition in video using support vector machines. In: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 258–264 (2003)Google Scholar
  34. 34.
    Hamilton, W.R.: On quaternions, or on a new system of imaginaries in algebra. Philos. Mag. 25(3), 489–495 (1844)Google Scholar
  35. 35.
    Chen, M., Meng, X., Wang, Z.: Quaternion fisher discriminant analysis for bimodal multi-feature fusion. In: Abraham, A., Jiang, X.H., Snášel, V., Pan, J.-S. (eds.) Intelligent Data Analysis and Applications. AISC, vol. 370, pp. 479–487. Springer, Cham (2015). doi: 10.1007/978-3-319-21206-7_41 CrossRefGoogle Scholar
  36. 36.
    Le Bihan, N., Sangwine, S.J.: Quaternion principal component analysis of color images. In: International Conference on Image Processing (ICIP), vol. 1, pp. I-809–I-812 (2003)Google Scholar
  37. 37.
    Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 46–53 (2000)Google Scholar
  38. 38.
    Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: IEEE International Conference on Multimedia and Expo, pp. 317–321 (2005)Google Scholar
  39. 39.
    Patras, I., Pantic, M.: Particle filtering with factorized likelihoods for tracking facial features. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 97–102 (2004)Google Scholar
  40. 40.
    Izard, C.E.: Human Emotions. Springer, New York (2013)Google Scholar
  41. 41.
    Nicolle, J., Rapp, V., Bailly, K., Prevost, L., Chetouani, M.: Robust continuous prediction of human emotions using multiscale dynamic cues. In: 14th ACM International Conference on Multimodal Interaction, pp. 501–508 (2012)Google Scholar
  42. 42.
    Littlewort, G.C., Bartlett, M.S., Lee, K.: Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis. Comput. 27(12), 1797–1803 (2009)CrossRefGoogle Scholar
  43. 43.
    Huang, K.C., Huang, S.Y., Kuo, Y.H.: Emotion recognition based on a novel triangular facial feature extraction method. In: 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1–6(2010)Google Scholar
  44. 44.
    Sariyanidi, E., Gunes, H., Gkmen, M., Cavallaro, A.: Local zernike moment representation for facial affect recognition. In: British Machine Vision Conference (2013)Google Scholar
  45. 45.
    Zhao, G., Pietikinen, M.: Boosted multi-resolution spatiotemporal descriptors for facial expression recognition. Pattern Recogn. Lett. 30(12), 1117–1127 (2009)CrossRefGoogle Scholar
  46. 46.
    Wllmer, M., Eyben, F., Reiter, S., Schuller, B., Cox, C., Douglas-Cowie, E., Cowie, R.: Abandoning emotion classes-towards continuous emotion recognition with modelling of long-range dependencies. Interspeech 12, 597–600 (2008)Google Scholar
  47. 47.
    Adali, T., Schreier, P.J., Scharf, L.L.: Complex-valued signal processing: the proper way to deal with impropriety. IEEE Trans. Sig. Process. (Overview Pap.) 59(11), 5101–5123 (2011)MathSciNetCrossRefGoogle Scholar
  48. 48.
    Li, X.-L., Adali, T., Anderson, M.: Noncircular principal component analysis and its application to model selection. IEEE Sig. Proc. 59(10), pp. 4516i–4528i (2011)Google Scholar
  49. 49.
    Chai, Z., Ma, K.K., Liu, Z.: Complex wavelet-based face recognition using independent component analysis. In: Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Proceedings, pp. 832–835 (2009)Google Scholar
  50. 50.
    Bonita, J.D., Ambolode, II., L.C.C., Rosenberg, B.M., Cellucci, C.J., Watanabe, T.A.A., Rapp, P.E., Albano, A.M.: Time domain measures of inter-channel EEG correlations: a comparison of linear, nonparametric and nonlinear measures. Cogn. Neurodyn. 8(1), 1–15 (2014)Google Scholar
  51. 51.
    Li, K., Sun, G., Zhang, B., Wu, S., Wu, G.: Correlation between forehead EEG and sensorimotor area EEG in motor imagery task. In: Eighth IEEE International Conference on Dependable, Autonomic and Secure Computing, DASC 2009, pp. 430–435 (2009)Google Scholar
  52. 52.
    Abadi, M.K., Subramanian, R., Kia, S.M., Avesani, P., Patras, I., Sebe, N.: DECAF: MEG-based multimodal database for decoding affective physiological responses. IEEE Trans. Affect. Comput. 6(3), 209–222 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Juan Manuel Fernandez Montenegro
    • 1
    Email author
  • Athanasios Gkelias
    • 2
  • Vasileios Argyriou
    • 1
  1. 1.Kingston UniversityKingston upon ThamesEngland
  2. 2.Imperial CollegeLondonEngland

Personalised recommendations