Classification of Emotions from Video Based Cardiac Pulse Estimation

  • Keya DasEmail author
  • Antony Lam
  • Hisato Fukuda
  • Yoshinori Kobayashi
  • Yoshinori Kuno
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10956)


Recognizing emotion from video is an active research theme with many applications such as human-computer interaction and affective computing. The classification of emotions from facial expression is a common approach but it is sometimes difficult to differentiate genuine emotions from faked emotions. In this paper, we use a remote video based cardiac activity sensing technique to obtain physiological data to identify emotional states. We show that from the remotely sensed cardiac pulse patterns alone, emotional states can be differentiated. Specifically, we conducted an experimental study on recognizing the emotions of people watching video clips. We recorded 26 subjects that all watched the same comedy and horror video clips and then we estimated their cardiac pulse signals from the video footage. From the cardiac pulse signal alone, we were able to classify whether the subjects were watching the comedy or horror video clip. We also compare against classifying for the same task using facial action units and discuss how the two modalities compare.


Video PPG Cardiac pulse Facial action units Emotion recognition Physiological signal processing 



This work was supported by JSPS KAKENHI Grant Numbers JP17K12709, JP17K18850 and the Tateisi and Technology Foundation.


  1. 1.
    Cohen, I., Sebe, N., Garg, A., Chen, L.S., Huang, T.S.: Facial expression recognition from video sequences. In: Proceedings of the IEEE International Conference on Multimedia and Expo, vol. 2, pp. 121–124 (2002)Google Scholar
  2. 2.
    Zhang, S., Zhao, X., Lei, B.: Facial Expression recognition based on local binary patters and local fisher discriminant analysis. In: PMC (2011)Google Scholar
  3. 3.
    Kim, K.H., Bang, S.W., Kim, S.R.: Emotion recognition system using short-term monitoring of physiological signals. Med. Biol. Eng. Comput. 42, 419–427 (2004)CrossRefGoogle Scholar
  4. 4.
    Zong, C., Chetouani, M.: Hilbert-Huang transform based physiological signals analysis for emotion recognition. In: Proceedings of the IEEE International Symposium on Signal Processing and Information Technology, pp. 334–339 (2009)Google Scholar
  5. 5.
    Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. Proc. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1176–1189 (2001)Google Scholar
  6. 6.
    Chanel, G., Kierkels, J.J.M., Soleymani, M., Pun, T.: Short-term emotion assessment in a recall paradigm. Int. J. Hum. Comput. Stud. 67, 607–627 (2009)CrossRefGoogle Scholar
  7. 7.
    Li, X., Chen, J., Zhao, G., Pietkainen, M.: Remote heart rate measurement from face videos under realistic situations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271 (2014)Google Scholar
  8. 8.
    Monkaresi, H., Sazzad, M., Calvo, R.A.: Using remote heart rate measurement for affect detection. In: The Twenty-Seventh International Flairs Conference of the Florida Artificial Intelligence Research Society Conference, pp. 119–123 (2014)Google Scholar
  9. 9.
    Wu, H.Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.T.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31(4), 1–8 (2012)CrossRefGoogle Scholar
  10. 10.
    Kwon, S., Kim, H., Park, K.S.: Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone. In: Proceedings of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2174–2177, August 2012Google Scholar
  11. 11.
    Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16, 21434–21445 (2008)CrossRefGoogle Scholar
  12. 12.
    Lam, A., Kuno, Y.: Robust heart rate measurement from video using select random patches. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3640–3648 (2015)Google Scholar
  13. 13.
    Poh, M.Z., McDuff, D., Picard, R.: Advancements in noncontact, multiparameter physiological measurements using a webcam. Proc. IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011)CrossRefGoogle Scholar
  14. 14.
    Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., Sebe, N.: Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 (2016)Google Scholar
  15. 15.
    Kaewkannate, K., Kim, S.: A comparison of wearable fitness devices (2016).
  16. 16.
    Evenson, K.R., Goto, M., Furberg, R.D.: Systematic review of the validity and reliability of consumer-wearable activity trackers. Int. J. Behav. Nutr. Phys. Act. 12, 159 (2015)CrossRefGoogle Scholar
  17. 17.
    Balakrishnan, G., Durand, F., Guttag, J.: Detecting pulse from head motions in video. In: Proceedings of the IEEE Computer Vision and Pattern Recognition, CVPR 2013 (2013)Google Scholar
  18. 18.
    Chakraborty, P.R., Zhang, L., Tjondronegoro, D., Chandra, V.: Using viewer’s facial expression and heart rate for sports video highlights detection, pp. 371–378. ACM (2015)Google Scholar
  19. 19.
    Amos, B., Ludwiczuk, B., Satyanarayanan, M.: OpenFace: a general-purpose face recognition library with mobile applications. CMU-CS-16-118, CMU School of Computer Science, Technical report (2016)Google Scholar
  20. 20.
    Ekman, P., Rosenberg, E.L.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, New York (1997)Google Scholar
  21. 21.
    Lam, A., Otsu, K., Das, K., Kuno, Y.: Towards taking pulses over youtube to determine interest in video content. In: Proceedings of the IEEE International Conference on Computer Vision (IW-FCV). IEEE (2018)Google Scholar
  22. 22.
    de Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013)CrossRefGoogle Scholar
  23. 23.
    Das, K., Ali, S., Otsu, K., Fukuda, H., Lam, A., Kobayashi, Y., Kuno, Y.: Detecting inner emotions from video based heart rate sensing. In: 13th International Conference on Intelligent Computing, ICIC 2017, pp. 48–57 (2017)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Keya Das
    • 1
    Email author
  • Antony Lam
    • 1
  • Hisato Fukuda
    • 1
  • Yoshinori Kobayashi
    • 1
  • Yoshinori Kuno
    • 1
  1. 1.Graduate School of Science and EngineeringSaitama UniversitySaitamaJapan

Personalised recommendations