Grounded Approach for Understanding Changes in Human Emotional States in Real Time Using Psychophysiological Sensory Apparatuses

  • Ryan A. KirkEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10284)


This paper discusses the technical and philosophical challenges that researchers and practitioners face when attempting to classify human emotion based upon raw physiological data. It proposes the use of a representational learning approach that adopts techniques from industrial internet of things (IoT) solutions. It applies this approach to the classification of emotional states using functional near infrared spectroscopy (fNIRS) sensor data.

The algorithm used first pre-processes the data using a combination of signal processing and vector quantization techniques. Next, it found the optimal number of natural clusters within human emotional states and used these as the target variables for either shallow or for deep learning classification. The deep learning variant used a Restricted Boltzmann Machine (RBM) to form a compressive representation of the input data prior to classification. A final single layer perception model learned the relationship between the input and output states.

This approach would be useful for detecting real-time changes in human emotional state. It is able automatically create emotional states that are both highly separable and balanced. It is able to distinguish between low v. high emotional states across all tasks (F1-score of 71.4%) and is better at forming this distinction for tasks intended to elicit higher cognitive load such as the Tetris video game (F1-score of 87.1%) or the Multi Attribute Task Battery (F1-score of 77%).


Affective computing Cognitive computing Brain signal processing Brain computer interfaces Decision-making Decision support systems DSS Machine learning Deep learning Classification 



The author thanks Mark Costa, Danushka Bandara, Leanne Hirshfield and Syracuse University for enabling the application of this research to the context of human physiological data. They provided historic data and offered detailed descriptions related to past experiments.


  1. 1.
    Bandara, D., Bratt,S., Hirshfield, L., Velipasalar, S.: Building predictive models of emotion with functional near-infrared spectroscopy. Int. J. Hum.-Comput. Stud. (in press)Google Scholar
  2. 2.
    Bandara, D., Song, S., Hirshfield, L., Velipasalar, S.: A more complete picture of emotion using electrocardiogram and electrodermal activity to complement cognitive data. In: Schmorrow, D.D., Fidopiastis, Cali M.M. (eds.) AC 2016. LNCS (LNAI), vol. 9743, pp. 287–298. Springer, Cham (2016). doi: 10.1007/978-3-319-39955-3_27 Google Scholar
  3. 3.
    Kalpana, V., Hamde, S.T., Waghmare, L.M.: ECG feature extraction using principal component analysis for studying the effect of diabetes. J. Med. Eng. Technol. 37(2), 116–126 (2013). doi: 10.3109/03091902.2012.753126 CrossRefGoogle Scholar
  4. 4.
    Taylor, G.W., Hinton, G.E., Roweis, S.: Modeling human motion using binary latent variables. In: 19th Proceedings of 2006 Conference on Advances in Neural Information Processing Systems, vol. 1, pp. 1345–1352. MIT Press (2007)Google Scholar
  5. 5.
    Thompson, D.R., Mandrake, L., Green, O.R., Chen, S.A.: A case study of spectral signature detection in multimodal and outlier-contaminated scenes. IEEE Geosci. Remote Sens. Lett. 10(5), 1021–1025 (2013)CrossRefGoogle Scholar
  6. 6.
    Jaderberg, M., et al.: Reinforcement learning with unsupervised auxiliary tasks. Comput. Res. Repos. 1611 (2016)Google Scholar
  7. 7.
    Chan, M., Lowrance, J., Murdock, J., Ruspini, E.H., Yang, J., Yeh, E.: Human-aided multi-sensor fusion. In: 7th International Conference on Information Fusion, p. 3 (2005). doi: 10.1109/ICIF.2005.1591827
  8. 8.
    Pearl, J.: Causality. Cambridge University Press, New York (2000)zbMATHGoogle Scholar
  9. 9.
    Michael, J.-B., et al.: Qualitative analysis of culture using millions of digitized books. Science 331(6014), 176–182 (2011)CrossRefGoogle Scholar
  10. 10.
    Veeramachaneni, K., et al.: AI2: training a big data machine to defend. In: 2016 IEEE 2nd International Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing (HPSC), and IEEE International Conference on Intelligent Data and Security (IDS), New York, NY, pp. 49–54 (2016). doi: 10.1109/BigDataSecurity-HPSC-IDS.2016.79
  11. 11.
    Kirk, R.A.: Evaluating a cognitive tool built to aid decision making using decision making approach as a theoretical framework and using unobtrusive, behavior-based measures for analysis. Doctoral dissertation (2015). Retrieved from ProQuest (3684297)Google Scholar
  12. 12.
    Kirk, R.A., Kirk, D.A., Pesheck, P.: Decision making for complex ecosystems: a technique for establishing causality in dynamic systems. In: Stephanidis, C. (ed.) HCI 2016. CCIS, vol. 617, pp. 110–115. Springer, Cham (2016). doi: 10.1007/978-3-319-40548-3_18 CrossRefGoogle Scholar
  13. 13.
    Jonassen, D.: Using cognitive tools to represent problems. J. Res. Technol. Educ. 35, 36–82 (2003)CrossRefGoogle Scholar
  14. 14.
    Vygotsky, L.S.: Consciousness as a problem in the psychology of behavior. Russ. Soc. Sci. Rev. 20(4), 47–79 (2010). [1061–1428] VygotskyCrossRefGoogle Scholar
  15. 15.
    Gardenfors, P.: Conceptual Spaces: The Geometry of Thought. MIT Press, Hong Kong (2000)Google Scholar
  16. 16.
    Russell, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39(6), 1161–1178 (1980)CrossRefGoogle Scholar
  17. 17.
    Moskowitz, G.B.: Social Cognition: Understanding Self and Others. Guilford Publications, New York (2013)CrossRefGoogle Scholar
  18. 18.
    Binh, L.N.: Optical Multi-bound Solitons. CRC Press, Boca Raton (2016)Google Scholar
  19. 19.
    Pedregosa, F., et al.: Scikit-learn: machine learning in Python. JMLR 2012, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Human Mental Workload. Advances in Psychology, vol. 52, pp. 139–183. North Holland, Amsterdam (1988). doi: 10.1016/S0166-4115(08)62386-9
  21. 21.
    Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Comput. Appl. Math. 20, 53–65 (1987). doi: 10.1016/0377-0427(87)90125-7 CrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Kirk LLCSeattleUSA

Personalised recommendations