Grounded Approach for Understanding Changes in Human Emotional States in Real Time Using Psychophysiological Sensory Apparatuses
This paper discusses the technical and philosophical challenges that researchers and practitioners face when attempting to classify human emotion based upon raw physiological data. It proposes the use of a representational learning approach that adopts techniques from industrial internet of things (IoT) solutions. It applies this approach to the classification of emotional states using functional near infrared spectroscopy (fNIRS) sensor data.
The algorithm used first pre-processes the data using a combination of signal processing and vector quantization techniques. Next, it found the optimal number of natural clusters within human emotional states and used these as the target variables for either shallow or for deep learning classification. The deep learning variant used a Restricted Boltzmann Machine (RBM) to form a compressive representation of the input data prior to classification. A final single layer perception model learned the relationship between the input and output states.
This approach would be useful for detecting real-time changes in human emotional state. It is able automatically create emotional states that are both highly separable and balanced. It is able to distinguish between low v. high emotional states across all tasks (F1-score of 71.4%) and is better at forming this distinction for tasks intended to elicit higher cognitive load such as the Tetris video game (F1-score of 87.1%) or the Multi Attribute Task Battery (F1-score of 77%).
KeywordsAffective computing Cognitive computing Brain signal processing Brain computer interfaces Decision-making Decision support systems DSS Machine learning Deep learning Classification
The author thanks Mark Costa, Danushka Bandara, Leanne Hirshfield and Syracuse University for enabling the application of this research to the context of human physiological data. They provided historic data and offered detailed descriptions related to past experiments.
- 1.Bandara, D., Bratt,S., Hirshfield, L., Velipasalar, S.: Building predictive models of emotion with functional near-infrared spectroscopy. Int. J. Hum.-Comput. Stud. (in press)Google Scholar
- 2.Bandara, D., Song, S., Hirshfield, L., Velipasalar, S.: A more complete picture of emotion using electrocardiogram and electrodermal activity to complement cognitive data. In: Schmorrow, D.D., Fidopiastis, Cali M.M. (eds.) AC 2016. LNCS (LNAI), vol. 9743, pp. 287–298. Springer, Cham (2016). doi: 10.1007/978-3-319-39955-3_27 Google Scholar
- 4.Taylor, G.W., Hinton, G.E., Roweis, S.: Modeling human motion using binary latent variables. In: 19th Proceedings of 2006 Conference on Advances in Neural Information Processing Systems, vol. 1, pp. 1345–1352. MIT Press (2007)Google Scholar
- 6.Jaderberg, M., et al.: Reinforcement learning with unsupervised auxiliary tasks. Comput. Res. Repos. 1611 (2016)Google Scholar
- 7.Chan, M., Lowrance, J., Murdock, J., Ruspini, E.H., Yang, J., Yeh, E.: Human-aided multi-sensor fusion. In: 7th International Conference on Information Fusion, p. 3 (2005). doi: 10.1109/ICIF.2005.1591827
- 10.Veeramachaneni, K., et al.: AI2: training a big data machine to defend. In: 2016 IEEE 2nd International Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing (HPSC), and IEEE International Conference on Intelligent Data and Security (IDS), New York, NY, pp. 49–54 (2016). doi: 10.1109/BigDataSecurity-HPSC-IDS.2016.79
- 11.Kirk, R.A.: Evaluating a cognitive tool built to aid decision making using decision making approach as a theoretical framework and using unobtrusive, behavior-based measures for analysis. Doctoral dissertation (2015). Retrieved from ProQuest (3684297)Google Scholar
- 15.Gardenfors, P.: Conceptual Spaces: The Geometry of Thought. MIT Press, Hong Kong (2000)Google Scholar
- 18.Binh, L.N.: Optical Multi-bound Solitons. CRC Press, Boca Raton (2016)Google Scholar
- 20.Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Human Mental Workload. Advances in Psychology, vol. 52, pp. 139–183. North Holland, Amsterdam (1988). doi: 10.1016/S0166-4115(08)62386-9