Abstract
Affective computing (AC) is an emerging research direction to deal with the great challenge of creating emotional intelligence for a machine. Affective computing is a cross-disciplinary research knowledge that integrates recognition, interpretation, and simulation of human emotion into a system. This article describes the design of multimodal perception of affective computing system. Our multimodal physiological channels include facial expression recognition, heart rate monitoring, blood oxygen level (SpO2), skin conductance response (SCR), and electroencephalogram (EEG) signals for building our affective computing system. To solve the various data sampling problem, we developed a concurrent control integration mechanism to automatically average all the sensors’ data into the same data sampling rate (one record per second) and rearrange all the data into the same time. We believed the proposed system design is benefit for helping researchers in collecting and integrating experiment data in affective computing area.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wu, C.-H., Huang, Y.-M., Hwang, J.-P.: Review of affective computing in education/learning: trends and challenges. Br. J. Educ. Technol. 47, 1304–1323 (2016)
Piaget, J.: Les Relations entre l’affectivité et l’intelligence dans le développement mental de l’enfant. Centre de documentation universitaire, Paris (1964)
Ben Ammar, M., Neji, M., Alimi, A.M., Gouardères, G.: The affective tutoring system. Expert Syst. Appl. 37, 3013–3023 (2010)
Dimigen, O., Sommer, W., Hohlfeld, A., Jacobs, A.M., Kliegl, R.: Coregistration of eye movements and EEG in natural reading: analyses and review. J. Exp. Psychol. Gen. 140, 552–572 (2011)
Latanov, A.V., Konovalova, N.S., Yermachenko, A.A.: EEG and EYE tracking for visual search task investigation in humans. Int. J. Psychophysiol. 69, 140 (2008)
Lin, T., Imamiya, A., Mao, X.: Using multiple data sources to get closer insights into user cost and task performance. Interact. Comput. 20, 364–374 (2008)
Schmid, P.C., Schmid Mast, M., Bombari, D., Mast, F.W., Lobmaier, J.S.: How mood states affect information processing during facial emotion recognition: an eye tracking study. Swiss J. Psychol. 70, 223–231 (2011)
Chen, C.-M., Wang, H.-P.: Using emotion recognition technology to assess the effects of different multimedia materials on learning emotion and performance. Libr. Inf. Sci. Res. 33, 244–255 (2011)
Zhang, C., Zheng, C.-X., Yu, X.-L.: Automatic recognition of cognitive fatigue from physiological indices by using wavelet packet transform and kernel learning algorithms. Exp. Syst. Appl. 36, 4664–4671 (2009)
Patel, M., Lal, S.K.L., Kavanagh, D., Rossiter, P.: Applying neural network analysis on heart rate variability data to assess driver fatigue. Exp. Syst. Appl. 38, 7235–7242 (2011)
Zhao, C., Zhao, M., Liu, J., Zheng, C.: Electroencephalogram and electrocardiograph assessment of mental fatigue in a driving simulator. Accid. Anal. Prev. 45, 83–90 (2012)
Bahreini, K., Nadolski, R., Westera, W.: Towards multimodal emotion recognition in e-learning environments. Interact. Learn. Env. 24, 590–605 (2016)
Wu, C.H.: New technology for developing facial expression recognition in e-learning. In: 2016 Portland International Conference on Management of Engineering and Technology (PICMET), pp. 1719–1722 (2016)
Chanel, G., Ansari-Asl, K., Pun, T.: Valence-arousal evaluation using physiological signals in an emotion recall paradigm. Lecturer Notes in Computer Science, vol. 1, pp. 530–537(2007)
Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3, 42–55 (2012)
Chen, L., Zhou, C., Shen, L.: Facial expression recognition based on SVM in E-learning. IERI Procedia 2, 781–787 (2012)
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U., Narayanan, S.: Analysis of emotion recognition using facial expressions, speech and multimodal information. In: Sixth International Conference on Multimodal Interfaces ICMI 2004 (2004)
Lin, H.-C.K., Wu, C.-H., Hsueh, Y.-P.: The influence of using affective tutoring system in accounting remedial instruction on learning performance and usability. Comput. Hum. Behav. 41, 514–522 (2014)
Thompson, N., McGill, T.J.: Genetics with jean: the design, development and evaluation of an affective tutoring system. Educ. Tech. Res. Dev. 65, 279–299 (2017)
Gonzalez-Sanchez, J., Chavez-Echeagaray, M.E., Atkinson, R., Burleson, W.: ABE: an agent-based software architecture for a multimodal emotion recognition framework. In: 2011 9th Working IEEE/IFIP Conference on Software Architecture (WICSA), pp. 187–193 (2011)
Acknowledgements
This work was financially supported by Ministry of Science and Technology for support (Grant No. MOST 104-2410-H-142-017-MY2).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wu, CH., Kuo, BC. (2018). An Exploratory Study of Multimodal Perception for Affective Computing System Design. In: Hung, J., Yen, N., Hui, L. (eds) Frontier Computing. FC 2017. Lecture Notes in Electrical Engineering, vol 464. Springer, Singapore. https://doi.org/10.1007/978-981-10-7398-4_20
Download citation
DOI: https://doi.org/10.1007/978-981-10-7398-4_20
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-7397-7
Online ISBN: 978-981-10-7398-4
eBook Packages: EngineeringEngineering (R0)