Abstract
Emotion recognition is an integral part of any Human–Machine Interaction (HMI) system. Proper emotion recognition allows for HMI systems to choose the successive appropriate responses, given context and the emotion expressed by human(s). The advent of deep learning using Deep Neural Networks (DNN) has made incredible strides in achieving and even exceeding human accuracy in image classification and face detection. Many papers have been published mentioning the successful applications of DNNs like the Convolutional Neural Networks (CNNs), which have now become the de facto algorithm for facial image classification tasks because they combine the feature extraction and classification steps into one mathematical model. They learn the desirable features by themselves from the input images and have been demonstrated to be robust to variations in facial image data. However, there is one big bottleneck for CNNs: the models with good accuracy have many hidden layers, and hence are very deep and require heavy computing power, memory space and, of course, time to train themselves. Our paper puts forward two experimental approaches that can be extremely beneficial to the domains of CNNs and HMI systems. For the first approach, we were able to achieve a very good accuracy in emotion classification using a shallow CNN with only three and four hidden layers. This was possible only because we passed the input images through a carefully designed pipeline of image preprocessing techniques before feeding them to the CNN for training. For the second approach, we developed the interpretation of emotion landscape or the distribution of emotion classes detected in static images or videos with many people’s faces visible. This is similar to the group-level emotion classification studies and publications with a distinct difference in possible applications. The rationale behind this integration is in advancing the idea of studying emotions expressed by people in a group setting and how they are mutually influential, visualize the change in the emotion distribution with time and thereby form an emotional landscape in time, and enhance the understanding of collective sentiments non-verbally expressed through facial emotions in gatherings of known social context.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Mehrabian, A.: Silent Messages. 1st edn. (1971)
Mower, E., Mataric, M.J., Narayanan, S.: A framework for automatic human emotion classification using emotion profiles. IEEE Trans. Audio Speech Lang. Process. 19(5), 1057–1070 (2011)
Vaish, A., Gupta, S.: Employing Kaze features for the purpose of emotion recognition. In: Progress in Computing, Analytics and Networking, pp. 679–685. Springer, Singapore (2018)
Gupta, A., Agrawal, D., Chauhan, H., Dolz, J., Pedersoli, M.: An attention model for group-level emotion recognition. In: Proceedings of the 2018 on International Conference on Multimodal Interaction, pp. 611–615. ACM (2018)
Rassadin, A., Gruzdev, A., Savchenko, A.: Group-level emotion recognition using transfer learning from face identification. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp. 544–548. ACM (2017)
Tan, L., Zhang, K., Wang, K., Zeng, X., Peng, X., Qiao, Y.: Group emotion recognition with individual facial emotion CNNs and global image based CNNs. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp. 549–552. ACM (2017).
Barsade, S.G., Gibson, D.E.: Group Emotion: A View from Top and Bottom (1998)
Kelly, J.R., Barsade, S.G.: Mood and emotions in small groups and work teams. Organ. Behav. Hum. Decis. Process. 86(1), 99–130 (2001)
Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J., Budynek, J.: The Japanese female facial expression (JAFFE) database. In: Proceedings of Third International Conference on Automatic Face and Gesture Recognition, pp. 14–16 (1998)
Cohn, J.F.: Cohn-Kanade AU-Coded Facial Expression Database. Pittsburgh University (1999)
OpenCV Project. Accessed from https://opencv.org
Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. CVPR 2001, vol. 1, pp. I–I. IEEE (2001)
Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)
Rosebrock, A.: Facial landmarks with dlib, OpenCV, and Python (3 April 2017). Accessed from https://www.pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/
Sagonas, C., Antonakos, E., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: database and results. Image Vis. Comput. 47, 3–18 (2016)
Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial feature localization. In: European Conference on Computer Vision, pp. 679–692. Springer, Berlin (2012)
Why are neural networks becoming deeper, but not wider? (asked July 9 2016 on stats.stackexchange.com). Accessed from https://stats.stackexchange.com/questions/222883/why-are-neural-networks-becoming-deeper-but-not-wider
What is the difference between a neural network and a deep neural network, and why do the deep ones work better? (asked on 20 November 2015 on stats.stackexchange.com). Accessed from https://stats.stackexchange.com/questions/182734/what-is-the-difference-between-a-neural-network-and-a-deep-neural-network-and-w
Image file Accessed from https://www.bfi.org.uk/sites/bfi.org.uk/files/styles/full/public/image/our-little-sister-2015-002-four-sisters-at-dining-table.jpg
Image file Accessed from https://i.ytimg.com/vi/B27yQMuTHWc/maxresdefault.jpg
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Tribedi, S., Barai, R.K. (2020). Generating Context-Free Group-Level Emotion Landscapes Using Image Processing and Shallow Convolutional Neural Networks. In: Das, H., Pattnaik, P., Rautaray, S., Li, KC. (eds) Progress in Computing, Analytics and Networking. Advances in Intelligent Systems and Computing, vol 1119. Springer, Singapore. https://doi.org/10.1007/978-981-15-2414-1_32
Download citation
DOI: https://doi.org/10.1007/978-981-15-2414-1_32
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-2413-4
Online ISBN: 978-981-15-2414-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)