Generating Context-Free Group-Level Emotion Landscapes Using Image Processing and Shallow Convolutional Neural Networks

  • Sabyasachi TribediEmail author
  • Ranjit Kumar Barai
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1119)


Emotion recognition is an integral part of any Human–Machine Interaction (HMI) system. Proper emotion recognition allows for HMI systems to choose the successive appropriate responses, given context and the emotion expressed by human(s). The advent of deep learning using Deep Neural Networks (DNN) has made incredible strides in achieving and even exceeding human accuracy in image classification and face detection. Many papers have been published mentioning the successful applications of DNNs like the Convolutional Neural Networks (CNNs), which have now become the de facto algorithm for facial image classification tasks because they combine the feature extraction and classification steps into one mathematical model. They learn the desirable features by themselves from the input images and have been demonstrated to be robust to variations in facial image data. However, there is one big bottleneck for CNNs: the models with good accuracy have many hidden layers, and hence are very deep and require heavy computing power, memory space and, of course, time to train themselves. Our paper puts forward two experimental approaches that can be extremely beneficial to the domains of CNNs and HMI systems. For the first approach, we were able to achieve a very good accuracy in emotion classification using a shallow CNN with only three and four hidden layers. This was possible only because we passed the input images through a carefully designed pipeline of image preprocessing techniques before feeding them to the CNN for training. For the second approach, we developed the interpretation of emotion landscape or the distribution of emotion classes detected in static images or videos with many people’s faces visible. This is similar to the group-level emotion classification studies and publications with a distinct difference in possible applications. The rationale behind this integration is in advancing the idea of studying emotions expressed by people in a group setting and how they are mutually influential, visualize the change in the emotion distribution with time and thereby form an emotional landscape in time, and enhance the understanding of collective sentiments non-verbally expressed through facial emotions in gatherings of known social context.


Convolutional neural networks Shallow neural networks Image processing Non-verbal sentimental analysis Social groups Emotion recognition Facial expressions Facial emotion recognition Inferring affect Group influence 


  1. 1.
    Mehrabian, A.: Silent Messages. 1st edn. (1971)Google Scholar
  2. 2.
    Mower, E., Mataric, M.J., Narayanan, S.: A framework for automatic human emotion classification using emotion profiles. IEEE Trans. Audio Speech Lang. Process. 19(5), 1057–1070 (2011)CrossRefGoogle Scholar
  3. 3.
    Vaish, A., Gupta, S.: Employing Kaze features for the purpose of emotion recognition. In: Progress in Computing, Analytics and Networking, pp. 679–685. Springer, Singapore (2018)Google Scholar
  4. 4.
    Gupta, A., Agrawal, D., Chauhan, H., Dolz, J., Pedersoli, M.: An attention model for group-level emotion recognition. In: Proceedings of the 2018 on International Conference on Multimodal Interaction, pp. 611–615. ACM (2018)Google Scholar
  5. 5.
    Rassadin, A., Gruzdev, A., Savchenko, A.: Group-level emotion recognition using transfer learning from face identification. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp. 544–548. ACM (2017)Google Scholar
  6. 6.
    Tan, L., Zhang, K., Wang, K., Zeng, X., Peng, X., Qiao, Y.: Group emotion recognition with individual facial emotion CNNs and global image based CNNs. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp. 549–552. ACM (2017).Google Scholar
  7. 7.
    Barsade, S.G., Gibson, D.E.: Group Emotion: A View from Top and Bottom (1998)Google Scholar
  8. 8.
    Kelly, J.R., Barsade, S.G.: Mood and emotions in small groups and work teams. Organ. Behav. Hum. Decis. Process. 86(1), 99–130 (2001)CrossRefGoogle Scholar
  9. 9.
    Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J., Budynek, J.: The Japanese female facial expression (JAFFE) database. In: Proceedings of Third International Conference on Automatic Face and Gesture Recognition, pp. 14–16 (1998)Google Scholar
  10. 10.
    Cohn, J.F.: Cohn-Kanade AU-Coded Facial Expression Database. Pittsburgh University (1999)Google Scholar
  11. 11.
    OpenCV Project. Accessed from
  12. 12.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. CVPR 2001, vol. 1, pp. I–I. IEEE (2001)Google Scholar
  13. 13.
    Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)Google Scholar
  14. 14.
    Rosebrock, A.: Facial landmarks with dlib, OpenCV, and Python (3 April 2017). Accessed from
  15. 15.
    Sagonas, C., Antonakos, E., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: database and results. Image Vis. Comput. 47, 3–18 (2016)CrossRefGoogle Scholar
  16. 16.
    Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial feature localization. In: European Conference on Computer Vision, pp. 679–692. Springer, Berlin (2012)Google Scholar
  17. 17.
    Why are neural networks becoming deeper, but not wider? (asked July 9 2016 on Accessed from
  18. 18.
    What is the difference between a neural network and a deep neural network, and why do the deep ones work better? (asked on 20 November 2015 on Accessed from
  19. 19.
  20. 20.

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  1. 1.Department of Electrical EngineeringJadavpur UniversityKolkataIndia

Personalised recommendations