Performance Analysis and Application of Expressiveness Detection on Facial Expression Videos Using Deep Learning Techniques

  • K. G. SrinivasaEmail author
  • Sriram Anupindi
Original Article


The emergence of various social media platforms has promoted a rapid growth in multimedia generation and proliferation. The interpretation of multimedia data pose a challenge for current computer systems and methodologies. The introduction of revolutionary and sophisticated methods such as Convoluted Neural Networks (CNN) and Long Short-Term Memory (LSTM) has improved the feasibility of extracting meaningful content from various data sources. The focus of this paper is to highlight how the abovementioned methods were utilized to determine the expressiveness of a subject’s response to a video commercial. A real-time expressiveness feedback solution is explored in this paper as well.


Deep learning Video analysis LSTM CNN 


  1. 1.
    D.J. McDuff, R. el Kaliouby, T. Senechal, M. Amr, J.F. Cohn, R.W. Picard, in Affectiva-MIT facial expression dataset (AM-FED): naturalistic and spontaneous facial expressions collected In-the-Wild. Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , (2013)Google Scholar
  2. 2.
    Facial action coding system,
  3. 3.
    Philipp Michel, Rana el Kaliouby, in Real time facial expression recognition in video using support vector machines. Proceedings of the 5th International Conference on Multimodal Interfaces (Vancouver, Canada, 2003)Google Scholar
  4. 4.
    B.E. Boser, I.M. Guyon, V.N. Vapnik, in A training algorithm for optimal margin classifiers. Proceedings of the fifth annual workshop on Computational learning theory, Pittsburgh, (1992), pp. 144–152Google Scholar
  5. 5.
    H. Mobahi, R. Collobert, J. Weston, in Deep learning from temporal coherence in video. Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, (2009)Google Scholar
  6. 6.
    Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, in Gradient-based learning applied to document recognition, Vol. 86, (1998), pp. 2278–2324Google Scholar
  7. 7.
    A. Krizhevsky, I. Sutskever, G.E. Hinton, in ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25 (NIPS’2012), (2012)Google Scholar
  8. 8.
    S.A. Nene, S.K. Nayar, H. Murase, Columbia object image library (COIL-100). Technical Report CUCS-006- 96, Columbia University. (1996)
  9. 9.
    A. Rao, N. Thiagarajan, Recognizing facial expressions from videos using deep belief networks. Stanford CS 229 Machine Learning Final Projects, Technical Report (2010)Google Scholar
  10. 10.
    J.Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, G. Toderici, Beyond short snippets: deep networks for video classification (2015)Google Scholar
  11. 11.
    J. Donahue, L.A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, T. Darrell, in Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2015)Google Scholar
  12. 12.
    Ffmpeg : Desciption,
  13. 13.
    P.W.D. Charles, Project Title, GitHub repository, (2013)
  14. 14.
    J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, Y.B. Theano, in A CPU and GPU math expression compiler. Proceedings of the Python for Scientific Computing Conference (SciPy). Oral, (2010)Google Scholar
  15. 15.
    T. Kanungo, D.M. Mount, N.S. Netanyahu, C.D. Piatko, R. Silverman, A.Y. Wu, An efficient k-means clustering algorithm: analysis and implementation. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 881–892 (2002)CrossRefGoogle Scholar
  16. 16.
    D. Müllner, Modern hierarchical, agglomerative clustering algorithms. arXiv:1109.2378 (2011)
  17. 17.
    M. Abadi, et al., Tensorflow: large-scale machine learning on heterogeneous systems. arXiv:1603.04467 (2016)
  18. 18.
    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  19. 19.
    M.D. Zeiler, ADADELTA: An adaptive learning rate method. arXiv:1212.5701 (2012)
  20. 20.
    S. Hochreiter, J. Schmidhuber, Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  21. 21.
    J. Martens, I. Sutskever, in Learning recurrent neural networks with hessian-free optimization. ICML, (2011)Google Scholar
  22. 22.
    LSTM Network for Semantic Analysis : Description,
  23. 23.
    J. Rafal, W. Zaremba, I. Sutskever, in An empirical exploration of recurrent network architectures. Proceedings of the 32nd International Conference on Machine Learning (ICML- 15), (2015)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.CBP Government Engineering CollegeNew DelhiIndia
  2. 2.M S Ramaiah Institute of TechnologyBangaloreIndia

Personalised recommendations