Distinguishing Posed and Spontaneous Smiles by Facial Dynamics

  • Bappaditya MandalEmail author
  • David Lee
  • Nizar Ouarti
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10116)


Smile is one of the key elements in identifying emotions and present state of mind of an individual. In this work, we propose a cluster of approaches to classify posed and spontaneous smiles using deep convolutional neural network (CNN) face features, local phase quantization (LPQ), dense optical flow and histogram of gradient (HOG). Eulerian Video Magnification (EVM) is used for micro-expression smile amplification along with three normalization procedures for distinguishing posed and spontaneous smiles. Although the deep CNN face model is trained with large number of face images, HOG features outperforms this model for overall face smile classification task. Using EVM to amplify micro-expressions did not have a significant impact on classification accuracy, while the normalizing facial features improved classification accuracy. Unlike many manual or semi-automatic methodologies, our approach aims to automatically classify all smiles into either ‘spontaneous’ or ‘posed’ categories, by using support vector machines (SVM). Experimental results on large UvA-NEMO smile database show promising results as compared to other relevant methods.


Support Vector Machine Optical Flow Face Image Face Region Feature Extraction Method 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. PAMI 31, 39–58 (2009)CrossRefGoogle Scholar
  2. 2.
    Ekman, P., Rosenberg, E.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System, 2nd edn. Oxford University Press, New York (2005)CrossRefGoogle Scholar
  3. 3.
    Hoque, M., McDuff, D., Picard, R.: Exploring temporal patterns in classifying frustrated and delighted smiles. IEEE Trans. Affect. Comput. 3, 323–334 (2012)CrossRefGoogle Scholar
  4. 4.
    Ambadar, Z., Cohn, J., Reed, L.: All smiles are not created equal: morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. J. Nonverbal Behav. 33, 17–34 (2009)CrossRefGoogle Scholar
  5. 5.
    Ekman, P.: Telling Lies: Cues To Deceit in the Marketplace, Politics, and Marriage. WW. Norton & Company, New York (1992)Google Scholar
  6. 6.
    Hadwin, J., Baron-Cohen, S., Howlin, P., Hill, K.: Can we teach children with autism to understand emotion, belief, or pretense? Dev. Psychopathol. 8, 345–365 (1996)CrossRefGoogle Scholar
  7. 7.
    Xu, Q., Ching, S., Mandal, B., Li, L., Lim, J.H., Mukawa, M., Tan, C.: Socio glass: social interaction assistance with face recognition on google glass. J. Sci. Phone Apps Mob. Devices 2, 1–4 (2016)CrossRefGoogle Scholar
  8. 8.
    Mandal, B., Lim, R.Y., Dai, P., Sayed, M.R., Li, L., Lim, J.H.: Trends in machine and human face recognition. In: Kawulok, M., Celebi, M.E., Smolka, B. (eds.) Advances in Face Detection and Facial Image Analysis, pp. 145–187. Springer, Cham (2016). doi: 10.1007/978-3-319-25958-1_7 Google Scholar
  9. 9.
    Mandal, B., Wang, Z., Li, L., Kassim, A.A.: Performance evaluation of local descriptors and distance measures on benchmarks and first-person-view videos for face identification. Neurocomputing 184, 107–116 (2016)CrossRefGoogle Scholar
  10. 10.
    Ekman, P., Friesen, W.: The Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press Inc., San Francisco (1978)Google Scholar
  11. 11.
    Krumhuber, E.G., Manstead, A.S.: Can duchenne smiles be feigned? new evidence on felt and false smiles. Emotion 9, 807–820 (2009)CrossRefGoogle Scholar
  12. 12.
    Ekman, P., Hager, J., Friesen, W.: The symmetry of emotional and deliberate facial actions. Psychophysiology 18, 101–106 (1981)CrossRefGoogle Scholar
  13. 13.
    Dibeklioğlu, H., Salah, A.A., Gevers, T.: Are you really smiling at me? spontaneous versus posed enjoyment smiles. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 525–538. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33712-3_38 CrossRefGoogle Scholar
  14. 14.
    Schmidt, K., Bhattacharya, S., Denlinger, R.: Comparison of deliberate and spontaneous facial movement in smiles and eyebrow raises. J. Nonverbal Behav. 33, 35–45 (2009)CrossRefGoogle Scholar
  15. 15.
    Cohn, J., Schmidt, K.: The timing of facial motion in posed and spontaneous smiles. Int. J. Wavelets, Multiresolut. Inf. Process. 2, 1–12 (2004)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Schmidt, K., Ambadar, Z., Cohn, J., Reed, I.: Movement differences between deliberate and spontaneous facial expressions: zygomaticus major action in smiling. J. Nonverbal Behav. 30, 37–52 (2006)CrossRefGoogle Scholar
  17. 17.
    Valstar, M.F., Pantic, M., Ambadar, Z., Cohn, J.F.: Spontaneous vs. posed facial behavior: automatic analysis of brow actions. In: Proceedings of ACM International Conference on Multimodal Interaction, pp. 162–170 (2006)Google Scholar
  18. 18.
    Tao, H., Huang, T.: Explanation-based facial motion tracking using a piecewise bézier volume deformation model. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 611–617 (1999)Google Scholar
  19. 19.
    Mandal, B., Ouarti, N.: Spontaneous vs. posed smiles - can we tell the difference? In: International Conference on Computer Vision and Image Processing (CVIP), vol. 460, pp. 261–271. Roorkee, India (2016)Google Scholar
  20. 20.
    Mandal, B., Chia, S.-C., Li, L., Chandrasekhar, V., Tan, C., Lim, J.-H.: A wearable face recognition system on google glass for assisting social interactions. In: Jawahar, C.V., Shan, S. (eds.) ACCV 2014. LNCS, vol. 9010, pp. 419–433. Springer, Cham (2015). doi: 10.1007/978-3-319-16634-6_31 Google Scholar
  21. 21.
    Mandal, B., Li, L., Chandrasekhar, V., Lim, J.H.: Whole space subclass discriminant analysis for face recognition. In: IEEE International Conference on Image Processing (ICIP), Quebec, Canada, pp. 329–333 (2015)Google Scholar
  22. 22.
    OpenCV: Open source computer vision (2014).
  23. 23.
    Yu, X., Han, W., Li, L., Shi, J.Y., Wang, G.: An eye detection and localization system for natural human and robot interaction without face detection. In: Groß, R., Alboul, L., Melhuish, C., Witkowski, M., Prescott, T.J., Penders, J. (eds.) TAROS 2011. LNCS, vol. 6856, pp. 54–65. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-23232-9_6 CrossRefGoogle Scholar
  24. 24.
    Mandal, B., Zhikai, W., Li, L., Kassim, A.A.: Evaluation of descriptors and distance measures on benchmarks and first-person-view videos for face identification. In: Jawahar, C.V., Shan, S. (eds.) ACCV 2014. LNCS, vol. 9008, pp. 585–599. Springer, Cham (2015). doi: 10.1007/978-3-319-16628-5_42 Google Scholar
  25. 25.
    Tomasi, C., Kanade, T.: Detection and tracking of point features. Carnegie Mellon University Technical Report CMU-CS-91-132 (1991)Google Scholar
  26. 26.
    Mandal, B., Eng, H.L.: 3-parameter based eigenfeature regularization for human activity recognition. In: 35th IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 954–957 (2010)Google Scholar
  27. 27.
  28. 28.
    Mandal, B., Jiang, X.D., Kot, A.: Verification of human faces using predicted eigenvalues. In: 19th International Conference on Pattern Recognition (ICPR), Tempa, Florida, USA, pp. 1–4 (2008)Google Scholar
  29. 29.
    Jiang, X.D., Mandal, B., Kot, A.: Face recognition based on discriminant evaluation in the whole space. In: IEEE 32nd International Conference on Acoustics, Speech and Signal Processing (ICASSP 2007), Honolulu, Hawaii, USA, pp. 245–248 (2007)Google Scholar
  30. 30.
    Wu, H.Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.T.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31, 65 (2012)CrossRefGoogle Scholar
  31. 31.
    Wadhwa, N., Rubinstein, M., Durand, F., Freeman, W.T.: Phase-based video motion processing. ACM Trans. Graph. 32 (2013)Google Scholar
  32. 32.
    Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikäinen, M.: Reading hidden emotions: spontaneous micro-expression spotting and recognition. CoRR abs/1511.00423 (2015)Google Scholar
  33. 33.
    Ojansivu, V., Heikkilä, J.: Blur insensitive texture classification using local phase quantization. Image Signal Process. 5099, 236–243 (2008)CrossRefGoogle Scholar
  34. 34.
    Ahonen, T., Rahtu, E., Ojansivu, V., Heikkilä, J.: Recognition of blurred faces using local phase quantization. In: 19th International Conference on Pattern Recognition, pp. 1–4 (2008)Google Scholar
  35. 35.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893 (2005)Google Scholar
  36. 36.
    Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part based models. IEEE Trans. Pattern Anal. Mach. Intell. 32, 1627–1645 (2009)CrossRefGoogle Scholar
  37. 37.
    Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Bigun, J., Gustavsson, T. (eds.) SCIA 2003. LNCS, vol. 2749, pp. 363–370. Springer, Heidelberg (2003). doi: 10.1007/3-540-45103-X_50 CrossRefGoogle Scholar
  38. 38.
    Hu, G., Yang, Y., Yi, D., Kittler, J., Christmas, W.J., Li, S.Z., Hospedales, T.M.: When face recognition meets with deep learning: an evaluation of convolutional neural networks for face recognition. CoRR abs/1504.02351 (2015)Google Scholar
  39. 39.
    Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: Proceedings of the British Machine Vision Conference (BMVC), vol. 41, no. 1–41, p. 12 (2015)Google Scholar
  40. 40.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (1995)zbMATHGoogle Scholar
  41. 41.
    Dibeklioglu, H., Salah, A., Gevers, T.: Recognition of genuine smiles. IEEE Trans. Multimedia 17, 279–294 (2015)CrossRefGoogle Scholar
  42. 42.
    Pfister, T., Li, X., Zhao, G., Pietikainen, M.: Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In: ICCV Workshop, pp. 868–875 (2011)Google Scholar
  43. 43.
    Dibeklioglu, H., Valenti, R., Salah, A., Gevers, T.: Eyes do not lie: spontaneous versus posed smiles. In: ACM Multimedia, pp. 703–706 (2010)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Visual Computing DepartmentInstitute for Infocomm ResearchSingaporeSingapore
  2. 2.Electrical and Computer EngineeringNational University of SingaporeSingaporeSingapore
  3. 3.Université Pierre et Marie CurieParisFrance
  4. 4.Sorbonne UniversitésParisFrance

Personalised recommendations