Body Gesture Modeling for Psychology Analysis in Job Interview Based on Deep Spatio-Temporal Approach

  • Intissar KhalifaEmail author
  • Ridha Ejbali
  • Mourad Zaied
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 931)


Social psychologists have long studied job interviews with the aim of knowing the relationships between behaviors, interview outcomes, and job performance. Several companies give great importance to psycho-test based on observation of the candidate is behavior more than the answers they even especially in sensitive positions like trade, marketing, investigation, etc. Our work will be a combination between two interesting topics of research in the last decades which are social psychology and affective computing. Some techniques were proposed until today to analyze automatically the candidate is non verbal behavior. This paper concentrates in body gestures which is an important non-verbal expression channel during affective communication that is not very studied in comparison to facial expressions. We proposed in this work a deep Spatio-temporal approach, it merges the temporal normalization method which is the energy binary motion information (EBMI) with deep learning based on stacked auto-encoder (SAE) for emotional body gesture recognition in job interview and the results prove the efficiency of our proposed approach.


Non-verbal behavior Body gestures Deep learning EBMI SAE 



The authors would like to acknowledge the financial support of this work by grants from General Direction of Scientific Research (DGRST), Tunisia, under the ARUB program.


  1. 1.
    Afdhal, R., Ejbali, R., Zaied, M.: Emotion recognition using the shapes of the wrinkles. In: The 19th international Conference on Computer and Information Technology ICCIT (2016‏)Google Scholar
  2. 2.
    Mehrabian, A.: Communication without words. Psychol. Today 4, 53–56 (1968)Google Scholar
  3. 3.
    Nguyen, L.S., Frauendorfer, D., Mast, M., Gatica-Perez, D.: Computational inference of hirability in employment interviews based on non verbal behavior. IEEE Trans. Multimed. 16, 1018–1031 (2014)CrossRefGoogle Scholar
  4. 4.
    Wang, W., Enescu, V., Sahli, H.: Adaptive real-time emotion recognition from body movements. ACM Trans. Interact. Intell. Syst. 5, 18 (2015)CrossRefGoogle Scholar
  5. 5.
    Abrilian, S.: Représentation de Comportements Emotionnels Multimodaux Spontanés: Perception, Annotation et Synthès., Thèse en informatique de l’Université Paris (2007)Google Scholar
  6. 6.
    Liang, H., Zhao, Y., Wei, J., Quan, D., Cheng, R., Wei, Y.: Robust hand detection and tracking based on monocular vision. In: IEEE International Conference on Intelligent Human-Machine Systems and Cybernetics (2014)Google Scholar
  7. 7.
    Bouchrika, T., Zaied, M., Jemai, O., Amar, C.B.: Neural solutions to interact with computers by hand gesture recognition. Multimed. Tools Appl. 72, 2949–2975 (2014)CrossRefGoogle Scholar
  8. 8.
    Khalifa, I., Ejbali, R., Zaied, M.: Hand motion modeling for psychology analysis in job interview using optical flow-history motion image (OF-HMI). In: The 10th International Conference on Machine Vision ICMV (2018)Google Scholar
  9. 9.
    de Gelder, B.: Why bodies? twelve reasons for including bodily expressions in affective neuroscience. Philos. Trans. Roy. Soc. B: Biol. Sci. 364, 3475–3484 (2009)CrossRefGoogle Scholar
  10. 10.
    Kipp, M.: Gesture Generation by Imitation From Human Behavior to Computer Character Animation., Boca Raton (2004)Google Scholar
  11. 11.
    Ekman, P., Friesen, W.V.: The repertoire of nonverbal behavior: categories, origins, and coding. Semiotica 1, 49–98 (1969)CrossRefGoogle Scholar
  12. 12.
    McNeill, D.: Gesture and Thought. The university of Chicago Press books, Chicago (2005)CrossRefGoogle Scholar
  13. 13.
    Burgoon, J.K., Jensen, M.L., Meservy, T.O., Kruse, J., Nunamaker, J.F.: Augmenting human identification of emotional states in video. In: International Conference on Intelligent Data Analysis (2005)Google Scholar
  14. 14.
    Coulson, M.: Attributing emotion to static body postures: recognition accuracy, confusions, and viewpoint dependence. J. Nonverbal Behav. 39, 117–139 (1992)Google Scholar
  15. 15.
    Gunes, H., Piccardi, M.: A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In: The 18th International Conference on Pattern Recognition ICPR (2006)Google Scholar
  16. 16.
    Savva, N., Bianchi-Berthouze, N.: Automatic recognition of affective body movement in a video game scenario. In: Camurri, A., Costa, C. (eds.) INTETAIN 2011. LNICST, vol. 78, pp. 149–159. Springer, Heidelberg (2012). Scholar
  17. 17.
    Chen, S., Tian, Y., Liu, Q., Metaxas, D.N.: Recognizing expressions from face and body gesture by temporal normalized motion and appearance features. J. Image Vis. Comput. 3, 175–185 (2013)CrossRefGoogle Scholar
  18. 18.
    Gunes, H., Piccardi, M.: Automatic temporal segment detection and affect recognition from face and body display. IEEE Trans. Syst. Man Cybern. 39, 64–84 (2009)CrossRefGoogle Scholar
  19. 19.
    Gunes, H., Piccardi, M.: Bi-modal emotion recognition from expressive face and body gestures. J. Netw. Comput. Appl. 30, 1334–1345 (2007)CrossRefGoogle Scholar
  20. 20.
    Kumthekar, A.V., Patil, J.K.: Key frame extraction using color histogram method. Int. J. Sci. Res. Eng. Technol. 2, 207–214 (2013)Google Scholar
  21. 21.
    Shi, Y., Yang, H., Gong, M., Liu, X., Xia, Y.: Fast and robust key frame extraction method for video copyright protection. J. Electr. Comput. Eng. 3, 1–17 (2017)Google Scholar
  22. 22.
    Liang, B., Zheng, L.: Gesture recognition from one example using depth images. J. Lect. Notes Softw. Eng. 1, 339–343 (2013)CrossRefGoogle Scholar
  23. 23.
    Hassairi, S., Ejbali, R., Zaied, M.: Supervised image classification using deep convolutional wavelets network. In: 27th International Conference on Tools with Artificial Intelligence ICTAI (2016)Google Scholar
  24. 24.
    Hassairi, S., Ejbal, R., Zaied, M.: A deep convolutional neural wavelet network to supervised Arabic letter image classification. In: 15th International Conference on Intelligent Systems Design and Applications ISDA (2015)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Research Team in Intelligent MachinesNational Engineering School of GabesGabesTunisia

Personalised recommendations