Eye Contact Detection from Third Person Video

  • Yuki OhshimaEmail author
  • Atsushi Nakazawa
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12047)


Eye contact is fundamental for human communication and social interactions; therefore much effort has been made to develop automated eye-contact detection using image recognition techniques. However, existing methods use first-person-videos (FPV) that need participants to equip wearable cameras. In this work, we develop an novel eye contact detection algorithm taken from normal viewpoint (third person video) assuming the scenes of conversations or social interactions. Our system have high affordability since it does not require special hardware or recording setups, moreover, can use pre-recorded videos such as Youtube and home videos. In designing algorithm, we first develop DNN-based one-sided gaze estimation algorithms which output the states whether the one subject looks at another. Afterwards, eye contact is found at the frame when the pair of one-sided gaze happens. To verify the proposed algorithm, we generate third-person eye contact video dataset using publicly available videos from Youtube. As the result, proposed algorithms performed 0.775 in precision and 0.671 in recall, while the existing method performed 0.484 in precision and 0.061 in recall, respectively.


Eye contact Deep neural nets Image recognition Third person video Human communication 



This work was supported by JST CREST Grant Number JPMJCR17A5 and JSPS KAKENHI 17H01779, Japan.


  1. 1.
    Adams Jr., R.B., Kleck, R.E.: Effects of direct and averted gaze on the perception of facially communicated emotion. Emotion 5(1), 3 (2005)CrossRefGoogle Scholar
  2. 2.
    Baltrušaitis, T., Robinson, P., Morency, L.P.: OpenFace: an open source facial behavior analysis toolkit. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10. IEEE (2016)Google Scholar
  3. 3.
    Chong, E., et al.: Detecting gaze towards eyes in natural social interactions and its use in child assessment. Proc. ACM Interact. Mobile Wearable Ubiquit. Technol. 1(3), 43 (2017)Google Scholar
  4. 4.
    Csibra, G., Gergely, G.: Social learning and social cognition: the case for pedagogy. Process. Change Brain Cogn. Dev. Atten. Perform. XXI 21, 249–274 (2006)Google Scholar
  5. 5.
    Dozat, T.: Incorporating nesterov momentum into adam (2016). Accessed 25 Aug 2018
  6. 6.
    Gineste, Y., Pellissier, J.: Humanitude: comprendre la vieillesse, prendre soin des hommes vieux(2007). A. ColinGoogle Scholar
  7. 7.
    Joseph, R.M., Ehrman, K., McNally, R., Keehn, B.: Affective response to eye contact and face recognition ability in children with ASD. J. Int. Neuropsychol. Soc. 14(06), 947–955 (2008)CrossRefGoogle Scholar
  8. 8.
    Krafka, K., et al.: Eye tracking for everyone. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2016)Google Scholar
  9. 9.
    Lian, D., Yu, Z., Gao, S.: Believe it or not, we know what you are looking at!. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11363, pp. 35–50. Springer, Cham (2019). Scholar
  10. 10.
    Lu, F., Sugano, Y., Okabe, T., Sato, Y.: Gaze estimation from eye appearance: a head pose-free method via eye image synthesis. IEEE Trans. Image Process. 24(11), 3680–3693 (2015)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Mitsuzumi, Y., Nakazawa, A., Nishida, T.: Deep eye contact detector: robust eye contact bid detection using convolutional neural network. In: Proceedings of the British Machine Vision Conference (BMVC) (2017)Google Scholar
  12. 12.
    Nakazawa, A., Okino, Y., Honda, M.: Evaluation of face-to-face communication skills for people with dementia using a head-mounted system (2016)Google Scholar
  13. 13.
    Petric, F., Miklić, D., Kovačić, Z.: Probabilistic eye contact detection for the robot-assisted ASD diagnostic protocol. In: Lončarić, S., Cupec, R. (eds.) Proceedings of the Croatian Computer Vision Workshop, Year 4, pp. 3–8. Center of Excellence for Computer Vision, University of Zagreb, Osijek (October 2016)Google Scholar
  14. 14.
    Recasens, A., Khosla, A., Vondrick, C., Torralba, A.: Where are they looking? In: Advances in Neural Information Processing Systems (NIPS) (2015)Google Scholar
  15. 15.
    Recasens, A., Vondrick, C., Khosla, A., Torralba, A.: Following gaze in video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1435–1443 (2017)Google Scholar
  16. 16.
    Senju, A., Johnson, M.H.: The eye contact effect: mechanisms and development. Trends Cogn. Sci. 13(3), 127–134 (2009). Scholar
  17. 17.
    Smith, B.A., Yin, Q., Feiner, S.K., Nayar, S.K.: Gaze locking: passive eye contact detection for human-object interaction. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, pp. 271–280. ACM (2013)Google Scholar
  18. 18.
    Ye, Z., Li, Y., Liu, Y., Bridges, C., Rozga, A., Rehg, J.M.: Detecting bids for eye contact using a wearable camera. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–8. IEEE (2015)Google Scholar
  19. 19.
    Zhang, X., Sugano, Y., Bulling, A.: Everyday eye contact detection using unsupervised gaze target discovery. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 193–203. ACM (2017)Google Scholar
  20. 20.
    Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Appearance-based gaze estimation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4511–4520 (2015)Google Scholar
  21. 21.
    Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: It’s written all over your face: full-face appearance-based gaze estimation (2016).

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of InformaticsKyoto UniversityKyotoJapan

Personalised recommendations