Facial Keypoint Detection Using Deep Learning and Computer Vision

  • Venkata Sai Rishita MiddiEmail author
  • Kevin Job Thomas
  • Tanvir Ahmed Harris
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 941)


With the advent of Computer Vision, research scientists across the world are working constantly working to expedite the advancement of Facial Landmarking system. It is a paramount step for various Facial processing operations. The applications range from facial recognition to Emotion recognition. These days, we have systems that identify people in images and tag them accordingly. There are mobile applications which identify the emotion of a person in an image and return the appropriate emoticon. The systems are put to use for applications ranging from personal security to national security. In this work, we have agglomerated computer vision techniques and Deep Learning algorithms to develop an end-to-end facial keypoint recognition system. Facial keypoints are discrete points around eyes, nose, mouth on any face. The implementation begins from Investigating OpenCV, pre-processing of images and Detection of faces. Further, a convolutional Neural network is trained for detecting eyes, nose and mouth. Finally, the CV pipeline is completed by the two parts mentioned above.


Pipeline Edges De-noising Blurring Detection Losses Landmark Recognition Robust 



We would like to thank Dr. Suresh D, Department of Electronics and Communication, RNS Institute of Technology for his technical and writing assistance.


  1. 1.
    Valstar, M., Martinez, B., Binefa, X., Pantic, M.: Facial point detection using boosted regression and graph models. In: Proceedings of Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, pp. 2729–2736 (2010)Google Scholar
  2. 2.
    Ding, L., Martinez, A.M.: Features versus context: an approach for precise and detailed detection and delineation of faces and facial features. IEEE Trans. Pattern Anal. Mach. Intell. 32(11), 2022–2038 (2010)CrossRefGoogle Scholar
  3. 3.
    Arca, S., Campadelli, P., Lanzarotti, R.: A face recognition system based on automatically determined facial fiducial points. Pattern Recogn. 39, 432–443 (2006)CrossRefGoogle Scholar
  4. 4.
    Belhumeur, P.N., Jacobs, D.W., Kriegman, D.J., Kumar, N.: Localizing parts of faces using a consensus of exemplars. In: Proceedings of Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, pp. 545–552 (2011)Google Scholar
  5. 5.
    Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 397–403 (2013)Google Scholar
  6. 6.
    Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Incremental face alignment in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1859–1866 (2014)Google Scholar
  7. 7.
    Wu, Y., Ji, Q.: Facial landmark detection: a literature survey. Int. J. Comput. Vis. Received Nov 2016. Accepted Apr 2018Google Scholar
  8. 8.
    Vong, C.M., Tai, K.I., Pun, C.M., Wong, P.K.: Fast and accurate face detection by sparse Bayesian extreme learning machine. Neural Comput. Appl. 26(5), 1149–1156 (2015)CrossRefGoogle Scholar
  9. 9.
    Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: Proceedings of the International Conference on Pattern Recognition, pp. 586–591 (1991)Google Scholar
  10. 10.
    Negi, R.S., Garg, R.: Face recognition using hausdroff distance as a matching algorithm. Int. J. Sci. Res. 4(9) (2016)Google Scholar
  11. 11.
    Dewan, M.A.A., Qiao, D., Lin, F., Wen, D.: An approach to improving single sample face recognition using high confident tracking trajectories. In: Canadian Conference on Artificial Intelligence, pp. 115–121. Springer, Cham (2016)Google Scholar
  12. 12.
    Shi, X., Wu, J., Ling, X., Zheng, Q., Pan, X., Zhao, Z.: Real-time face recognition method based on the threshold determination of the positive face sequence. In: Proceedings of the 22nd International Conference on Industrial Engineering and Engineering Management 2015, pp. 125–136. Atlantis Press (2016)Google Scholar
  13. 13.
    Goswami, G., Vatsa, M., Singh, R.: Face recognition with RGB-D images using kinect. In: Face Recognition Across the Imaging Spectrum, pp. 281–303. Springer, Cham (2016)Google Scholar
  14. 14.
    Dahal, B., Alsadoon, A., Prasad, P.W.C., Elchouemi, A.: Incorporating skin color for improved face detection and tracking system. In: 2016 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI), pp. 173–176. IEEE (2016)Google Scholar
  15. 15.
    Aghaei, M., Dimiccoli, M., Radeva, P.: Multi-face tracking by extended bag-of-tracklets in egocentric photostreams. Comput. Vis. Image Underst. 149, 146–156 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Venkata Sai Rishita Middi
    • 1
    Email author
  • Kevin Job Thomas
    • 2
  • Tanvir Ahmed Harris
    • 3
  1. 1.RNS Institute of TechnologyBangaloreIndia
  2. 2.VITVelloreIndia
  3. 3.IIT BombayMumbaiIndia

Personalised recommendations