Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Visual tracking of hands, faces and facial features of multiple persons


This paper presents an integrated approach for tracking hands, faces and specific facial features (eyes, nose, and mouth) in image sequences. For hand and face tracking, we employ a state-of-the-art blob tracker which is specifically trained to track skin-colored regions. In this paper we extend the skin color tracker by proposing an incremental probabilistic classifier, which can be used to maintain and continuously update the belief about the class of each tracked blob, which can be left-hand, right hand or face as well as to associate hand blobs with their corresponding faces. An additional contribution of this paper is related to the employment of a novel method for the detection and tracking of specific facial features within each detected facial blob which consists of an appearance-based detector and a feature-based tracker. The proposed approach is intended to provide input for the analysis of hand gestures and facial expressions that humans utilize while engaged in various conversational states with robots that operate autonomously in public places. It has been integrated into a system which runs in real time on a conventional personal computer which is located on a mobile robot. Experimental results confirm its effectiveness for the specific task at hand.

This is a preview of subscription content, log in to check access.


  1. 1

    Yang M.H., Kriegman D., Ahuja D.: Detecting faces in images: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 24(1), 34–58 (2002)

  2. 2

    Zabulis X., Baltzakis H., Argyros A.: Vision-based hand gesture recognition for human-computer interaction. In: Stefanides, C. (eds) The Universal Access Handbook, Human Factors and Ergonomics., pp. 34.1–34.30. Lawrence Erlbaum Associates, Inc. (LEA), New Jersey (2009)

  3. 3

    Jones M.J., Rehg J.M.: Statistical color models with application to skin detection. Int. J. Comput. Vis. 46(1), 81–96 (2002)

  4. 4

    Argyros, A.A., Lourakis, M.I.A.: Real-time tracking of multiple skin-colored objects with a possibly moving camera. In: Proceedings of the European Conference on Computer Vision, Prague, Czech Republic, May 2004, pp. 368–379

  5. 5

    Nickel, K., Seemann, E., Stiefelhagen, R.: 3D-tracking of head and hands for pointing gesture recognition in a human-robot interaction scenario. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, Seoul, Korea, May 2004, pp. 565–570

  6. 6

    Baltzakis, H., Argyros, A., Lourakis, M., Trahanias, P.: Tracking of human hands and faces through probabilistic fusion of multiple visual cues. In: Proceedings of the International Conference on Computer Vision Systems (ICVS), Santorini, Greece, May 2008, pp. 33–42

  7. 7

    Sigalas, M., Baltzakis, H., Trahanias, P.: Visual tracking of independently moving body and arms. In: Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS ’09), St. Louis, MO, USA, October 2009

  8. 8

    Deutscher, J., Blake, A., Reid, I.: Articulated body motion capture by annealed particle filtering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 126–133 (2000)

  9. 9

    Stenger B., Thayananthan A., Torr P.H.S., Cipolla R.: Model-based hand tracking using a hierarchical bayesian filter. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1372–1384 (2006)

  10. 10

    Baltzakis, H., Argyros, A.: Propagation of pixel hypotheses for multiple objects tracking. In: Proceedings of the International Symposium on Visual Computating (ISVC), Las Vegas, Nevada, USA, November 2009

  11. 11

    Zhang, X., Xu, Y., Du, L.: Locating facial features with color information. In: Proceedings of the Fourth International Conference on Signal Processing (ICSP), vol. 2, pp. 889–892 (1998)

  12. 12

    Hsu R.-L., Abdel-Mottaleb M., Jain A.K.: Face detection in color images. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 696–706 (2002)

  13. 13

    Sobottka K., Pitas I.: A novel method for automatic face segmentation, facial feature extraction and tracking. Signal Process. Image Commun. 12(3), 263–281 (1998)

  14. 14

    Nguyen, T., Nguyen, V., Kim, H.: Robust feature extraction for facial image quality assessment. In: Chung, Y., Yung, M. (eds.) Information Security Applications. Lecture Notes in Computer Science, vol. 6513, pp. 292–306. Springer, Berlin (2011)

  15. 15

    Yuille A.L., Hallinan P.W., Cohen D.S.: Feature extraction from faces using deformable templates. Int. J. Comput. Vis. 8(2), 99–111 (1992)

  16. 16

    Herpers, R., Sommer, G.: An attentive processing strategy for the analysis of facial features. In: Face recognition: From Theory to Applications, pp. 457–468 (1998)

  17. 17

    Pardas, M., Losada, M.: Facial parameter extraction system based on active contours. In Proceedings of the International Conference on Image Processing (ICIP’01), Thessaloniki, Greece, October 2001, vol. 1, pp. 1058–1061

  18. 18

    Kawaguchi T., Rizon M., Hidaka D.: Detection of eyes from human faces by hough transform and separability filter. Electron. Commun. Japan 88(5), 29–39 (2005)

  19. 19

    Cootes T., Edwards G., Taylor C.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)

  20. 20

    Cristinacce, D., Cootes, T.F., Scott, I.: A multi-stage approach to facial feature detection. In: 15th British Machine Vision Conference, pp. 231–240 (2004)

  21. 21

    Zuo F., de With P.H.: Facial feature extraction by a cascade of model-based algorithms. Signal Process Image Commun. 23(3), 194–211 (2008)

  22. 22

    Zhou Y., Li Y., Wu Z., Ge M.: Robust facial feature point extraction in color images. Eng. Appl. Artif. Intell. 24, 195–200 (2011)

  23. 23

    Kim, H.-C., Kim, D., Bang, S.-Y.: A pca mixture model with an efficient model selection method. In: Proceedings of the International Joint Conference on Neural Networks, 2001 (IJCNN ’01), vol. 1, pp. 430–435 (2001)

  24. 24

    Phimoltares, S., Lursinsap, C., Chamnongthai, K.: Locating essential facial features using neural visual model. In: Proceedings of the International Conference on Machine Learning and Cybernetics, 2002, vol. 4, pp. 1914–1919 (2002)

  25. 25

    Wilson P., Fernandez J.: Facial feature detection using haar classifiers. J. Comput. Sci. Coll. 21(4), 127–133 (2006)

  26. 26

    Viola P., Jones M.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

  27. 27

    Pateraki, M., Baltzakis, H., Kondaxakis, P., Trahanias, P.: Tracking of facial features to support human-robot interaction. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA ’09), pp. 3755–3760 (2009)

  28. 28

    Stauffer, C., Grimson, W.E.L.: Adaptive background mixture models for real-time tracking. In: Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR),Ft. Collins, USA, June 1999, pp. 2246–2252

  29. 29

    Tian, Y.: Evaluation of face resolution for expression analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04), pp. 82–89. IEEE Computer Society, New York (2004)

  30. 30

    Sohail A.S.Md, Bhattacharya P.: Detection of facial feature points using anthropometric face model. In: Damiani, E., Ytongnon, K., Schelkens, P., Dipanda, A., Legrand, L., Chbeir, R. (eds) Signal Processing for Image Enhancement and Multimedia Processing. Multimedia Systems and Applications, vol. 31, pp. 189–200. Springer US, USA (2008)

  31. 31

    Kanade, T., Cohn, J., Tian, Y.: Comprehensive database for facial expression analysis. In Proceedings of the IEEE 4th International Conference on Automatic Face and Gesture Recognition (FG ’00), Grenoble, France, pp. 46–53 (2000)

  32. 32

    Gunes, H., Piccardi, M.: A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In: Proceedings of ICPR 2006 the 18th International Conference on Pattern Recognition, Hong Kong, China, August 2006

  33. 33

    Jesorsky, O., Kirchberg, K.J., Frischholz, R.W.: Robust face detection using the hausdorff distance. In: Proceedings of the 3rd International Conference on Audio- and Video-based Biometric Person Authentication, Halmstad, Sweden, June 2001. Lecture Notes in Computer Science, vol. 2091, pp. 90–95. Springer, Berlin

Download references

Author information

Correspondence to Haris Baltzakis.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Baltzakis, H., Pateraki, M. & Trahanias, P. Visual tracking of hands, faces and facial features of multiple persons. Machine Vision and Applications 23, 1141–1157 (2012). https://doi.org/10.1007/s00138-012-0409-5

Download citation


  • Blob tracking
  • Hand and face tracking
  • Facial features
  • Incremental classification
  • Template matching