Face Tracking and Recognition in Video

  • Rama ChellappaEmail author
  • Ming Du
  • Pavan Turaga
  • Shaohua Kevin Zhou


In this chapter, we describe the utility of videos in enhancing performance of image-based recognition tasks. We discuss a joint tracking-recognition framework that allows for using the motion information in a video to better localize and identify the person in the video using still galleries. We discuss how to jointly capture facial appearance and dynamics to obtain a parametric representation for video-to-video recognition. We discuss recognition in multi-camera networks where the probe and gallery both consist of multi-camera videos. Concluding remarks and directions for future research are provided.


Face Recognition Video Sequence Spherical Harmonics Grassmann Manifold Dynamic Texture 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



Supported by a MURI Grant N00014-08-1-0638 from the Office of Naval Research. The authors would like to thank Dr. Aswin Sankaranarayanan for helpful discussions related to Sect. 13.5.


  1. 1.
    Aggarwal, G., Roy-Chowdhury, A., Chellappa, R.: A system identification approach for video-based face recognition. In: International Conference on Pattern Recognition, Cambridge, UK, August 2004 Google Scholar
  2. 2.
    Anderson, B., Moore, J.: Optimal Filtering. Prentice Hall, Englewood Cliffs (1979) zbMATHGoogle Scholar
  3. 3.
    Arandjelovic, O., Cipolla, R.: Face recognition from video using the generic shape-illumination manifold. In: European Conference on Computer Vision, pp. 27–40 (2006) Google Scholar
  4. 4.
    Arandjelovic, O., Shakhnarovich, G., Fisher, J., Cipolla, R., Darrell, T.: Face recognition with image sets using manifold density divergence. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 581–588, San Diego, USA, June 2005 Google Scholar
  5. 5.
    Basri, R., Jacobs, D.: Lambertian reflectance and linear subspaces. In: Proceedings of IEEE International Conference on Computer Vision, vol. 2, pp. 383–390 (2001) Google Scholar
  6. 6.
    Bissacco, A., Chiuso, A., Ma, Y., Soatto, S.: Recognition of human gaits. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 52–57, Hawaii, USA, December 2001 Google Scholar
  7. 7.
    Brocker, T., Dieck, T.: Representations of Compact Lie Groups. Springer, Berlin (2003) Google Scholar
  8. 8.
    Chikuse, Y.: Statistics on Special Manifolds. Lecture Notes in Statistics. Springer, New York (2003) zbMATHGoogle Scholar
  9. 9.
    Choudhury, T., Clarkson, B., Jebara, T., Pentland, A.: Multimodal person recognition using unconstrained audio and video. In: Proc. of Intl. Conf. on Audio- and Video-Based Person Authentication, pp. 176–181 (1999) Google Scholar
  10. 10.
    Cock, K.D., Moor, B.D.: Subspace angles between ARMA models. Syst. Control Lett. 46, 265–270 (2002) zbMATHCrossRefGoogle Scholar
  11. 11.
    Doretto, G., Chiuso, A., Wu, Y.N., Soatto, S.: Dynamic textures. Int. J. Comput. Vis. 51(2), 91–109 (2003) zbMATHCrossRefGoogle Scholar
  12. 12.
    Doucet, A., Godsill, S., Andrieu, C.: On sequential Monte Carlo sampling methods for Bayesian filtering. Stat. Comput. 10(3), 197–209 (2000) CrossRefGoogle Scholar
  13. 13.
    Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20(2), 303–353 (1999) MathSciNetCrossRefGoogle Scholar
  14. 14.
    Fan, W., Yeung, D.-Y.: Locally linear models on face appearance manifolds with application to dual-subspace based classification. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1384–1390, New York, NY, USA, June 2006 Google Scholar
  15. 15.
    Hamm, J., Lee, D.D.: Grassmann discriminant analysis: a unifying view on subspace-based learning. In: International Conference on Machine Learning, pp. 376–383, Helsinki, Finland, June 2008 Google Scholar
  16. 16.
    Kazhdan, M., Funkhouser, T., Rusinkiewicz, S.: Rotation invariant spherical harmonic representation of 3d shape descriptors. In: Proceedings of Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, pp. 156–164 (2003) Google Scholar
  17. 17.
    Kim, T.K., Kittler, J., Cipolla, R.: Discriminative learning and recognition of image set classes using canonical correlations. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 1005–1018 (2007) CrossRefGoogle Scholar
  18. 18.
    Kitagawa, G.: Monte Carlo filter and smoother for non-Gaussian nonlinear state space models. J. Comput. Graph. Stat. 5, 1–25 (1996) MathSciNetCrossRefGoogle Scholar
  19. 19.
    Lee, K.-C., Ho, J., Kriegman, D.J.: Acquiring linear subspaces for face recognition under variable lighting. IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 684–698 (2005) CrossRefGoogle Scholar
  20. 20.
    Lee, K.-C., Ho, J., Yang, M.-H., Kriegman, D.J.: Visual tracking and recognition using probabilistic appearance manifolds. Comput. Vis. Image Underst. 99(3), 303–331 (2005) CrossRefGoogle Scholar
  21. 21.
    Li, Y., Gong, S., Liddell, H.: Video-based online face recognition using identity surfaces. In: Proceedings of the IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, pp. 40–46 (2001) CrossRefGoogle Scholar
  22. 22.
    Liu, J., Chen, R.: Sequential Monte Carlo for dynamic systems. J. Am. Stat. Assoc. 93, 1031–1041 (1998) Google Scholar
  23. 23.
    Liu, X., Chen, T.: Video-based face recognition using adaptive hidden Markov model. In: Proceedings of Computer Vision and Pattern Recognition, vol. 1, pp. 340–345 (2003) Google Scholar
  24. 24.
    Liu, X., Chen, T., Thornton, S.M.: Eigenspace updating for non-stationary process and its application to face recognition. Pattern Recognit. 36, 1945–1959 (2003) zbMATHCrossRefGoogle Scholar
  25. 25.
    Moghaddam, B., Pentland, A.: Probabilistic visual learning for object representation. IEEE Trans. Pattern Anal. Mach. Intell. 19, 696–710 (1997) CrossRefGoogle Scholar
  26. 26.
    O’Toole, A.J., Roark, D., Abdi, H.: Recognizing moving faces: a psychological and neural synthesis. Trends Cogn. Sci. 6, 261–266 (2002) CrossRefGoogle Scholar
  27. 27.
    Overschee, P.V., Moor, B.D.: Subspace algorithms for the stochastic identification problem. Automatica 29(3), 649–660 (1993) MathSciNetzbMATHCrossRefGoogle Scholar
  28. 28.
    Park, U., Jain, A.K., Ross, A.: Face recognition in video: adaptive fusion of multiple matchers. In: Proceedings of IEEE Computer Society Workshop on Biometrics (In Conjunction with CVPR), pp. 1–8 (2007) Google Scholar
  29. 29.
    Philipps, P., Moon, H., Rivzi, S., Ross, P.: The Feret evaluation methodology for face-recognition algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1090–1104 (2000) CrossRefGoogle Scholar
  30. 30.
    Ramamoorthi, R.: Analytic pca construction for theoretical analysis of lighting variability in images of a Lambertian object. IEEE Trans. Pattern Anal. Mach. Intell. 24(10), 1322–1333 (2002) CrossRefGoogle Scholar
  31. 31.
    Roark, D.A., Barrett, S.E., O’Toole, A.J., Abdi, H.: Learning the moves: The effect of familiarity and facial motion on person recognition across large changes in viewing format. Perception 761–773 (2006) Google Scholar
  32. 32.
    Ross, A.A., Nandakumar, K., Jain, A.K.: Handbook of Multibiometrics. International Series on Biometrics. Springer, New York (2006) Google Scholar
  33. 33.
    Saisan, P., Doretto, G., Wu, Y.N., Soatto, S.: Dynamic texture recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 58–63, Hawaii, USA, December 2001 Google Scholar
  34. 34.
    Shakhnarovich, G., Fisher, J.W., Darrell, T.: Face recognition from long-term observations. In: Proceedings of the European Conference on Computer Vision, vol. 3, pp. 851–865, May 2002 Google Scholar
  35. 35.
    Turaga, P., Veeraraghavan, A., Chellappa, R.: Statistical analysis on Stiefel and Grassmann manifolds with applications in computer vision. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8, Alaska, USA, June 2008 CrossRefGoogle Scholar
  36. 36.
    Turaga, P., Veeraraghavan, A., Srivastava, A., Chellappa, R.: Statistical analysis on manifolds and its applications to video analysis. In: Schonfeld, D., Shan, C., Tao, D., Wang, L. (eds.) Video Search and Mining. Studies in Computational Intelligence, Chap. 5. Springer, Berlin (2010) Google Scholar
  37. 37.
    Veeraraghavan, A., Roy-Chowdhury, A., Chellappa, R.: Matching shape sequences in video with an application to human movement analysis. IEEE Trans. Pattern Anal. Mach. Intell. 27(12), 1896–1909 (2005) CrossRefGoogle Scholar
  38. 38.
    Xu, Y., Roy-Chowdhury, A., Patel, K.: Pose and illumination invariant face recognition in video. In: Proceedings of IEEE Computer Society Workshop on Biometrics (In Conjunction with CVPR), pp. 1–7 (2007) Google Scholar
  39. 39.
    Zhang, L., Samaras, D.: Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics. IEEE Trans. Pattern Anal. Mach. Intell. 28(3), 351–363 (2006) CrossRefGoogle Scholar
  40. 40.
    Zhao, W.Y., Chellappa, R., Rosenfeld, A., Phillips, P.: Face recognition: a literature survey. ACM Comput. Surv. 35 (2003) Google Scholar
  41. 41.
    Zhou, S.K., Chellappa, R.: From sample similarity to ensemble similarity: probabilistic distance measures in reproducing kernel Hilbert space. IEEE Trans. Pattern Anal. Mach. Intell. 28(6), 917–929 (2006) CrossRefGoogle Scholar
  42. 42.
    Zhou, S., Krueger, V., Chellappa, R.: Probabilistic recognition of human faces from video. Comput. Vis. Image Underst. 91, 214–245 (2003) CrossRefGoogle Scholar
  43. 43.
    Zhou, S., Chellappa, R., Moghaddam, B.: Visual tracking and recognition using appearance-adaptive models in particle filters. IEEE Trans. Image Process. (2004) Google Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Rama Chellappa
    • 1
    Email author
  • Ming Du
    • 1
  • Pavan Turaga
    • 1
  • Shaohua Kevin Zhou
    • 2
  1. 1.Department of Electrical and Computer Engineering, Center for Automation ResearchUniversity of MarylandCollege ParkUSA
  2. 2.Siemens Corporate ResearchPrincetonUSA

Personalised recommendations