Facial Action Tracking

  • Jörgen AhlbergEmail author
  • Igor S. Pandzic


This chapter explains the basics of parametric face models used for face and facial action tracking as well as fundamental strategies and methodologies for tracking. A few tracking algorithms serving as pedagogical examples are described in more detail.


Feature Point Extended Kalman Filter Face Model Facial Action Active Appearance Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Ahlberg, J.: An active model for facial feature tracking. In: Proceedings of the International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Tampere, Finland, pp. 63–67 (2001) Google Scholar
  2. 2.
    Ahlberg, J.: Candide 3—an updated parameterized face. Technical Report LiTH-ISY-R-2326, Linköping University, Sweden (2001) Google Scholar
  3. 3.
    Ahlberg, J.: Model-based coding—extraction, coding and evaluation of face model parameters. PhD thesis, Linköping University, Sweden (2002) Google Scholar
  4. 4.
    Ahlberg, J.: An active model for facial feature tracking. EURASIP J. Appl. Signal Process. 2002(6), 566–571 (2002) CrossRefGoogle Scholar
  5. 5.
    Ahlberg, J., Forchheimer, R.: Face tracking for model-based coding and face animation. Int. J. Imaging Syst. Technol. 13(1), 8–22 (2003) CrossRefGoogle Scholar
  6. 6.
    Azerbayejani, A., Pentland, A.: Recursive estimation of motion, structure, and focal length. IEEE Trans. Pattern Anal. Mach. Intell. 17(6), 562–575 (1995) CrossRefGoogle Scholar
  7. 7.
    Basu, S., Essa, I., Pentland, A.: Motion regularization for model-based head-tracking. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), Vienna, Austria, pp. 611–616 (1996) Google Scholar
  8. 8.
    Black, M.J., Yacoob, Y.: Recognizing facial expressions in image sequences using local parameterized models of image motion. Int. J. Comput. Vis. 25(1), 23–48 (1997) CrossRefGoogle Scholar
  9. 9.
    Caunce, A., Cristinacce, D., Taylor, C., Cootes, T.: Locating facial features and pose estimation using a 3D shape model. In: Proceedings of the International Symposium on Visual Computing, Las Vegas, USA (2009) Google Scholar
  10. 10.
    Craw, I., Cameron, P.: Parameterising images for recognition and reconstruction. In: Proceedings of the British Machine Vision Conference (BMVC), pp. 367–370. Springer, London (1991) Google Scholar
  11. 11.
    DeCarlo, D., Metaxas, D.: Optical flow constraints on deformable models with applications to face tracking. Int. J. Comput. Vis. 38(72), 99–127 (2000) zbMATHCrossRefGoogle Scholar
  12. 12.
    Dornaika, F., Ahlberg, J.: Face model adaptation using robust matching and the active appearance algorithm. In: Proceedings of the IEEE International Workshop on Applications of Computer Vision (WACV), pp. 3–7, Orlando, FL (2002) Google Scholar
  13. 13.
    Dornaika, F., Ahlberg, J.: Face and facial feature tracking using deformable models. Int. J. Image Graph. 4(3), 499–532 (2004) CrossRefGoogle Scholar
  14. 14.
    Dornaika, F., Ahlberg, J.: Fast and reliable active appearance model search for 3D face tracking. IEEE Trans. Syst. Man Cybern., Part B, Cybern. 34(4), 1838–1853 (2004) CrossRefGoogle Scholar
  15. 15.
    Dornaika, F., Davoine, F.: On appearance based face and facial action tracking. IEEE Trans. Circuits Syst. Video Technol. 16(9), 1107–1124 (2006) CrossRefGoogle Scholar
  16. 16.
    Eisert, P., Girod, B.: Model-based estimation of facial expression parameters from image sequences. In: Proceedings of the International Conference on Image Processing (ICIP), pp. 418–421, Santa Barbara, CA, USA (1997) CrossRefGoogle Scholar
  17. 17.
    Ekman, P., Friesen, W.V.: Facial action coding system. Consulting Psychologists Press, Palo Alto (1977) Google Scholar
  18. 18.
    Fanelli, G., Fratarcangeli, M.: A non-invasive approach for driving virtual talking heads from real facial movements. In: Proceedings of the IEEE 3DTV Conference, Kos, Greece, May 2007 Google Scholar
  19. 19.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981) MathSciNetCrossRefGoogle Scholar
  20. 20.
    Fratarcangeli, M., Schaerf, M., Forchheimer, R.: Facial motion cloning with radial basis functions in mpeg-4 fba. Graph. Models 69(2), 106–118 (2007) CrossRefGoogle Scholar
  21. 21.
    Gelb, A.: Applied Optimal Estimation. MIT Press, Cambridge (1974) Google Scholar
  22. 22.
    Gokturk, S.B., Bouguet, J.Y., Grzeszczuk, R.: A data-driven model for monocular face tracking. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 701–708, Vancouver, Canada (2001) Google Scholar
  23. 23.
    Heckbert, P.: Survey of texture mapping. IEEE Comput. Graph. Appl. (1986) Google Scholar
  24. 24.
    Ingemars, N., Ahlberg, J.: Feature-based face tracking using extended Kalman filtering. In: Proceedings of the Swedish Symposium on Image Analysis (SSBA), pp. 141–144, Linköping, Sweden (2007) Google Scholar
  25. 25.
    InterFace. European Union 5th Framework Programme project 2000–2002 Google Scholar
  26. 26.
    Jebara, T., Pentland, A.: Parameterized structure from motion for 3D adaptive feedback tracking of faces. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 144–150. San Juan, Puerto Rico (1997) CrossRefGoogle Scholar
  27. 27.
    Kahler, K., Haber, J., Seidel, H.-P.: Geometry-based muscle modeling for facial animation. In: Proceedings of Graphics Interface, pp. 37–46 (2001) Google Scholar
  28. 28.
    Kalman, R.E.: A new approach to linear filtering and prediction problems. J. Basic Eng. 82D, 25–45 (1960) Google Scholar
  29. 29.
    Kavan, L., Zara, J.: Spherical blend skinning: A real-time deformation of articulated models. In: Proceedings I3D, New York: ACM SIGGRAPH, pp. 9–16 Google Scholar
  30. 30.
    Kirby, M., Sirovich, L.: Application of the Karhunen–Loeve procedure for the characterization of human faces. IEEE Trans. Pattern Anal. Mach. Intell. 12(1), 103–108 (1990) CrossRefGoogle Scholar
  31. 31.
    La Cascia, M., Sclaroff, S., Athitsos, V.: Fast, reliable head tracking under varying illumination: An approach based on registration of texture-mapped 3D models. IEEE Trans. Pattern Anal. Mach. Intell. 22(4), 322–336 (2000) CrossRefGoogle Scholar
  32. 32.
    Lefevre, S., Odobez, J.-M.: Structure and appearance features for robust 3d facial actions tracking. In: International Conference on Multimedia and Expo, New York, NY, USA (2009) Google Scholar
  33. 33.
    Li, H., Roivanen, P., Forchheimer, R.: 3-D motion estimation in model-based facial image coding. IEEE Trans. Pattern Anal. Mach. Intell. 15(6), 545–555 (1993) CrossRefGoogle Scholar
  34. 34.
    Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI), pp. 674–679, April 1981 Google Scholar
  35. 35.
    Magnenat-Thalmann, N., Primeau, N.E., Thalmann, D.: Abstract muscle actions procedures for human face animation. Vis. Comput. 3(5), 290–297 (1988) CrossRefGoogle Scholar
  36. 36.
    Malciu, M., Prêteux, F.: A robust model-based approach for 3d head tracking in video sequences. In: Proceedings of the International Conference on Face and Gesture Recognition, pp. 169–174. Grenoble, France (2000) CrossRefGoogle Scholar
  37. 37.
    Matthews, I., Baker, S.: Active appearance models revisited. Int. J. Comput. Vis. 60(2), 135–164 (2004) CrossRefGoogle Scholar
  38. 38.
    Messer, K., Matas, J., Kittler, J., Luettin, J., Maitre, G.: XM2VTSDB: The Extended M2VTS Database. In: Proceedings of the Second International Conference on Audio and Video-based Biometric Person Authentication (AVBPA), Washington, DC (1999) Google Scholar
  39. 39.
    Mohr, A., Gleicher, M.: Building efficient, accurate character skins from examples. In: Proceedings of SIGGRAPH, pp. 562–568, San Diego, CA, 27–31 July 2003. ACM, New York (2003) Google Scholar
  40. 40.
    Morency, L.-P., Whitehill, J., Movellan, J.R.: Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation. In: Proceedings of the International Conference on Face and Gesture Recognition, pp. 1–8, Amsterdam, The Netherlands (2008) Google Scholar
  41. 41.
    Moving Picture Experts Group. ISO/IEC 14496—MPEG-4 international standard (1999).
  42. 42.
    Nelder, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7(4), 308–313 (1965) zbMATHGoogle Scholar
  43. 43.
    Noh, J., Neumann, U.: Expression cloning. In: Proceedings of SIGGRAPH, Los Angeles, CA (2001) Google Scholar
  44. 44.
  45. 45.
    Pandzic, I.S.: Life on the web. Softw. Focus 2(2), 52–59 (2001) CrossRefGoogle Scholar
  46. 46.
    Pandzic, I.S.: Facial motion cloning. Graph. Models 65(6), 385–404 (2003) CrossRefGoogle Scholar
  47. 47.
    Pandzic, I.S., Forchheimer, R. (eds.): MPEG-4 Facial Animation: The Standard, Implementations, and Applications. Wiley, Chichester (2002) Google Scholar
  48. 48.
    Parke, F.I.: Parameterized models for facial animation. IEEE Comput. Graph. Appl. 2(9), 61–68 (1982) CrossRefGoogle Scholar
  49. 49.
    Parke, F.I., Waters, K.: Computer Facial Animation, 2nd edn. AK Peters, Wellesley (2008) Google Scholar
  50. 50.
    Pighin, F., Szeliski, S., Salesin, S.: Resynthesizing facial animation through 3d model-based tracking. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 143–150, Kerkyra, Greece (1999) CrossRefGoogle Scholar
  51. 51.
    Roivanen, P.: Motion estimation in model-based coding of human faces. Licentiate thesis, Linköping University, Sweden (1990) Google Scholar
  52. 52.
    Rydfalk, M.: Candide, a parameterized face. Technical Report LiTH-ISY-I-866, Linköping University, Sweden (1987) Google Scholar
  53. 53.
    Sirovich, L., Kirby, M.: Low-dimensional procedure for the characterization of human faces. J. Opt. Soc. Am. 4(3), 519–524 (1987) CrossRefGoogle Scholar
  54. 54.
    Sonka, M., Hlavac, V., Boyle, R.: Image Processing, Analysis, and Machine Vision. PWS, Pacific Grove (1998) Google Scholar
  55. 55.
    Ström, J.: Model-based head tracking and coding. PhD thesis, Linköping University, Sweden (2002) Google Scholar
  56. 56.
    Ström, J.: Model-based real-time head tracking. EURASIP J. Appl. Signal Process. 2002(10), 1039–1052 (2002) zbMATHCrossRefGoogle Scholar
  57. 57.
    Ström, J., Jebara, T., Basu, S., Pentland, A.: Real time tracking and modeling of faces: an EKF-based analysis by synthesis approach. In: IEEE ICCV Workshop on Modelling People, Kerkyra, Greece (1999) Google Scholar
  58. 58.
    Tao, H., Lopez, R., Huang, T.: Tracking of face features using probabilistic network. In: Proceedings of the International Conference on Face and Gesture Recognition, Nara, Japan (1998) Google Scholar
  59. 59.
    Terzopoulos, D., Waters, K.: Physically-based facial modeling, analysis and animation. J. Vis. Comput. Animat. 1(4), 73–80 (1990) CrossRefGoogle Scholar
  60. 60.
    Turk, M., Pentland, A.: Eigenfaces for recognition. Int. J. Cogn. Neurosci. 3(1), 71–86 (1991) CrossRefGoogle Scholar
  61. 61.
    Waters, K.: A muscle model for animating three-dimensional facial expressions. Comput. Graph. 21(4), 17–24 (1987) MathSciNetCrossRefGoogle Scholar
  62. 62.
    Welsh, B.: Model-based coding of images. PhD thesis, British Telecom Research Lab (1991) Google Scholar
  63. 63.
    Wiles, C.S., Maki, A., Matsuda, N.: Hyperpatches for 3D model acquisition. IEEE Trans. Pattern Anal. Mach. Intell. 23(12), 1391–1403 (2001) CrossRefGoogle Scholar
  64. 64.
    Xiao, J., Moriyama, T., Kanade, T., Cohn, J.: Robust full motion recovery of head for facial expression analysis. Int. J. Imaging Syst. Technol. 13, 85–94 (2003) CrossRefGoogle Scholar
  65. 65.
    Zhou, S., Chellappa, R., Moghaddam, B.: Visual tracking and recognition using appearance-adaptive models in particle filters. IEEE Trans. Image Process. 13(11), 1491–1506 (2004) CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  1. 1.Division of Information SystemsSwedish Defence Research Agency (FOI)LinköpingSweden
  2. 2.Faculty of Electrical Engineering and ComputingUniversity of ZagrebZagrebCroatia

Personalised recommendations