Bilinear Kernel Reduced Rank Regression for Facial Expression Synthesis

  • Dong Huang
  • Fernando De la Torre
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6312)


In the last few years, Facial Expression Synthesis (FES) has been a flourishing area of research driven by applications in character animation, computer games, and human computer interaction. This paper proposes a photo-realistic FES method based on Bilinear Kernel Reduced Rank Regression (BKRRR). BKRRR learns a high-dimensional mapping between the appearance of a neutral face and a variety of expressions (e.g. smile, surprise, squint). There are two main contributions in this paper: (1) Propose BKRRR for FES. Several algorithms for learning the parameters of BKRRR are evaluated. (2) Propose a new method to preserve subtle person-specific facial characteristics (e.g. wrinkles, pimples). Experimental results on the CMU Multi-PIE database and pictures taken with a regular camera show the effectiveness of our approach.


Facial Expression Neutral Face Facial Animation Alternate Little Square Ground Truth Image 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Gratch, J., Rickel, J., Andre, E., Cassell, J., Petajan, E., Badler, N.: Creating interactive virtual humans: some assembly required. IEEE Intelligent Systems 17, 54–63 (2002)Google Scholar
  2. 2.
    Choi, C., Aizawa, K., Harashima, H., Takebe, T.: Analysis and synthesis of facial image sequences in model-based image coding. IEEE Trans. CSVT 4, 257–275 (1994)Google Scholar
  3. 3.
    Breen, D., Lin, M.: Vision-based control of 3D facial animation. In: SCA, pp. 193–206 (2003)Google Scholar
  4. 4.
    Noh, J., Neumann, U.: Expression cloning. SIGGRAPH 1, 277–288 (2001)Google Scholar
  5. 5.
    Keeve, E., Girod, S., Kikinis, R., Girod, B.: Deformable modeling of facial tissue for craniofacial surgery simulation. Computer Aided Surgery 3, 223–228 (1998)CrossRefGoogle Scholar
  6. 6.
    Liu, Z., Shan, Y., Zhang, Z.: Expressive expression mapping with ratio images. In Proc. of Ann. Conf. on Computer Graphics and Interactive Techniques (2001)Google Scholar
  7. 7.
    Zhang, Q., Liu, Z., Guo, B., Shum, H.: Geometry-driven photorealistic facial expression synthesis. IEEE Trans. VCG 12, 48–60 (2006)Google Scholar
  8. 8.
    Chung, K.: Gross Anatomy (Board Review). Lippincott Williams & Wilkins, Hagerstown (2005)Google Scholar
  9. 9.
    Nguyen, M., Lalonde, J., Efros, A., De la Torre, F.: Image-based shaving. Computer Graphics Forum (Eurographics) 27, 627–635 (2008)CrossRefGoogle Scholar
  10. 10.
    Vasilescu, M., Terzopoulos, D.: Multilinear analysis of image ensembles: Tensorfaces. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 447–460. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  11. 11.
    Tenenbaum, J., Freeman, W.: Separating style and content with bilinear models. Neural Computation 12, 1247–1283 (2000)CrossRefGoogle Scholar
  12. 12.
    Wang, H., Ahuja, N.: Facial expression decomposition. In: ICCV (2003)Google Scholar
  13. 13.
    Abboud, B., Davoine, F.: Appearance factorization for facial expression analysis. In: BMVC (2004)Google Scholar
  14. 14.
    Vlasic, D., Brand, M., Pfister, H., Popovic, J.: Face transfer with multiliner models. ACM Trans. Graphics 24, 426–433 (2005)CrossRefGoogle Scholar
  15. 15.
    Macedo, I., Brazil, E., Velho, L.: Expression transfer between photographs through multilinear aam’s. In: SIBGRAPI, pp. 239–246 (2006)Google Scholar
  16. 16.
    Bettinger, F., Cootes, T., Taylor, C.: Modelling facial behaviours. In: BMVC, vol. 2 (2002)Google Scholar
  17. 17.
    Zalewski, L., Gong, S.: Synthesis and recognition of facial expressions in virtual 3D views. In: AFGR (2004)Google Scholar
  18. 18.
    Chang, Y., Hu, C., Ferisa, R., Turk, M.: Manifold based analysis of facial expression. Image and Vision Computing 24, 605–614 (2005)CrossRefGoogle Scholar
  19. 19.
    Kouadio, C., Poulin, P., Lachapelle, P.: Real-time facial animation based upon a bank of 3D facial expressions. In: Proc. of Computer Animation (1998)Google Scholar
  20. 20.
    Pighin, F., Szeliski, R., Salesin, D.: Resynthesizing facial animation through 3D model-based tracking. In: ICCV (1999)Google Scholar
  21. 21.
    Pyun, H., Kim, Y., Chae, W., Kang, H., Shin, S.: An example-based approach for facial expression cloning. In: SIGGRAPH/Eurographics SCA, pp. 167–176 (2003)Google Scholar
  22. 22.
    Parke, F.I., Waters, K.: Computer facial animation. AK Peters, Wellesley (1996)Google Scholar
  23. 23.
    Anderson, T.: Estimating linear restrictions on regression coefficients for multivariate normal distributions. Annals of Mathematics Statistics 12, 327–351 (1951)CrossRefGoogle Scholar
  24. 24.
    Scharf, L.: The SVD and reduced rank signal processing. Signal Processing 25, 113–133 (2002)CrossRefGoogle Scholar
  25. 25.
    Diamantaras, K.: Principal Component Neural Networks (Therory and Applications). John Wiley & Sons, Chichester (1996)Google Scholar
  26. 26.
    De la Torre, F., Black, M.: Dynamic coupled component analysis. In: CVPR (2001)Google Scholar
  27. 27.
    Baldi, P., Hornik, K.: Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks 2, 53–58 (1989)CrossRefGoogle Scholar
  28. 28.
    Bathe, K., Wilson, E.: Numerical Methods in Finite Element. Prentice-Hall, Englewood Cliffs (1971)Google Scholar
  29. 29.
    De la Torre, F., Gross, R., Baker, S., Kumar, V.: Representational oriented component analysis for face recognition with one sample image per training class. In: CVPR (2005)Google Scholar
  30. 30.
    Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: The CMU multi-pose, illumination, and expression (multi-pie) face database. Tech. rep., Robotics Institute, Carnegie Mellon University,TR-07-08 (2007)Google Scholar
  31. 31.
    Weinberger, K., Tesauro, G.: Metric learning for kernel regression. In: AISTATS (2007)Google Scholar
  32. 32.
    Huang, D., De la Torre, F.: Bilinear kernel reduced rank regression for facial expression synthesis. Tech. rep., Robotics Institute, Carnegie Mellon University,TR-10-23 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Dong Huang
    • 1
  • Fernando De la Torre
    • 1
  1. 1.Robotics InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations