Joint Face Alignment: Rescue Bad Alignments with Good Ones by Regularized Re-fitting

  • Xiaowei Zhao
  • Xiujuan Chai
  • Shiguang Shan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7573)


Nowadays, more and more applications need to jointly align a set of facial images from one specific person, which forms the so-called joint face alignment problem. To address this problem, in this paper, starting from an initial face alignment results, we propose to enhance the alignments by a fundamentally novel idea: rescuing the bad alignments with their well-aligned neighbors. In our method, a discriminative alignment evaluator is well designed to assess the initial face alignments and separate the well-aligned images from the badly-aligned ones. To correct the bad ones, a robust regularized re-fitting algorithm is proposed by exploiting the appearance consistency between the badly-aligned image and its k well-aligned nearest neighbors. Experiments conducted on faces in the wild demonstrate that our method greatly improves the initial face alignment results of an off-the-shelf facial landmark locator. In addition, the effectiveness of our method is validated through comparing with other state-of-the-art methods in joint face alignment under complex conditions.


Facial Image Appearance Model Alignment Result Active Appearance Model Active Shape Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Wang, R., Shan, S., Chen, X., Gao, W.: Manifold-manifold distance with application to face recognition based on image set. In: CVPR (2008)Google Scholar
  2. 2.
    Zhu, C., Wen, F., Sun, J.: A rank-order distance based clustering algorithm for face tagging. In: CVPR, pp. 481–488 (2011)Google Scholar
  3. 3.
    Kemelmacher-Shlizerman, I., Seitz, S.M.: Face reconstruction in the wild. In: ICCV, pp. 1746–1753 (2011)Google Scholar
  4. 4.
    Learned-Miller, E.: Data driven image models through continuous joint alignment. IEEE T-PAMI 28, 236–250 (2006)CrossRefGoogle Scholar
  5. 5.
    Huang, G.B., Jain, V., Learned-Miller, E.: Unsupervised joint alignment of complex images. In: ICCV (2007)Google Scholar
  6. 6.
    Cox, M., Sridharan, S., Lucey, S., Cohn, J.: Least squares congealing for unsupervised alignment of images. In: CVPR (2008)Google Scholar
  7. 7.
    Peng, Y., Ganesh, A., Wright, J., Xu, W.: RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images. In: CVPR, pp. 763–770 (2010)Google Scholar
  8. 8.
    Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P.N., Nayar, S.K.: Face Swapping: Automatically replacing faces in photographs. ACM SIGGRAPH (2008)Google Scholar
  9. 9.
    Saragih, J., Lucey, S., Cohn, J.: Real-time avatar animation from a single image. In: FG (2011)Google Scholar
  10. 10.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models: Their training and application. CVIU 61, 38–59 (1995)Google Scholar
  11. 11.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active Appearance Models. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 484–498. Springer, Heidelberg (1998)Google Scholar
  12. 12.
    Cootes, T.F., Marsland, S., Twining, C.J., Smith, K., Taylor, C.J.: Groupwise Diffeomorphic Non-rigid Registration for Automatic Model Building. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 316–327. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  13. 13.
    Cootes, T.F., Twining, C.J., Petrovic, V., Schestowitz, R.: Groupwise construction of appearance models using piece-wise affine deformations. In: BMVC, pp. 879–888 (2005)Google Scholar
  14. 14.
    Liu, X.: Generic face alignment using boosted appearance model. In: CVPR, pp. 1079–1088 (2007)Google Scholar
  15. 15.
    Wu, H., Liu, X., Doretto, G.: Face alignment via boosted ranking model. In: CVPR (2008)Google Scholar
  16. 16.
    Gu, L., Kanade, T.: A Generative Shape Regularization Model for Robust Face Alignment. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 413–426. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  17. 17.
    Liang, L., Xiao, R., Wen, F., Sun, J.: Face Alignment Via Component-Based Discriminative Search. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 72–85. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  18. 18.
    Zhao, C., Cham, W.K., Wang, X.: Joint face alignment with a generic deformable face model. In: CVPR, pp. 561–568 (2011)Google Scholar
  19. 19.
    Tong, Y., Liu, X., Wheeler, F.W., Tu, P.: Automatic facial landmark labeling with minimal supervision. In: CVPR, pp. 2097–2104 (2009)Google Scholar
  20. 20.
    Tipping, M.E., Bishop, C.M.: Mixtures of probabilistic principal component analyzers. Neural Computation 11, 443–482 (1999)CrossRefGoogle Scholar
  21. 21.
    Zhao, X., Chai, X., Niu, Z., Heng, C., Shan, S.: Context constrained facial landmark localization based on discontinuous haar-like feature. In: FG, pp. 673–678 (2011)Google Scholar
  22. 22.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: CVPR, pp. 511–518 (2001)Google Scholar
  23. 23.
    Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. The Annals of Statistics 28, 337–407 (2000)zbMATHMathSciNetCrossRefGoogle Scholar
  24. 24.
    Gelman, A., Carlin, J., Stern, H., Rubin, D.: Bayesian data analysis. Chapman Hall, London (1995)Google Scholar
  25. 25.
    Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Attribute and simile classifiers for face verification. In: ICCV, pp. 365–372 (2009)Google Scholar
  26. 26.
    Sim, T., Baker, S., Bsat, M.: The cmu pose, illumination, and expression (PIE) database. In: FG, pp. 46–51 (2002)Google Scholar
  27. 27.
    Phillips, P.J., Flynn, P.J., Scruggs, T., Bowyer, K.W., Chang, J., Hoffman, K., Marques, J., Min, J., Worek, W.: Overview of the face recognition grand challenge. In: CVPR, pp. 947–954 (2005)Google Scholar
  28. 28.
    Phillips, P.J., Wechsler, H., Huang, J., Rauss, P.J.: The FERET database and evaluation procedure for face-recognition algorithms. IVC 16, 295–306 (1998)CrossRefGoogle Scholar
  29. 29.
    Gao, W., Cao, B., Shan, S., Chen, X., Zhou, D., Zhang, X., Zhao, D.: The CAS-PEAL large-scale chinese face database and baseline evaluations. SMC, Part A 38, 149–161 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Xiaowei Zhao
    • 1
    • 2
  • Xiujuan Chai
    • 1
  • Shiguang Shan
    • 1
  1. 1.Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS)Institute of Computing Technology, CASBeijingChina
  2. 2.Graduate University of Chinese Academy of SciencesBeijingChina

Personalised recommendations