Advertisement

On 3D face reconstruction via cascaded regression in shape space

  • Feng Liu
  • Dan Zeng
  • Jing Li
  • Qi-jun Zhao
Article

Abstract

Cascaded regression has been recently applied to reconstruct 3D faces from single 2D images directly in shape space, and has achieved state-of-the-art performance. We investigate thoroughly such cascaded regression based 3D face reconstruction approaches from four perspectives that are not well been studied: (1) the impact of the number of 2D landmarks; (2) the impact of the number of 3D vertices; (3) the way of using standalone automated landmark detection methods; (4) the convergence property. To answer these questions, a simplified cascaded regression based 3D face reconstruction method is devised. This can be integrated with standalone automated landmark detection methods and reconstruct 3D face shapes that have the same pose and expression as the input face images, rather than normalized pose and expression. An effective training method is also proposed by disturbing the automatically detected landmarks. Comprehensive evaluation experiments have been carried out to compare to other 3D face reconstruction methods. The results not only deepen the understanding of cascaded regression based 3D face reconstruction approaches, but also prove the effectiveness of the proposed method.

Key words

3D face reconstruction Cascaded regressor Shape space Real-time 

CLC number

TP319 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abiantun, R., Prabhu, U., Savvides, M., 2014. Sparse feature extraction for pose-tolerant face recognition. IEEE Trans. Patt. Anal. Mach. Intell., 36(10):2061–2073. https://doi.org/10.1109/TPAMI.2014.2313124CrossRefGoogle Scholar
  2. Aldrian, O., Smith, W.A.P., 2013. Inverse rendering of faces with a 3D morphable model. IEEE Trans. Patt. Anal. Mach. Intell., 35(5):1080–1093. https://doi.org/10.1109/TPAMI.2012.206CrossRefGoogle Scholar
  3. Bagdanov, A.D., Del Bimbo, A., Masi, I., 2011. The florence 2D/3D hybrid face dataset. Workshop on Human Gesture and Behavior Understanding, p.79–80. https://doi.org/10.1145/2072572.2072597Google Scholar
  4. Barron, J.T., Malik, J., 2012. Shape, albedo, and illumination from a single image of an unknown object. IEEE Conf. on Computer Vision and Pattern Recognition, p.334–341. https://doi.org/10.1109/CVPR.2012.6247693Google Scholar
  5. Bas, A., Smith, W.A.P., Bolkart, T., et al., 2016. Fitting a 3D morphable model to edges: a comparison between hard and soft correspondences. IEEE Asian Conf. on Computer Vision, p.377–391. https://doi.org/10.1007/978-3-319-54427-4_28Google Scholar
  6. Blanz, V., Vetter, T., 1999. A morphable model for the synthesis of 3D faces. Proc. SIGGRAPH, p.187–194. https://doi.org/10.1145/311535.311556Google Scholar
  7. Blanz, V., Vetter, T., 2003. Face recognition based on fitting a 3D morphable model. IEEE Trans. Patt. Anal. Mach. Intell., 25(9):1063–1074. https://doi.org/10.1109/TPAMI.2003.1227983CrossRefGoogle Scholar
  8. Booth, J., Roussos, A., Zafeiriou, S., et al., 2016. A 3D morphable model learnt from 10 000 faces. IEEE Conf. on Computer Vision and Pattern Recognition, p.5543–5552. https://doi.org/10.1109/CVPR.2016.598Google Scholar
  9. Cao, C., Hou, Q.M., Zhou, K., 2014a. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans. Graph., 33(4):43.1-43.10. https://doi.org/10.1145/2601097.2601204MATHGoogle Scholar
  10. Cao, C., Weng, Y.L., Zhou, S., et al., 2014b. Facewarehouse: a 3D facial expression database for visual computing. IEEE Trans. Vis. Comput. Graph., 20(3):413–425. https://doi.org/10.1109/TVCG.2013.249CrossRefGoogle Scholar
  11. Cao, C., Wu, H.Z., Weng, Y.L., et al., 2016. Real-time facial animation with image-based dynamic avatars. ACM Trans. Graph., 35(4):126.1-126.12. https://doi.org/10.1145/2897824.2925873CrossRefGoogle Scholar
  12. Chu, B., Romdhani, S., Chen, L.M., 2014. 3D-aided face recognition robust to expression and pose variations. IEEE Conf. on Computer Vision and Pattern Recognition, p.1907–1914. https://doi.org/10.1109/CVPR.2014.245Google Scholar
  13. Han, H., Jain, A.K., 2012. 3D face texture modeling from uncalibrated frontal and profile images. Int. Conf. on Biometrics: Theory, Applications and Systems, p.223–230. https://doi.org/10.1109/BTAS.2012.6374581Google Scholar
  14. Horn, B.K.P., Brooks, M.J., 1989. Shape from Shading. MIT Press, Cambridge, MA, USA.MATHGoogle Scholar
  15. Hu, J.L., Lu, J.W., Tan, Y.P., 2014. Discriminative deep metric learning for face verification in the wild. IEEE Conf. on Computer Vision and Pattern Recognition, p.1875–1882. https://doi.org/10.1109/CVPR.2014.242Google Scholar
  16. Jourabloo, A., Liu, X.M., 2015. Pose-invariant 3D face alignment. IEEE Int. Conf. on Computer Vision, p.3694–3702. https://doi.org/10.1109/ICCV.2015.421Google Scholar
  17. Jourabloo, A., Liu, X.M., 2016. Large-pose face alignment via CNN-based dense 3D model fitting. IEEE Conf. on Computer Vision and Pattern Recognition, p.4188–4196. https://doi.org/10.1109/CVPR.2016.454Google Scholar
  18. Jourabloo, A., Liu, X.M., 2017. Pose-invariant face alignment via CNN-based dense 3D model fitting. Int. J. Comput. Vis., 4:1–17. https://doi.org/10.1007/s11263-017-1012-zMathSciNetGoogle Scholar
  19. Kazemi, V., Sullivan, J., 2014. One millisecond face alignment with an ensemble of regression trees. IEEE Conf. on Computer Vision and Pattern Recognition, p.1867–1874. https://doi.org/10.1109/CVPR.2014.241Google Scholar
  20. Kemelmacher-Shlizerman, I., Basri, R., 2011. 3D face reconstruction from a single image using a single reference face shape. IEEE Trans. Patt. Anal. Mach. Intell., 33(2):394–405. https://doi.org/10.1109/TPAMI.2010.63CrossRefGoogle Scholar
  21. Köstinger, M., Wohlhart, P., Roth, P.M., et al., 2011. Annotated facial landmarks in the wild: a large-scale, realworld database for facial landmark localization. IEEE Int. Conf. on Computer VisionWorkshops, p.2144–2151. https://doi.org/10.1109/ICCVW.2011.6130513Google Scholar
  22. Li, J., Long, S.Q., Zeng, D., et al., 2015. Example-based 3D face reconstruction from uncalibrated frontal and profile images. IEEE Int. Conf. on Biometrics, p.193–200. https://doi.org/10.1109/ICB.2015.7139051Google Scholar
  23. Li, X., Xu, Y.D., Lv, Q., et al., 2016. Affine-transformation parameters regression for face alignment. IEEE Signal Process. Lett., 23(1):55–59. https://doi.org/10.1109/LSP.2015.2499778CrossRefGoogle Scholar
  24. Liu, F., Zeng, D., Zhao, Q.J., et al., 2016. Joint face alignment and 3D face reconstruction. European Conf. on Computer Vision, p.545–560. https://doi.org/10.1007/978-3-319-46454-1_33Google Scholar
  25. Paysan, P., Knothe, R., Amberg, B., et al., 2009. A 3D face model for pose and illumination invariant face recognition. IEEE Conf. on Advanced Video and Signal-based Surveillance, p.296–301. https://doi.org/10.1109/AVSS.2009.58Google Scholar
  26. Ren, J.F., Jiang, X.D., Yuan, J.S., 2016. Face and facial expressions recognition and analysis. Context Aware Human-Robot and Human-Agent Interaction, p.3–29. https://doi.org/10.1007/978-3-319-19947-4_1CrossRefGoogle Scholar
  27. Romdhani, S., Vetter, T., 2005. Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. IEEE Conf. on Computer Vision and Pattern Recognition, p.986–993. https://doi.org/10.1109/CVPR.2005.145Google Scholar
  28. Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., et al., 2013. 300 faces in-the-wild challenge: the first facial landmark localization challenge. IEEE Int. Conf. on Computer Vision Workshops, p.397–403. https://doi.org/10.1109/ICCVW.2013.59Google Scholar
  29. Sun, Y., Wang, X.G., Tang, X.O., 2013. Deep convolutional network cascade for facial point detection. IEEE Conf. on Computer Vision and Pattern Recognition, p.3476–3483. https://doi.org/10.1109/CVPR.2013.446Google Scholar
  30. Suwajanakorn, S., Kemelmacher-Shlizerman, I., Seitz, S.M., 2014. Total moving face reconstruction. European Conf. on Computer Vision, p.796–812. https://doi.org/10.1007/978-3-319-10593-2_52Google Scholar
  31. Xiong, X.H., de la Torre, F., 2013. Supervised descent method and its applications to face alignment. IEEE Conf. on Computer Vision and Pattern Recognition, p.532–539. https://doi.org/10.1109/CVPR.2013.75Google Scholar
  32. Yin, L.J., Wei, X.Z., Sun, Y., et al., 2006. A 3D facial expression database for facial behavior research. IEEE Int. Conf. on Automatic Face and Gesture Recognition, p.211–216. https://doi.org/10.1109/FGR.2006.6Google Scholar
  33. Zeng, D., Zhao, Q.J., Long, S.Q., et al., 2017. Examplar coherent 3D face reconstruction from forensic mugshot database. Image Vis. Comput., 58:193–203. https://doi.org/10.1016/j.imavis.2016.03.001CrossRefGoogle Scholar
  34. Zhang, Z.P., Luo, P., Chen, C.L., et al., 2014. Facial landmark detection by deep multi-task learning. European Conf. on Computer Vision, p.94–108. https://doi.org/10.1007/978-3-319-10599-4_7Google Scholar
  35. Zhou, X.W., Leonardos, S., Hu, X.Y., et al., 2015. 3D shape estimation from 2D landmarks: a convex relaxation approach. IEEE Conf. on Computer Vision and Pattern Recognition, p.4447–4455. https://doi.org/10.1109/CVPR.2015.7299074Google Scholar
  36. Zhu, S.Z., Li, C., Chen, C.L., et al., 2015. Face alignment by coarse-to-fine shape searching. IEEE Conf. on Computer Vision and Pattern Recognition, p.4998–5006. https://doi.org/10.1109/CVPR.2015.7299134Google Scholar
  37. Zhu, X.X., Ramanan, D., 2012. Face detection, pose estimation, and landmark localization in the wild. IEEE Conf. on Computer Vision and Pattern Recognition, p.2879–2886. https://doi.org/10.1109/CVPR.2012.6248014Google Scholar
  38. Zhu, X.Y., Yi, D., Lei, Z., et al., 2014. Robust 3D morphable model fitting by sparse SIFT flow. IEEE Int. Conf. on Pattern Recognition, p.4044–4049. https://doi.org/10.1109/ICPR.2014.693Google Scholar
  39. Zhu, X.Y., Lei, Z., Yan, J.J., et al., 2015. High-fidelity pose and expression normalization for face recognition in the wild. IEEE Conf. on Computer Vision and Pattern Recognition, p.787–796. https://doi.org/10.1109/CVPR.2015.7298679Google Scholar

Copyright information

© Zhejiang University and Springer-Verlag GmbH Germany, part of Springer Nature 2017

Authors and Affiliations

  1. 1.College of Computer ScienceSichuan UniversityChengduChina

Personalised recommendations