Advertisement

Deep Learning in Face Recognition Across Variations in Pose and Illumination

  • Xiaoyue JiangEmail author
  • Yaping Hou
  • Dong Zhang
  • Xiaoyi Feng
Chapter
  • 1k Downloads

Abstract

Even though face recognition in frontal view and normal lighting conditions works very well, the performance drops sharply in extreme conditions. Recently there is plenty of work dealing with pose and illumination problems, respectively. However both the lighting and pose variations always happen simultaneously in general conditions, and consequently we propose an end-to-end face recognition algorithm to deal with two variations at the same time based on convolutional neural networks. In order to achieve better performance, we extract discriminative nonlinear features that are invariant to pose and illumination. We propose to use the 1 × 1 convolutional kernels to extract the local features. Furthermore a parallel multi-stream convolutional neural network is developed to extract multi-hierarchy features which are more efficient than single-scale features. In the experiments we obtain the average face recognition rate of 96.9% on MultiPIE dataset. Even for profile position, the average recognition rate is also around 98.5% in different lighting conditions, which improves the state-of-the-art face recognition across poses and illumination by 7.5%.

Notes

Acknowledgements

This chapter is partly supported by the National Natural Science Foundation of China (No.61502388), Ph.D. Programs Foundation of Ministry of Education of China (No. 20136102120041), the Fundamental Research Funds for the Central Universities (No. 3102015BJ (II)ZS016), and the Shaanxi Province International Science and Technology Cooperation and Exchange Program (2017KW002).

References

  1. 1.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(12), 2037–2041 (2006). DOI 10.1109/TPAMI.2006.244Google Scholar
  2. 2.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. IEEE transactions on pattern analysis and machine intelligence 28(12), 2037–2041 (2006)CrossRefGoogle Scholar
  3. 3.
    Andrew, G., Arora, R., Bilmes, J., Livescu, K.: Deep canonical correlation analysis. In: International Conference on Machine Learning, pp. 1247–1255 (2013)Google Scholar
  4. 4.
    Ashraf, A.B., Lucey, S., Chen, T.: Learning patch correspondences for improved viewpoint invariant face recognition. In: Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1–8. IEEE (2008)Google Scholar
  5. 5.
    Basri, R., Jacobs, D.W.: Lambertian reflectance and linear subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(2), 218–233 (2003)CrossRefGoogle Scholar
  6. 6.
    Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (surf). Computer vision and image understanding 110(3), 346–359 (2008)CrossRefGoogle Scholar
  7. 7.
    Belhumeur, P.N., Kriegman, D.J.: What is the set of images of an object under all possible illumination conditions? International Journal of Computer Vision 28(3), 245–260 (1998)CrossRefGoogle Scholar
  8. 8.
    Biswas, S., Aggarwal, G., Flynn, P.J., Bowyer, K.W.: Pose-robust recognition of low-resolution face images. IEEE transactions on pattern analysis and machine intelligence 35(12), 3037–3049 (2013)CrossRefGoogle Scholar
  9. 9.
    Castillo, C.D., Jacobs, D.W.: Using stereo matching with general epipolar geometry for 2d face recognition across pose. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(12), 2298–2304 (2009)CrossRefGoogle Scholar
  10. 10.
    Chen, D., Cao, X., Wen, F., Sun, J.: Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3025–3032 (2013)Google Scholar
  11. 11.
    Ding, C., Choi, J., Tao, D., Davis, L.S.: Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(3), 518–531 (2016). DOI 10.1109/TPAMI.2015.2462338Google Scholar
  12. 12.
    Ding, C., Tao, D.: Pose-invariant face recognition with homography-based normalization. Pattern Recognition 66, 144–152 (2017)CrossRefGoogle Scholar
  13. 13.
    Ding, C., Xu, C., Tao, D.: Multi-task pose-invariant face recognition. IEEE Transactions on Image Processing 24(3), 980–993 (2015)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Ding, C., Xu, C., Tao, D.: Multi-task pose-invariant face recognition. IEEE Transactions on Image Processing 24(3), 980–93 (2015)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Gao, Y., Leung, M.K.: Face recognition using line edge map. IEEE transactions on pattern analysis and machine intelligence 24(6), 764–779 (2002)CrossRefGoogle Scholar
  16. 16.
    Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-pie. Image and Vision Computing 28(5), 807–813 (2010)CrossRefGoogle Scholar
  17. 17.
    Gunther, M., Costa-Pazo, A., Ding, C., Boutellaa, E.: The 2013 face recognition evaluation in mobile environment. 2013 International Conference on Biometrics (ICB) pp. 1–7 (2013)Google Scholar
  18. 18.
    He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.J.: Face recognition using laplacian faces. IEEE transactions on pattern analysis and machine intelligence 27(3), 328–340 (2005)CrossRefGoogle Scholar
  19. 19.
    Ho, H.T., Chellappa, R.: Pose-invariant face recognition using markov random fields. IEEE transactions on image processing 22(4), 1573–1584 (2013)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Hu, J., Lu, J., Tan, Y.P.: Discriminative deep metric learning for face verification in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1875–1882 (2014)Google Scholar
  21. 21.
    Huang, S.C., Cheng, F.C., Chiu, Y.S.: Efficient contrast enhancement using adaptive gamma correction with weighting distribution. IEEE Transactions on Image Processing 22(3), 1032–1041 (2013)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Jiang, X., Cheng, Y., Xiao, R., Li, Y., Zhao, R.: Spherical harmonic based linear face de-lighting and compensation. Applied Mathematics and Computation 185(2), 857–868 (2007). https://doi.org/10.1016/j.amc.2006.06.090. http://www.sciencedirect.com/science/article/pii/S0096300306007673. Special Issue on Intelligent Computing Theory and Methodology
  23. 23.
    Jiang, X., Feng, X., Wu, J., Peng, J.: Lighting alignment for image sequences. In: International Conference on Image and Graphics, pp. 462–474. Springer (2015)Google Scholar
  24. 24.
    Jiang, X., Kong, Y.O., Huang, J., Zhao, R., Zhang, Y.: Learning from real images to model lighting variations for face images. In: European Conference on Computer Vision (ECCV), pp. 284–297 (2008)Google Scholar
  25. 25.
    Kafai, M., An, L., Bhanu, B.: Reference face graph for face recognition. IEEE Transactions on Information Forensics and Security 9(12), 2132–2143 (2014)CrossRefGoogle Scholar
  26. 26.
    Kan, M., Shan, S., Chang, H., Chen, X.: Stacked progressive auto-encoders (spae) for face recognition across poses. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1883–1890 (2014)Google Scholar
  27. 27.
    Kan, M., Shan, S., Xilin., C.: Multi-view deep network for cross-view classification. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  28. 28.
    Kan, M., Shan, S., Zhang, H., Lao, S., Chen, X.: Multi-view discriminant analysis. IEEE transactions on pattern analysis and machine intelligence 38(1), 188–194 (2016)CrossRefGoogle Scholar
  29. 29.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105 (2012)Google Scholar
  30. 30.
    Lee, K.C., Ho, J., Kriegman, D.: Acquiring linear subspaces for face recognition under variable lighting. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(5), 684–698 (2005)CrossRefGoogle Scholar
  31. 31.
    Li, A., Shan, S., Chen, X., Gao, W.: Maximizing intra-individual correlations for face recognition across pose differences. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 605–611. IEEE (2009)Google Scholar
  32. 32.
    Li, A., Shan, S., Gao, W.: Coupled bias-variance tradeoff for cross-pose face recognition. IEEE Transactions on Image Processing 21(1), 305–15 (2012)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Li, H., Hua, G., Lin, Z., Brandt, J., Yang, J.: Probabilistic elastic matching for pose variant face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3499–3506 (2013)Google Scholar
  34. 34.
    Li, S., Liu, X., Chai, X., Zhang, H., Lao, S., Shan, S.: Morphable displacement field based image matching for face recognition across pose. European Conference on Computer Vision 2012 pp. 102–115 (2012)Google Scholar
  35. 35.
    Liao, Q., Leibo, J.Z., Poggio, T.: Learning invariant representations and applications to face verification. In: Advances in Neural Information Processing Systems, pp. 3057–3065 (2013)Google Scholar
  36. 36.
    Liao, S., Jain, A.K., Li, S.Z.: Partial face recognition: Alignment-free approach. IEEE Transactions on pattern analysis and machine intelligence 35(5), 1193–1205 (2013)CrossRefGoogle Scholar
  37. 37.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  38. 38.
    Majumdar, A., Singh, R., Vatsa, M.: Face recognition via class sparsity based supervised encoding. IEEE transactions on pattern analysis and machine intelligence (2016)Google Scholar
  39. 39.
    Masi, I., Rawls, S., Medioni, G., Natarajan, P.: Pose-aware face recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4838–4846 (2016)Google Scholar
  40. 40.
    Parkhi, O.M., Vedaldi, A., Zisserman, A., et al.: Deep face recognition. In: British Machine Vision Conference, vol. 1, p. 6 (2015)Google Scholar
  41. 41.
    Peng, X., Yu, X., Sohn, K., Metaxas, D., Chandraker, M.: Reconstruction for feature disentanglement in pose-invariant face recognition. arXiv preprint arXiv:1702.03041 (2017)Google Scholar
  42. 42.
    Pentland, A., Moghaddam, B., Starner, T., et al.: View-based and modular eigenspaces for face recognition. In: CVPR, vol. 94, pp. 84–91 (1994)Google Scholar
  43. 43.
    Pizer, S.M., Amburn, E.P., Austin, J.D., Cromartie, R., Geselowitz, A., Greer, T., ter Haar Romeny, B., Zimmerman, J.B., Zuiderveld, K.: Adaptive histogram equalization and its variations. Computer vision, graphics, and image processing 39(3), 355–368 (1987)CrossRefGoogle Scholar
  44. 44.
    Prince, S.J., Elder, J.H., Warrell, J., Felisberti, F.M.: Tied factor analysis for face recognition across large pose differences. IEEE Transactions on pattern analysis and machine intelligence 30(6), 970–984 (2008)CrossRefGoogle Scholar
  45. 45.
    Ramamoorthi, R., Hanrahan, P.: On the relationship between radiance and irradiance: determining the illumination from images of a convex lambertian object. Journal of the Optical Society of America, A 18(10), 2448–2459 (2001)MathSciNetCrossRefGoogle Scholar
  46. 46.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems (NIPS) pp. 91–99 (2015)Google Scholar
  47. 47.
    Rupnik, J., Shawe-Taylor, J.: Multi-view canonical correlation analysis. In: Conference on Data Mining and Data Warehouses (SiKDD 2010), pp. 1–4 (2010)Google Scholar
  48. 48.
    Rupnik, J., Shawe-Taylor, J.: Multi-view canonical correlation analysis. Conference on Data Mining and Data Warehouses(SiKDD 2010) pp. 1–4 (2010)Google Scholar
  49. 49.
    Schölkopf, B., Smola, A., Müller, K.R.: Nonlinear component analysis as a kernel eigenvalue problem. Neural computation 10(5), 1299–1319 (1998)CrossRefGoogle Scholar
  50. 50.
    Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: A unified embedding for face recognition and clustering. Computer Vision and Pattern Recognition pp. 815–823 (2015)Google Scholar
  51. 51.
    Schroff, F., Treibitz, T., Kriegman, D., Belongie, S.: Pose, illumination and expression invariant pairwise face-similarity measure via doppelgänger list comparison. In: Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 2494–2501. IEEE (2011)Google Scholar
  52. 52.
    Sharma, A.: Generalized multiview analysis: A discriminative latent space. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2160–2167 (2012)Google Scholar
  53. 53.
    Sharma, A., Al Haj, M., Choi, J., Davis, L.S., Jacobs, D.W.: Robust pose invariant face recognition using coupled latent space discriminant analysis. Computer Vision and Image Understanding 116(11), 1095–1110 (2012)CrossRefGoogle Scholar
  54. 54.
    Sharma, A., Jacobs, D.W.: Bypassing synthesis: Pls for face recognition with pose, low-resolution and sketch. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 593–600 (2011)Google Scholar
  55. 55.
    Sharma, A., Kumar, A., Daume, H., Jacobs, D.W.: Generalized multiview analysis: A discriminative latent space. In: Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pp. 2160–2167. IEEE (2012)Google Scholar
  56. 56.
    Shashua, A., Riklin-Raviv, T.: The quotient image: Class-based re-rendering and recognition with varying illuminations. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2), 129–139 (2001)CrossRefGoogle Scholar
  57. 57.
    Sun, Y., Wang, X., Tang, X.: Deeply learned face representations are sparse, selective, and robust. Computer Vision and Pattern RecognitionGoogle Scholar
  58. 58.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  59. 59.
    Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: Closing the gap to human-level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708 (2014)Google Scholar
  60. 60.
    Tan, X., Triggs, B.: Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE transactions on image processing 19(6), 1635–1650 (2010)MathSciNetCrossRefGoogle Scholar
  61. 61.
    Tran, L., Yin, X., Liu, X.: Disentangled representation learning gan for pose-invariant face recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 3, p. 7 (2017)Google Scholar
  62. 62.
    Turk, M., Pentland, A.: Eigenfaces for recognition. Journal of cognitive neuroscience 3(1), 71–86 (1991)CrossRefGoogle Scholar
  63. 63.
    Wagner, A., Wright, J., Ganesh, A., Zhou, Z., Mobahi, H., Ma, Y.: Toward a practical face recognition system: robust alignment and illumination by sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(2), 372–386 (2012)CrossRefGoogle Scholar
  64. 64.
    Wang, H., Li, S.Z., Wang, Y.: Generalized quotient image. In: Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, vol. 2, pp. II–II. IEEE (2004)Google Scholar
  65. 65.
    Wang, X., Han, T.X., Yan, S.: An hog-lbp human detector with partial occlusion handling. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 32–39 (2009). DOI 10.1109/ICCV.2009.5459207Google Scholar
  66. 66.
    Wen, Y., Zhang, K., Li, Z., Qiao, Y.: A discriminative feature learning approach for deep face recognition. In: B. Leibe, J. Matas, N. Sebe, M. Welling (eds.) Computer Vision – ECCV 2016, pp. 499–515. Springer International Publishing, Cham (2016)CrossRefGoogle Scholar
  67. 67.
    Wiskott, L., Krüger, N., Kuiger, N., Von Der Malsburg, C.: Face recognition by elastic bunch graph matching. IEEE Transactions on pattern analysis and machine intelligence 19(7), 775–779 (1997)CrossRefGoogle Scholar
  68. 68.
    Xie, X., Zheng, W.S., Lai, J., Yuen, P.C., Suen, C.Y.: Normalization of face illumination based on large-and small-scale features. IEEE Transactions on Image Processing 20(7), 1807–1821 (2011)MathSciNetCrossRefGoogle Scholar
  69. 69.
    Yang, J., Frangi, A.F., Yang, J.y., Zhang, D., Jin, Z.: Kpca plus lda: a complete kernel fisher discriminant framework for feature extraction and recognition. IEEE Transactions on pattern analysis and machine intelligence 27(2), 230–244 (2005)Google Scholar
  70. 70.
    Zhang, Y., Shao, M., Wong, E.K., Fu, Y.: Random faces guided sparse many-to-one encoder for pose-invariant face recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2416–2423 (2013)Google Scholar
  71. 71.
    Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face recognition: A literature survey. ACM Computing Surveys 35(4), 399–458 (2003)CrossRefGoogle Scholar
  72. 72.
    Zhou, H., Sadka, A.H.: Combining perceptual features with diffusion distance for face recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 41(5), 577–588 (2011)Google Scholar
  73. 73.
    Zhu, Z., Luo, P., Wang, X., Tang, X.: Deep learning identity-preserving face space. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 113–120 (2013)Google Scholar
  74. 74.
    Zhu, Z., Luo, P., Wang, X., Tang, X.: Multi-view perceptron: a deep model for learning face identity and view representations. Advances in Neural Information Processing Systems (NIPS) pp. 217–225 (2014)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  • Xiaoyue Jiang
    • 1
    Email author
  • Yaping Hou
    • 1
  • Dong Zhang
    • 1
  • Xiaoyi Feng
    • 1
  1. 1.School of Electronics and InformationNorthwestern Polytechnical UniversityXi’anChina

Personalised recommendations