Nuclear Norm Regularized Structural Orthogonal Procrustes Regression for Face Hallucination with Pose

  • Dong Zhu
  • Guangwei GaoEmail author
  • Hao Gao
  • Huimin Lu
Part of the Studies in Computational Intelligence book series (SCI, volume 810)


In real applications, the observed low-resolution (LR) face images usually have pose variations. Conventional learning based methods ignore these variations, thus the learned representations are not beneficial for the following reconstruction. In this paper, we propose a nuclear norm regularized structural orthogonal Procrustes regression (N2SOPR) method to learn pose-robust feature representations for efficient face hallucination. The orthogonal Procrustes regression (OPR) seeks an optimal transformation between two images to correct the pose from one to the other. Additionally, our N2SOPR uses the nuclear norm constraint on the error term to keep image’s structural information. A low-rank constraint on the representation coefficients is imposed to adaptively select the training samples that belong to the same subspace as the inputs. Moreover, a locality constraint is also enforced to preserve the locality and the sparsity simultaneously. Experimental results on standard face hallucination databases indicate that our proposed method can produce more reasonable near frontal face images for recognition purpose.


Face hallucination Regression analysis Pose variations Low-rank constraint 



This work was partially supported by the National Natural Science Foundation of China under Grant nos. 61502245, 61503195, 61772568, the Natural Science Foundation of Jiangsu Province under Grant no. BK20150849, Research Fund of SKL of Ocean Engineering in Shanghai Jiaotong University (1315;1510), Research Fund of SKL of Marine Geology in Tongji University (MGK1608), Open Fund Project of Fujian Provincial Key Laboratory of Information Processing and Intelligent Control (Minjiang University) (No. MJUKF201717). Guangwei Gao is the corresponding author.


  1. 1.
    Park, S.C., Min, K.P., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag. 20(3), 21–36 (2003)CrossRefGoogle Scholar
  2. 2.
    Lu, H., Li, Y., Chen, M., Kim, H., Serikawa, S.: Brain intelligence: go beyond artificial intelligence. Mob. Netw. Appl. 23, 368–375 (2018)CrossRefGoogle Scholar
  3. 3.
    Li, Y., Lu, H., Li, K., Kim, H., Serikawa, S.: Non-uniform de-scattering and de-blurring of underwater images. Mob. Netw. Appl. 23, 352–362 (2018)CrossRefGoogle Scholar
  4. 4.
    Lu, H., Li, Y., Uemura, T., Kim, H., Serikawa, S.: Low illumination underwater light field images reconstruction using deep convolutional neural networks. Future Gener. Comput. Syst. 82, 142–148 (2018)CrossRefGoogle Scholar
  5. 5.
    Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. Proceedings of the Computer Vision & Pattern Recognition, vol. 1, pp. 275–282, Jan 2004Google Scholar
  6. 6.
    Ma, X., Zhang, J., Qi, C.: Hallucinating face by position-patch. Pattern Recogn. 43(6), 2224–2236 (2010)CrossRefGoogle Scholar
  7. 7.
    Wright, J., Yang, A.Y., Ganesh, A., et al.: Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31(2), 210–227 (2009)Google Scholar
  8. 8.
    Jiang, J., Hu, R., Wang, Z., Han, Z.: Noise robust face hallucination via locality constrained representation. IEEE Trans. Multimed. 16(5), 1268–1281 (2014)CrossRefGoogle Scholar
  9. 9.
    Pang, H., Gao, G., Jing, X., et al.: Kernel locality-constrained adaptive iterative neighbor embedding for face hallucination. In: IEEE International Conference on Wireless Communications & Signal Processing, pp. 1–5 (2016)Google Scholar
  10. 10.
    Schölkopf, B., Smola, A., Müller, K.R.: Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput. 10(5), 1299–1319 (1998)CrossRefGoogle Scholar
  11. 11.
    Gao, G., Pang, H., Wang, C., et al.: Locality-constrained iterative matrix regression for robust face hallucination. In: International Conference on Neural Information Processing, pp. 613–621 (2017)Google Scholar
  12. 12.
    Jung, C., Jiao, L., Liu, B., et al.: Position-Patch Based Face Hallucination Using Convex Optimization. IEEE Signal Process. Lett. 18(6), 367–370 (2011)CrossRefGoogle Scholar
  13. 13.
    Rohit, U., Abdu, R.V., George, S.N.: A robust face hallucination technique based on adaptive learning method. Multimed. Tools Appl. 76, 1–21 (2017)CrossRefGoogle Scholar
  14. 14.
    Lu, T., Pan, L., Wang, H., Zhang, Y., Wang, B., et al.: Face hallucination using deep collaborative representation for local and non-local patches. In: IEEE International Symposium on Circuits & Systems, pp. 1–4 (2017)Google Scholar
  15. 15.
    Shi, J., Qi, C.: Face hallucination based on PCA dictionary pairs. In: IEEE International Conference on Image Processing, pp. 933–937 (2014)Google Scholar
  16. 16.
    Li, Y., Cai, C., Qiu, G., Lam, K.M.: Face hallucination based on sparse local-pixel structure. Pattern Recogn. 47(3), 1261–1270 (2014)CrossRefGoogle Scholar
  17. 17.
    Wang, Z., Jiang, J., Xiong, Z., Hu, R., Shao, Z.: Face hallucination via weighted sparse representation. IEEE Int. Conf. Acoust. 32(3), 2198–2201 (2013)Google Scholar
  18. 18.
    Jiang, J., Hu, R., Wang, Z., Xiong, Z.: Support-driven sparse coding for face hallucination. In: IEEE International Symposium on Circuits & Systems, pp. 2980–2983 (2013)Google Scholar
  19. 19.
    Lin, Z., Chen, M., Ma, Y.: The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. Eprint Arxiv (2010)Google Scholar
  20. 20.
    Yin, W.: The alternating direction method of multipliers. In: Imaging Science a Workshop in Honor of Stanley Osher (2012)Google Scholar
  21. 21.
    Ghadimi, E., Teixeira, A., Shames, I., Johansson, M.: Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems. IEEE Trans. Autom. Control 60(3), 644–658 (2013)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Cai, J.F., Cand, E.J.S., Shen, Z.: A singular value thresholding algorithm for matrix completion. Siam J. Optim. 20(4), 1956–1982 (2008)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Phillips, P.J., Wechsler, H., Huang, J., et al.: The FERET database and evaluation procedure for face-recognition algorithms. Image Vis. Comput. 16(5), 295–306 (1998)CrossRefGoogle Scholar
  24. 24.
    Dong, C., Loy, C.C., He, K., et al.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  25. 25.
    Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: International Conference on Pattern Recognition, pp. 2366–2369 (2010)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of AutomationNanjing University of Posts and TelecommunicationNanjingChina
  2. 2.Institute of Advanced TechnologyNanjing University of Posts and TelecommunicationNanjingChina
  3. 3.Fujian Provincial Key Laboratory of Information Processing and Intelligent ControlMinjiang UniversityFuzhouChina
  4. 4.Department of Mechanical and Control EngineeringKyushu Institute of TechnologyKitakyushuJapan

Personalised recommendations