Deep Cascaded Bi-Network for Face Hallucination

  • Shizhan Zhu
  • Sifei Liu
  • Chen Change LoyEmail author
  • Xiaoou Tang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9909)


We present a novel framework for hallucinating faces of unconstrained poses and with very low resolution (face size as small as 5pxIOD). In contrast to existing studies that mostly ignore or assume pre-aligned face spatial configuration (e.g. facial landmarks localization or dense correspondence field), we alternatingly optimize two complementary tasks, namely face hallucination and dense correspondence field estimation, in a unified framework. In addition, we propose a new gated deep bi-network that contains two functionality-specialized branches to recover different levels of texture details. Extensive experiments demonstrate that such formulation allows exceptional hallucination quality on in-the-wild low-res faces with significant pose and illumination variations.


Facial Image Dense Field Convolutional Neural Network Super Resolution Facial Landmark 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work is partially supported by SenseTime Group Limited and the Hong Kong Innovation and Technology Support Programme.


  1. 1.
    Yang, S., Luo, P., Loy, C.C., Tang, X.: Wider face: A face detection benchmark. arXiv preprint arXiv:1511.06523 (2015)
  2. 2.
    Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. IJCV 107(2), 177–190 (2014)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: CVPR, pp. 532–539 (2013)Google Scholar
  4. 4.
    Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: CVPR (2015)Google Scholar
  5. 5.
    Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: DeepFace: Closing the gap to human-level performance in face verification. In: CVPR (2014)Google Scholar
  6. 6.
    Jin, Y., Bouganis, C.S.: Robust multi-image based blind face hallucination. In: CVPR (2015)Google Scholar
  7. 7.
    Tappen, M.F., Liu, C.: A bayesian approach to alignment-based image hallucination. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VII. LNCS, vol. 7578, pp. 236–249. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  8. 8.
    Yang, C.Y., Liu, S., Yang, M.H.: Structured face hallucination. In: CVPR (2013)Google Scholar
  9. 9.
    Kolouri, S., Rohde, G.K.: Transport-based single frame super resolution of very low resolution face images. In: CVPR (2015)Google Scholar
  10. 10.
    Wang, X., Tang, X.: Hallucinating face by eigentransformation. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 35, 425–434 (2005)CrossRefGoogle Scholar
  11. 11.
    Chakrabarti, A., Rajagopalan, A., Chellappa, R.: Super-resolution of face images using kernel PCA-based prior. IEEE Trans. Multimedia 9(4), 888–892 (2007)CrossRefGoogle Scholar
  12. 12.
    Liu, C., Shum, H.Y., Freeman, W.T.: Face hallucination: theory and practice. IJCV 75, 115–134 (2007)CrossRefGoogle Scholar
  13. 13.
    Baker, S., Kanade, T.: Hallucinating faces. In: AFGR (2000)Google Scholar
  14. 14.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. In: PAMI (2015)Google Scholar
  15. 15.
    Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In: ICCV (2015)Google Scholar
  16. 16.
    Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: CVPR (2015)Google Scholar
  17. 17.
    Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., Zhang, L.: Convolutional sparse coding for image super-resolution. In: ICCV (2015)Google Scholar
  18. 18.
    Bruna, J., Sprechmann, P., LeCun, Y.: Super-resolution with deep convolutional sufficient statistics. In: ICLR (2016)Google Scholar
  19. 19.
    Salvador, J., Perez-Pellitero, E.: Naive bayes super-resolution forest. In: ICCV (2015)Google Scholar
  20. 20.
    Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, PART II. LNCS, vol. 9906, pp. 391–407. Springer, Heidelberg (2016)Google Scholar
  21. 21.
    Hui, T.W., Loy, C.C., Tang, X.: Depth map super resolution by deep multi-scale guidance. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, PART III. LNCS, vol. 9907, pp. 353–369. Springer, Heidelberg (2016)Google Scholar
  22. 22.
    Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 fps via regressing local binary features. In: CVPR (2014)Google Scholar
  23. 23.
    Tzimiropoulos, G.: Project-out cascaded regression with an application to face alignment. In: CVPR (2015)Google Scholar
  24. 24.
    Smith, B.M., Zhang, L., Brandt, J., Lin, Z., Yang, J.: Exemplar-based face parsing. In: CVPR, pp. 3484–3491 (2013)Google Scholar
  25. 25.
    Cui, Z., Chang, H., Shan, S., Zhong, B., Chen, X.: Deep network cascade for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part V. LNCS, vol. 8693, pp. 49–64. Springer, Heidelberg (2014)Google Scholar
  26. 26.
    Zhou, E., Fan, H., Cao, Z., Jiang, Y., Yin, Q.: Learning face hallucination in the wild. In: AAAI (2015)Google Scholar
  27. 27.
    Dollár, P., Welinder, P., Perona, P.: Cascaded pose regression. In: CVPR, pp. 1078–1085 (2010)Google Scholar
  28. 28.
    Zhang, J., Shan, S., Kan, M., Chen, X.: Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part II. LNCS, vol. 8690, pp. 1–16. Springer, Heidelberg (2014)Google Scholar
  29. 29.
    Zhu, S., Li, C., Loy, C.C., Tang, X.: Face alignment by coarse-to-fine shape searching. In: CVPR (2015)Google Scholar
  30. 30.
    Zhu, S., Li, C., Loy, C.C., Tang, X.: Unconstrained face alignment via cascaded compositional learning. In: CVPR (2016)Google Scholar
  31. 31.
    Wang, X., Valstar, M., Martinez, B., Haris Khan, M., Pridmore, T.: Tric-track: tracking by regression with incrementally learned cascades. In: ICCV (2015)Google Scholar
  32. 32.
    Tenenbaum, J.B., Freeman, W.T.: Separating style and content with bilinear models. Neural Comput. 12, 1247–1283 (2000)CrossRefGoogle Scholar
  33. 33.
    Pirsiavash, H., Ramanan, D., Fowlkes, C.C.: Bilinear classifiers for visual recognition. In: NIPS (2009)Google Scholar
  34. 34.
    Xiong, Y., Zhu, K., Lin, D., Tang, X.: Recognize complex events from static images by fusing deep channels. In: CVPR (2015)Google Scholar
  35. 35.
    Lin, T.Y., RoyChowdhury, A., Maji, S.: Bilinear CNN models for fine-grained visual recognition. In: ICCV (2015)Google Scholar
  36. 36.
    Gao, Y., Beijbom, O., Zhang, N., Darrell, T.: Compact bilinear pooling. In: CVPR (2016)Google Scholar
  37. 37.
    Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NIPS (2014)Google Scholar
  38. 38.
    Alabort-i Medina, J., Zafeiriou, S.: Unifying holistic and parts-based deformable model fitting. In: CVPR (2015)Google Scholar
  39. 39.
    Snape, P., Roussos, A., Panagakis, Y., Zafeiriou, S.: Face flow. In: ICCV (2015)Google Scholar
  40. 40.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. TPAMI 23(6), 681–685 (2001)CrossRefGoogle Scholar
  41. 41.
    Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-pie. Image Vis. Comput. 28(5), 807–813 (2010)CrossRefGoogle Scholar
  42. 42.
    Jesorsky, O., Kirchberg, K.J., Frischholz, R.W.: Robust face detection using the Hausdorff distance. In: Bigun, J., Smeraldi, F. (eds.) AVBPA 2001. LNCS, vol. 2091, pp. 90–95. Springer, Heidelberg (2001). doi: 10.1007/3-540-45344-X_14 CrossRefGoogle Scholar
  43. 43.
    Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Attribute and simile classifiers for face verification. In: CVPR (2009)Google Scholar
  44. 44.
    Pinto, N., Stone, Z., Zickler, T., Cox, D.: Scaling up biologically-inspired computer vision: a case study in unconstrained face recognition on facebook. In: CVPRW (2011)Google Scholar
  45. 45.
    Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial feature localization. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part III. LNCS, vol. 7574, pp. 679–692. Springer, Heidelberg (2012)Google Scholar
  46. 46.
    Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: ICCVW (2013)Google Scholar
  47. 47.
    Zhang, X., Yin, L., Cohn, J.F., Canavan, S., Reale, M., Horowitz, A., Liu, P., Girard, J.M.: BP4D-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database. Image Vis. Comput. 32(10), 692–706 (2014)CrossRefGoogle Scholar
  48. 48.
    Zhang, X., Yin, L., Cohn, J.F., Canavan, S., Reale, M., Horowitz, A., Liu, P.: A high-resolution spontaneous 3d dynamic facial expression database. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (2013)Google Scholar
  49. 49.
    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: ICCV (2015)Google Scholar
  50. 50.
    Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 111–126. Springer, Heidelberg (2015)Google Scholar
  51. 51.
    Capel, D., Zisserman, A.: Super-resolution from multiple views using learnt image models. In: CVPR (2001)Google Scholar
  52. 52.
    Ma, X., Zhang, J., Qi, C.: Hallucinating face by position-patch. Pattern Recogn. 43(6), 2224–2236 (2010)CrossRefGoogle Scholar
  53. 53.
    Efrat, N., Glasner, D., Apartsin, A., Nadler, B., Levin, A.: Accurate blur models vs. image priors in single image super-resolution. In: ICCV (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Shizhan Zhu
    • 1
  • Sifei Liu
    • 1
    • 2
  • Chen Change Loy
    • 1
    • 3
    Email author
  • Xiaoou Tang
    • 1
    • 3
  1. 1.Department of Information EngineeringThe Chinese University of Hong KongHong KongChina
  2. 2.University of California, MercedMercedUSA
  3. 3.Shenzhen Institutes of Advanced TechnologyChinese Academy of SciencesShenzhenChina

Personalised recommendations