Advertisement

Neural Computing and Applications

, Volume 31, Issue 12, pp 9175–9184 | Cite as

Transfer deep feature learning for face sketch recognition

  • Weiguo Wan
  • Yongbin Gao
  • Hyo Jong LeeEmail author
Original Article
  • 173 Downloads

Abstract

Sketch-to-photo recognition is an important challenge in face recognition because it requires matching face images in different domains. Even the deep learning, which has recently been deployed in face recognition, is not efficient for face sketch recognition due to the limited sketch datasets. In this paper, we propose a novel face sketch recognition approach based on transfer learning. We design a three-channel convolutional neural network architecture in which the triplet loss is adopted in order to learn discriminative features and reduce intra-class variations. Moreover, we propose a hard triplet sample selection strategy to augment the number of training samples and avoid slow convergence. With the proposed method, facial features from digital photos and from sketches taken from the same person are closer; the opposite occurs if the digital photo and sketch are from different identities. Experimental results on multiple public datasets indicate that the proposed face sketch recognition method outperforms the existing approaches.

Keywords

Face sketch recognition Transfer learning VGG-Face network Hard triplet sample selection 

Notes

Acknowledgements

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (GR 2016R1D1A3B03931911). This study was also financially supported by the grants of China Scholarship Council (CSC No. 201708260057). This work was also supported in part by the National Natural Science Foundation of China under Grants 61802253, in part by the Collaborative Innovation Center for Economic Crime Investigation and Prevention Technology of Jiangxi Province under Grant JXJZXTCX-027, in part by the Shanghai Chenguang Talented Program under Grant 17CG59.

Compliance with ethical standards

Conflict of interest

The authors declare no conflict of interest. We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work entitled “Transfer Deep Feature Learning for Face Sketch Recognition.”

References

  1. 1.
    Peng C, Wang N, Gao X, Li J (2018) Face recognition from multiple stylistic sketches: scenarios, datasets, and evaluation. Pattern Recognit 84:262–272CrossRefGoogle Scholar
  2. 2.
    Wang N, Gao X, Sun J, Li J (2017) Anchored neighborhood index for face sketch synthesis. IEEE Trans Circuits Syst Video Technol 28:2154–2163CrossRefGoogle Scholar
  3. 3.
    Samma H, Suandi SA, Mohamad-Saleh J (2018) Face sketch recognition using a hybrid optimization model. Neural Comput Appl.  https://doi.org/10.1007/s00521-018-3475-4 CrossRefGoogle Scholar
  4. 4.
    Liu D, Li J, Wang N, Peng C, Gao X (2018) Composite components-based face sketch recognition. Neurocomputing 302:46–54CrossRefGoogle Scholar
  5. 5.
    Zhang W, Wang X, Tang X (2011) Coupled information-theoretic encoding for face photo-sketch recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 513–520Google Scholar
  6. 6.
    Klare B, Li Z, Jain A (2011) Matching forensic sketches to mug shot photos. IEEE Trans Pattern Anal Mach Intell 33:639–646CrossRefGoogle Scholar
  7. 7.
    Galoogahi H, Sim T (2012) Inter-modality face sketch recognition. In: Proceedings of the IEEE international conference on multimedia and expo, pp 224–229Google Scholar
  8. 8.
    Alex A, Asari V, Mathew A (2013) Local difference of gaussian binary pattern: robust features for face sketch recognition. In: IEEE international conference on systems, man, and cybernetics, pp 1211–1216Google Scholar
  9. 9.
    Klare BF, Jain AK (2013) Heterogeneous face recognition using kernel prototype similarities. IEEE Trans Pattern Anal Mach Intell 35:1410–1422CrossRefGoogle Scholar
  10. 10.
    Tang X, Wang X (2003) Face sketch synthesis and recognition. In: Proceedings of the IEEE conference on computer vision, pp 687–694Google Scholar
  11. 11.
    Liu Q, Tang X, Jin H, Lu H, Ma S (2005) A nonlinear approach for face sketch synthesis and recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1005–1010Google Scholar
  12. 12.
    Li J, Yu X, Peng C, Wang N (2017) Adaptive representation-based face sketch-photo synthesis. Neurocomputing 269:152–159CrossRefGoogle Scholar
  13. 13.
    Wang N, Gao X, Li J (2018) Random sampling for fast face sketch synthesis. Pattern Recognit 76:215–227CrossRefGoogle Scholar
  14. 14.
    Zhang L, Lin L, Wu X, Ding S, Zhang L (2015) End-to-end photo-sketch generation via fully convolutional representation learning. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, pp 627–634Google Scholar
  15. 15.
    Wang N, Zha W, Li J, Gao X (2018) Back projection: an effective postprocessing method for gan-based face sketch synthesis. Pattern Recognit Lett 107:59–65CrossRefGoogle Scholar
  16. 16.
    Jiao L, Zhang S, Li L, Liu F, Ma W (2018) A modified convolutional neural network for face sketch synthesis. Pattern Recognit 76:125–136CrossRefGoogle Scholar
  17. 17.
    Jiang J, Yu Y, Wang Z, Liu X, Ma J (2019) Graph-regularized locality-constrained joint dictionary and residual learning for face sketch synthesis. IEEE Trans Image Process 28:628–641MathSciNetCrossRefGoogle Scholar
  18. 18.
    Lin D, Tang X (2006) Inter-modality face recognition. In: Proceedings of the 9th European conference on computer vision, pp 13–26CrossRefGoogle Scholar
  19. 19.
    Lei Z, Li S (2009) Coupled spectral regression for matching heterogeneous faces. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1123–1128Google Scholar
  20. 20.
    Lei Z, Zhou C, Yi D, Jain A, Li S (2012) An improved coupled spectral regression for heterogeneous face recognition. In: Proceedings of the international conference on biomedical, pp 7–12Google Scholar
  21. 21.
    Sharma A, Jacobs D (2011) Bypass synthesis: PLS for face recognition with pose, low-resolution and sketch. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 593–600Google Scholar
  22. 22.
    Mignon A, Jurie F (2012) CMML: a new metric learning approach for cross modal matching. In: Proceedings of the Asian conference on computer vision, pp 1–14Google Scholar
  23. 23.
    Kan M, Shan S, Zhang H, Lao S, Chen X (2012) Multi-view discriminant analysis. In: Proceedings of the 12th European conference on computer vision, pp 808–821CrossRefGoogle Scholar
  24. 24.
    Huo J, Gao Y, Shi Y, Wang Y, Yin H (2018) Heterogeneous face recognition by margin-based cross-modality metric learning. IEEE Trans Cybern 48:1814–1826CrossRefGoogle Scholar
  25. 25.
    Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823Google Scholar
  26. 26.
    Parkhi OM, Andrea V, Andrew Z (2015) Deep face recognition. Br Mach Vis Conf 1:1–6Google Scholar
  27. 27.
    Galea C, Farrugia RA (2017) Forensic face photo-sketch recognition using a deep learning-based architecture. IEEE Signal Process Lett 24:1586–1590CrossRefGoogle Scholar
  28. 28.
    Mittal P, Vatsa M, Singh R (2015) Composite sketch recognition via deep network—a transfer learning approach. In: Biometrics (ICB), 2015 IEEE international conference on, pp 251–256Google Scholar
  29. 29.
    Zhang H, Ji Y, Huang W, Liu L (2018) Sitcom-star-based clothing retrieval for video advertising: a deep learning framework. Neural Comput Appl.  https://doi.org/10.1007/s00521-018-3579-x CrossRefGoogle Scholar
  30. 30.
    Zhang H, Cao X, Ho JKL, Chow TWS (2017) Object-level video advertising: an optimization framework. IEEE Trans Ind Inf 13:520–531CrossRefGoogle Scholar
  31. 31.
    Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31:196–1955Google Scholar
  32. 32.
    Bhatt HS, Bharadwaj S, Singh R, Vatsa M (2011) Memetic approach for matching sketches with digital face images. Technical Report TR-2011-006, Indraprastha Institute of Information Technology, New Delhi, IndiaGoogle Scholar
  33. 33.
    King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758Google Scholar
  34. 34.
    Bolle RM, Connell JH, Pankanti S et al (2005) The relation between the ROC curve and the CMC. In: Fourth IEEE workshop on automatic identification advanced technologies, pp 15–20Google Scholar
  35. 35.
    Lenc L, Pavel K (2015) Automatic face recognition system based on the SIFT features. Comput Electr Eng 46:256–272CrossRefGoogle Scholar
  36. 36.
    Ding C, Choi J, Tao D, Davis LS (2016) Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans Pattern Anal Mach Intell 38:518–531CrossRefGoogle Scholar
  37. 37.
    Wu X, He R, Sun Z, Tan T (2018) A light CNN for deep face representation with noisy labels. IEEE Trans Inf Forens Secur 13:2884–2896CrossRefGoogle Scholar
  38. 38.
    Yang S, Luo P, Loy CC, Tang X (2015) From facial parts responses to face detection: a deep learning approach. In: IEEE international conference on computer vision, pp 3676–3684Google Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Division of Computer Science and EngineeringChonbuk National UniversityJeonjuKorea
  2. 2.School of Electrics and Electronic EngineeringShanghai University of Engineering ScienceShanghaiChina
  3. 3.Center for Advanced Image and Information TechnologyChonbuk National UniversityJeonjuKorea

Personalised recommendations