A Joint Local and Global Deep Metric Learning Method for Caricature Recognition

  • Wenbin Li
  • Jing Huo
  • Yinghuan Shi
  • Yang GaoEmail author
  • Lei Wang
  • Jiebo Luo
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11364)


Caricature recognition is a novel, interesting, yet challenging problem. Due to the exaggeration and distortion, there is a large cross-modal gap between photographs and caricatures, making it nontrivial to match the features of photographs and caricatures. To address the problem, a joint local and global metric learning method (LGDML) is proposed. First, joint local and global feature representation is learnt with convolutional neural networks to find both discriminant features of local facial parts and global distinctive features of the whole face. Next, in order to fuse the local and global similarities of features, a unified feature representation and similarity measure learning framework is proposed. Various methods are evaluated on the caricature recognition task. We have verified that both local and global features are crucial for caricature recognition. Moreover, experimental results show that, compared with the state-of-the-art methods, LGDML can obtain superior performance in terms of Rank-1 and Rank-10.


Caricature recognition Deep metric learning 



This work is supported by the National NSF of China (Nos. 61432008, 61673203, 61806092, U1435214), Primary R&D Plan of Jiangsu Province, China (Nos. BE2015213), Jiangsu Natural Science Foundation (Nos. BK20180326), CCF-Tencent RAGR (Nos. 20180114) and the Collaborative Innovation Center of Novel Software Technology and Industrialization.


  1. 1.
    Abaci, B., Akgul, T.: Matching caricatures to photographs. SIVP 9(1), 295–303 (2015)Google Scholar
  2. 2.
    Ahmed, E., Jones, M., Marks, T.K.: An improved deep learning architecture for person re-identification. In: CVPR, pp. 3908–3916 (2015)Google Scholar
  3. 3.
    Akleman, E.: Making caricatures with morphing. In: SIGGRAPH, p. 145 (1997)Google Scholar
  4. 4.
    Akleman, E., Reisch, J.: Modeling expressive 3D caricatures. In: SIGGRAPH, p. 61 (2004)Google Scholar
  5. 5.
    Brennan, S.E.: Caricature generator: the dynamic exaggeration of faces by computer. Leonardo 40(4), 392–400 (2007)CrossRefGoogle Scholar
  6. 6.
    Chechik, G., Sharma, V., Shalit, U., Bengio, S.: Large scale online learning of image similarity through ranking. JMLR 11, 1109–1135 (2010)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Cui, Y., Zhou, F., Lin, Y., Belongie, S.: Fine-grained categorization and dataset bootstrapping using deep metric learning with humans in the loop. In: CVPR, pp. 1153–1162 (2016)Google Scholar
  8. 8.
    Davis, J.V., Kulis, B., Jain, P., Sra, S., Dhillon, I.S.: Information-theoretic metric learning. In: ICML, pp. 209–216 (2007)Google Scholar
  9. 9.
    Duan, Y., Lu, J., Feng, J., Zhou, J.: Deep localized metric learning. TCSVT 28, 2644–2656 (2017)Google Scholar
  10. 10.
    Friedman, J., Hastie, T., Tibshirani, R.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics. Springer, Heidelberg (2009). Scholar
  11. 11.
    Han, X., Leung, T., Jia, Y., Sukthankar, R., Berg, A.C.: MatchNet: unifying feature and metric learning for patch-based matching. In: CVPR, pp. 3279–3286 (2015)Google Scholar
  12. 12.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)Google Scholar
  13. 13.
    Hu, J., Lu, J., Tan, Y.P.: Discriminative deep metric learning for face verification in the wild. In: CVPR, pp. 1875–1882 (2014)Google Scholar
  14. 14.
    Hu, J., Lu, J., Tan, Y.P.: Deep metric learning for visual tracking. TCSVT 26(11), 2056–2068 (2016)Google Scholar
  15. 15.
    Huang, C., Loy, C.C., Tang, X.: Local similarity-aware deep feature embedding. In: NIPS, pp. 1262–1270 (2016)Google Scholar
  16. 16.
    Huo, J., Gao, Y., Shi, Y., Yin, H.: Variation robust cross-modal metric learning for caricature recognition. In: ACMMM Workshop, pp. 340–348. ACM (2017)Google Scholar
  17. 17.
    Huo, J., Li, W., Shi, Y., Gao, Y., Yin, H.: WebCaricature: a benchmark for caricature face recognition. In: BMVC (2018)Google Scholar
  18. 18.
    Jain, P., Kulis, B., Dhillon, I.S., Grauman, K.: Online metric learning and fast similarity search. In: NIPS, pp. 761–768 (2009)Google Scholar
  19. 19.
    Klare, B.F., Bucak, S.S., Jain, A.K., Akgul, T.: Towards automated caricature recognition. In: ICB, pp. 139–146 (2012)Google Scholar
  20. 20.
    Koestinger, M., Hirzer, M., Wohlhart, P., Roth, P.M., Bischof, H.: Large scale metric learning from equivalence constraints. In: CVPR, pp. 2288–2295 (2012)Google Scholar
  21. 21.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)Google Scholar
  22. 22.
    Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Attribute and simile classifiers for face verification. In: ICCV, pp. 365–372 (2009)Google Scholar
  23. 23.
    Li, W., Gao, Y., Wang, L., Zhou, L., Huo, J., Shi, Y.: OPML: a one-pass closed-form solution for online metric learning. Pattern Recogn. 75, 302–314 (2018)CrossRefGoogle Scholar
  24. 24.
    Li, W., Huo, J., Shi, Y., Gao, Y., Wang, L., Luo, J.: Online deep metric learning. arXiv:1805.05510 (2018)
  25. 25.
    van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. JMLR 9(Nov), 2579–2605 (2008)Google Scholar
  26. 26.
    Mauro, R., Kubovy, M.: Caricature and face recognition. Mem. Cogn. 20(4), 433–440 (1992)CrossRefGoogle Scholar
  27. 27.
    Mignon, A., Jurie, F.: PCCA: a new approach for distance learning from sparse pairwise constraints. In: CVPR, pp. 2666–2672 (2012)Google Scholar
  28. 28.
    Oh Song, H., Xiang, Y., Jegelka, S., Savarese, S.: Deep metric learning via lifted structured feature embedding. In: CVPR, pp. 4004–4012 (2016)Google Scholar
  29. 29.
    Ouyang, S., Hospedales, T., Song, Y.-Z., Li, X.: Cross-modal face matching: beyond viewed sketches. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9004, pp. 210–225. Springer, Cham (2015). Scholar
  30. 30.
    Reale, C., Nasrabadi, N.M., Kwon, H., Chellappa, R.: Seeing the forest from the trees: a holistic approach to near-infrared heterogeneous face recognition. In: CVPR Workshop, pp. 54–62 (2016)Google Scholar
  31. 31.
    Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: CVPR, pp. 815–823 (2015)Google Scholar
  32. 32.
    Shi, Y., Li, W., Gao, Y., Cao, L., Shen, D.: Beyond IID: learning to combine non-IID metrics for vision tasks. In: AAAI, pp. 1524–1531 (2017)Google Scholar
  33. 33.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)
  34. 34.
    Szegedy, C., et al.: Going deeper with convolutions. In: CVPR, pp. 1–9 (2015)Google Scholar
  35. 35.
    Tanaka, J.W., Farah, M.J.: Parts and wholes in face recognition. Q. J. Exp. Psychol. 46(2), 225–245 (1993)CrossRefGoogle Scholar
  36. 36.
    Tseng, C.C., Lien, J.J.J., Member, I.: Colored exaggerative caricature creation using inter-and intra-correlations of feature shapes and positions. IVC 30(1), 15–25 (2012)CrossRefGoogle Scholar
  37. 37.
    Vedaldi, A., Lenc, K.: MatConvNet: convolutional neural networks for MATLAB. In: ACMMM, pp. 689–692 (2015)Google Scholar
  38. 38.
    Wang, J., et al.: Learning fine-grained image similarity with deep ranking. In: CVPR, pp. 1386–1393 (2014)Google Scholar
  39. 39.
    Weinberger, K.Q., Blitzer, J., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. In: NIPS, pp. 1473–1480 (2005)Google Scholar
  40. 40.
    Yang, W., Toyoura, M., Xu, J., Ohnuma, F., Mao, X.: Example-based caricature generation with exaggeration control. TVC 32(3), 383–392 (2016)CrossRefGoogle Scholar
  41. 41.
    Yi, D., Lei, Z., Liao, S., Li, S.Z.: Deep metric learning for person re-identification. In: ICPR, pp. 34–39 (2014)Google Scholar
  42. 42.
    Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: NIPS, pp. 3320–3328 (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Wenbin Li
    • 1
  • Jing Huo
    • 1
  • Yinghuan Shi
    • 1
  • Yang Gao
    • 1
    Email author
  • Lei Wang
    • 2
  • Jiebo Luo
    • 3
  1. 1.National Key Laboratory for Novel Software TechnologyNanjing UniversityNanjingChina
  2. 2.School of Computing and Information TechnologyUniversity of WollongongWollongongAustralia
  3. 3.Department of Computer ScienceUniversity of RochesterRochesterUSA

Personalised recommendations