Pairwise Generalization Network for Cross-Domain Image Recognition

  • Y. B. Liu
  • T. T. Han
  • Z. GaoEmail author


In recent years, convolutional neural networks have received increasing attention from the computer vision and machine learning communities. Due to the differences in the distribution, tone and brightness of the training domain and test domain, researchers begin to focus on cross-domain image recognition. In this paper, we propose a Pairwise Generalization Network (PGN) for addressing the problem of cross-domain image recognition where Instance Normalization and Batch Normalization are added to enhance their abilities in the original domain and to expand to the new domain. Meanwhile, the Siamese architecture is utilized in the PGN to learn an embedding subspace that is discriminative, and map positive sample pairs aligned and negative sample pairs separated, which can work well even with only few labeled target data samples. We also add residual architecture and MMD loss for the PGN model to further improve its performance. Extensive experiments on two different public benchmarks show that our PGN solution significantly outperforms the state-of-the-art methods.


Cross-domain Image recognition Pairwise 



  1. 1.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. NIPSGoogle Scholar
  2. 2.
    Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. NIPSGoogle Scholar
  3. 3.
    Chen L, Papandreou G, Kokkinos I, Murphy K, Yuille A (2017) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMIGoogle Scholar
  4. 4.
    Tzeng E, Hoffman J, Darrell T, Saenko K (2015) Simultaneous deep transfer across domains and tasks. In: ICCVGoogle Scholar
  5. 5.
    Koniusz P, Tas Y, Porikli F (2017) Domain adaptation by mixture of alignments of second- or higher-order scatter tensors. In: The IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  6. 6.
    Gao Z, Han TT, Zhu L, Zhang H, Wang Y (2018) Exploring the cross-domain action recognition problem by deep feature learning and cross-domain learning. IEEE Access 6:68989–69008. CrossRefGoogle Scholar
  7. 7.
    Liu M-Y, Tuzel O (2016) Coupled generative adversarial networks. In: Advances in neural information processing systems, pp 469–477Google Scholar
  8. 8.
    Tzeng E, Hoffman J, Saenko K, Darrell T (2017) Adversarial discriminative domain adaptation. In: The IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  9. 9.
    Yao T, Pan Y, Ngo C-W, Li H, Mei T (2015) Semi-supervised domain adaptation with subspace learning for visual recognition. In: The IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  10. 10.
    Haeusser P, Frerix T, Mordvintsev A, et al (2017) Associative domain adaptation. In: ICCV, 2017: 2784–2792Google Scholar
  11. 11.
    Pan SJ, Tsang IW, Kwok JT, Yang Q (2011) Domain adaptation via transfer component analysis. IEEE Trans Neural Netw 22(2):199–210. CrossRefGoogle Scholar
  12. 12.
    Gong B, Shi Y, Sha F, Grauman K (2012) Geodesic flow kernel for unsupervised domain adaptation. In: IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  13. 13.
    Long M, Wang J, Ding G, et al. (2013) Transfer feature learning with joint distribution adaptation. In: Proceedings of the IEEE international conference on computer vision, pp 2200–2207Google Scholar
  14. 14.
    Long M, Wang J, Ding G, Sun J, Yu PS (2014) Transfer joint matching for unsupervised domain adaptation. In: IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  15. 15.
    Daum III H (2009) Frustratingly easy domain adaptation. CoRR, arXiv:0907.1815
  16. 16.
    Motiian S, Piccirilli M, Adjeroh DA, Doretto G (2017) Unified deep supervised domain adaptation and generalization. In: The IEEE international conference on computer vision (ICCV), pp 5715–5725Google Scholar
  17. 17.
    Long M, Cao Y, Wang J, et al. (2015) Learning transferable features with deep adaptation networks. In: International conference on machine learning, 97–105Google Scholar
  18. 18.
    Lanckriet GRG, Cristianini N, Ghaoui LE, Bartlett P, Jordan MI (2004) Learning the kernel matrix with semidefinite programming. J Mach Learn Res 5:27–72MathSciNetzbMATHGoogle Scholar
  19. 19.
    Duan L, Tsang IW, Xu D (2012) Domain transfer multiple kernel learning. IEEE Trans Pattern Anal Mach Intell 34(3):465–479CrossRefGoogle Scholar
  20. 20.
    Borgwardt KM et al (2006) Integrating structured biological data by kernel maximum mean discrepancy. In: Proceedings of international conference on intelligence system molecular biology, Fortaleza, Brazil, 49–57Google Scholar
  21. 21.
    Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky VS (2016) Domain-adversarial training of neural networks. J Mach Learn ResGoogle Scholar
  22. 22.
    Bousmalis K, Trigeorgis G, Silberman N, Erhan D, Krishnan D (2016) Domain separation networks. In: Annual conference on neural information processing systems (NIPS)Google Scholar
  23. 23.
    Long M, Wang J, Jordan MI (2016) Deep transfer learning with joint adaptation networks. CoRR arXiv:1605.06636
  24. 24.
    Gopalan R, Li R, Chellappa R (2011) Domain adaptation for object recognition: an unsupervised approach. In: ICCV 2011, vol, 24, no. 4, 999–1006 (2011)Google Scholar
  25. 25.
    Fernando B, Habrard A, Sebban M, et al (2014) Unsupervised visual domain adaptation using subspace alignment. In: IEEE international conference on computer vision. IEEE, 2960–2967Google Scholar
  26. 26.
    Kulis B, Saenko K, Darrell T (2011) What you saw is not what you get: domain adaptation using asymmetric kernel transforms. In: IEEE conference on computer vision and pattern recognition. IEEE Computer Society, 1785–1792Google Scholar
  27. 27.
    Baktashmotlagh M, Harandi MT, Lovell BC, et al (2013) Unsupervised domain adaptation by domain invariant projection. In: IEEE International conference on computer vision. IEEE Computer Society, 769–776Google Scholar
  28. 28.
    Aytar Y, Zisserman A (2011) Tabula rasa: model transfer for object category detection. In: Computer vision (ICCV), 2011 IEEE International Conference on. IEEE, 2252–2259Google Scholar
  29. 29.
    Becker CJ, Christoudias CM, Fua P (2013) Non-linear domain adaptation with boosting. In: Advances in neural information processing systems, 485–493Google Scholar
  30. 30.
    Bergamo A, Torresani L (2010) Exploiting weakly-labeled web images to improve object classification: a domain adaptatio napproach. In: Advances in neural information processing systems, 181–189Google Scholar
  31. 31.
    Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification. In: Computer vision and pattern recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol 1, 539–546. IEEEGoogle Scholar
  32. 32.
    Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR, San Diego, California, USAGoogle Scholar
  33. 33.
    Varior RR, Shuai B, Lu J, Xu D, Wang G (2016) A siamese long short-term memory architecture for human reidentification. In: European conference on computer vision. Springer, Berlin, 135–153Google Scholar
  34. 34.
    Kumar B, Carneiro G, Reid I, et al (2016) Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 5385–5394Google Scholar
  35. 35.
    Sun B, Saenko K (2016) Deep coral: correlation alignment for deep domain adaptation. In: Computer vision–ECCV 2016 workshops, Springer, Berlin, 443–450Google Scholar
  36. 36.
    Rozantsev A, Salzmann M, Fua P (2016) Beyond sharing weights for deep domain adaptation. arXiv preprint arXiv:1603.06432
  37. 37.
    Rozantsev A, Salzmann M, Fua P (2018) Residual parameter transfer for deep domain adaptation. In: CVPRGoogle Scholar
  38. 38.
    Blanchard G, Lee G, Scott C (2011) Generalizing from several related classification tasks to a new unlabeled sample. In: Advances in neural information processing systems, 2178–2186Google Scholar
  39. 39.
    Muandet K, Balduzzi D, Scholkopf B (2013) Domain generalization via invariant feature representation. In: ICML(1), 10–18Google Scholar
  40. 40.
    Ghifary M, Bastiaan Kleijn W, Zhang M, Balduzzi D (2015) Domain generalization for object recognition with multi-task autoencoders. In: Proceedings of the IEEE international conference on computer vision, 2551–2559Google Scholar
  41. 41.
    Ghifary M, Balduzzi D, Kleijn WB, Zhang M (2017) Scatter component analysis: a unified framework for domain adaptation and domain generalization. IEEE Trans Pattern Anal Mach IntellGoogle Scholar
  42. 42.
    Xu Z, Li W, Niu L, Xu D (2014) Exploiting low-rank structure from latent domains for domain generalization. In: ECCV, 628–643Google Scholar
  43. 43.
    Niu L, Li W, Xu D, Cai J (2017) An exemplar-based multiview domain generalization framework for visual recognition. IEEE Trans Neural Netw Learn Syst 29(2):259–272CrossRefGoogle Scholar
  44. 44.
    Niu L, Li W, Xu D (2016) Multi-view domain generalization for visual recognition. In: IEEE International conference on computer vision. IEEE, 4193–4201 (2016)Google Scholar
  45. 45.
    Khosla A, Zhou T, Malisiewicz T, Efros AA, Torralba A (2012) Undoing the damage of dataset bias. In: European conference on computer vision. Springer, Berlin, 158–171Google Scholar
  46. 46.
    Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. ICMLGoogle Scholar
  47. 47.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A, et al (2015) Going deeper with convolutions. CVPRGoogle Scholar
  48. 48.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. CVPRGoogle Scholar
  49. 49.
    Huang G, Liu Z, Weinberger KQ, van der Maaten L (2017) Densely connected convolutional networks. CVPRGoogle Scholar
  50. 50.
    He X, He Z, Song J et al. (2018) NAIS: neural attentive item similarity model for recommendation. IEEE Trans Knowl Data Eng 1–1Google Scholar
  51. 51.
    Zhang H, Kyaw Z, Yu J, et al (2017) PPR-FCN: weakly supervised visual relation detection via parallel pairwise R-FCNGoogle Scholar
  52. 52.
    Chen J, Zhang H, He X, et al (2017) Attentive collaborative filtering: multimedia recommendation with item- and component-level attention. In: International ACM SIGIR conference on research and development in information retrievalGoogle Scholar
  53. 53.
    Cheng Z, Chang X, Zhu L et al (2018) MMALFM: explainable recommendation by leveraging reviews and images. ACM Trans Inf SystGoogle Scholar
  54. 54.
    Gao Z, Wang DY, Xue YB, Xu GP, Zhang H, Wang YL (2018) 3D object recognition based on pairwise multi-view convolutional neural networks. J Vis Commun Image Represent 56:305–315CrossRefGoogle Scholar
  55. 55.
    Gao Z, Wang D, Wan SH, Zhang H, Wang YL (2019) Cognitive-inspired class-statistic matching with triple-constrain for camera free 3D object retrieval. Future Gener Comput Syst 94:641–653CrossRefGoogle Scholar
  56. 56.
    Nie W, Liu A, Gao Y, Su Y (2018) Hyper-clique graph matching and applications. In: IEEE transactions on circuits and systems for video technology.
  57. 57.
    Nie W, Cheng H, Su Y (2017) Modeling temporal information of mitotic for mitotic event detection. IEEE Trans Big Data, (99): 1–1Google Scholar
  58. 58.
    Liu AA, Nie WZ, Yue G et al (2017) View-based 3-D model retrieval: a benchmark. IEEE Trans Cybern 48(3):916–928Google Scholar
  59. 59.
    Gao Z, Zhang H, Xu GP, Xue YB, Hauptmannc AG (2015) Multi-view discriminative and structured dictionary learning with group sparsity for human action recognition. Signal Process 112:83–97CrossRefGoogle Scholar
  60. 60.
    Ulyanov D, Vedaldi A, Lempitsky V (2017) Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis. CVPRGoogle Scholar
  61. 61.
    Dumoulin V, Shlens J, Kudlur M (2017) A learned representation for artistic style. ICLRGoogle Scholar
  62. 62.
    Huang X, Belongie S (2017) Arbitrary style transfer in real-time with adaptive instance normalization. ICCVGoogle Scholar
  63. 63.
    Pan X, Luo P, Shi J, et al (2018) Two at once: enhancing learning and generalization capacities via IBN-NetGoogle Scholar
  64. 64.
    Saenko K, Kulis B, Fritz M, Darrell T (2010) Adapting visual category models to new domains. In: ECCV, 213–226 (2010)Google Scholar
  65. 65.
    LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient based learning applied to document recognition. Proc IEEE 86(11):2278–2324CrossRefGoogle Scholar
  66. 66.
    Hull JJ (1994) A database for handwritten text recognition research. IEEE Trans Pattern Anal Mach Intell 16(5):550–554CrossRefGoogle Scholar
  67. 67.
    Fernando B, Tommasi T, Tuytelaarsc T (2015) Joint cross-domain classification and subspace learning for unsupervised adaptation. Pattern Recogit Lett 65:60–66CrossRefGoogle Scholar
  68. 68.
    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet large scale visual recognition challenge. IJCVGoogle Scholar
  69. 69.
    Tzeng E, Hoffman J, Saenko K, Darrell T (2017) Adversarial discriminative domain adaptation. In: The IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Tianjin Key Laboratory of Intelligence Computing and Novel Software Technology and Key Laboratory of Computer Vision and System, Ministry of EducationTianjin University of TechnologyTianjinChina
  2. 2.Qilu University of Technology (Shandong Academy of Sciences), Shandong Computer Science Center (National Supercomputer Center in Jinan)Shandong Artifical Intelligence InstituteJinanPeople’s Republic of China

Personalised recommendations