Domain Adaptation via Identical Distribution Across Models and Tasks

  • Xuhong WeiEmail author
  • Yefei ChenEmail author
  • Jianbo SuEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11301)


Deep convolution neural network (CNN) models with millions of parameters trained in large-scale datasets make domain adaptation difficult to be realized. In order to be applied for different application scenarios, various light weight network models have been proposed. These models perform well in large-scale datasets but are hard to train from randomly initialized weights when lack of data. Our framework is proposed to connect a pre-trained deep model with a light weight model by enforcing feature distributions of the two models being identical. It is proved in our work that knowledge in source model can be transferred to target light weight model by identical distribution loss. Meanwhile, distribution loss allows training dataset to utilize sparse labeled data in semi-supervised classification task. Moreover, distribution loss can be applied to large amount of unlabeled data from target domain. In the experiments, several standard benchmarks on domain adaptation are evaluated and our work gets state-of-the-art performance.


Domain adaptation Model compression Identical distribution Semi-supervised method 



This paper was partially financially supported by National Natural Science Foundation of China under grants 61533012, 91748120 and 61521063.


  1. 1.
    Aytar, Y., Zisserman, A.: Tabula rasa: model transfer for object category detection. In: International Conference on Computer Vision, pp. 2252–2259 (2011)Google Scholar
  2. 2.
    Ba, L.J., Caruana, R.: Do deep nets really need to be deep? In: Advances in Neural Information Processing Systems, pp. 2654–2662 (2013)Google Scholar
  3. 3.
    Bergamo, A., Torresani, L.: Exploiting weakly-labeled web images to improve object classification: a domain adaptation approach. In: Neural Information Processing Systems, pp. 181–189 (2010)Google Scholar
  4. 4.
    Chopra, S.: DLID: deep learning for domain adaptation by interpolating between domains. In: ICML Workshop on Challenges in Representation Learning (2013)Google Scholar
  5. 5.
    Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (almost) from scratch. Mach. Learn. Res. 12(Aug), 2493–2537 (2011)zbMATHGoogle Scholar
  6. 6.
    Donahue, J., et al.: Decaf: a deep convolutional activation feature for generic visual recognition 50(1), I-647 (2013)Google Scholar
  7. 7.
    Duan, L., Tsang, I.W., Xu, D.: Domain transfer multiple kernel learning. Pattern Anal. Mach. Intell. 34(3), 465–479 (2012)CrossRefGoogle Scholar
  8. 8.
    Fernando, B., Habrard, A., Sebban, M., Tuytelaars, T.: Unsupervised visual domain adaptation using subspace alignment. In: International Conference on Computer Vision, pp. 2960–2967 (2013)Google Scholar
  9. 9.
    Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: International Conference on Machine Learning, pp. 1180–1189 (2015)Google Scholar
  10. 10.
    Girshick, R.B., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Computer Vision and Pattern Recognition, pp. 580–587 (2014)Google Scholar
  11. 11.
    Glorot, X., Bordes, A., Bengio, Y.: Domain adaptation for large-scale sentiment classification: a deep learning approach. In: Machine Learning, pp. 513–520 (2011)Google Scholar
  12. 12.
    Gong, Y., Liu, L., Yang, M., Bourdev, L.: Compressing deep convolutional networks using vector quantization. Computer Science (2014)Google Scholar
  13. 13.
    Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification. In: Computer Vision and Pattern Recognition (2017)Google Scholar
  14. 14.
    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. Comput. Sci. 14(7), 38–39 (2015)Google Scholar
  15. 15.
    Hoffman, J., et al.: LSDA: large scale detection through adaptation. In: Advances in Neural Information Processing Systems, pp. 3536–3544 (2014)Google Scholar
  16. 16.
    Hoffman, J., Rodner, E., Donahue, J., Darrell, T., Saenko, K.: Efficient learning of domain-invariant image representations. Computer Science (2013)Google Scholar
  17. 17.
    Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. In: Computer Vision and Pattern Recognition (2017)Google Scholar
  18. 18.
    Jia, Y., et al.: Caffe: convolutional architecture for fast feature embedding. In: ACM Multimedia, pp. 675–678 (2014)Google Scholar
  19. 19.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  20. 20.
    Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  21. 21.
    Mansour, Y., Mohri, M., Rostamizadeh, A.: Domain adaptation: learning bounds and algorithms. arXiv preprint arXiv:0902.3430 (2009)
  22. 22.
    Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)Google Scholar
  23. 23.
    Pan, S.J., Yang, Q., et al.: A survey on transfer learning. Knowl. Data Eng. 22(10), 1345–1359 (2010)CrossRefGoogle Scholar
  24. 24.
    Parkhi, O.M., Vedaldi, A., Zisserman, A., et al.: Deep face recognition. In: The British Machine Vision Conference, vol. 1, p. 6 (2015)Google Scholar
  25. 25.
    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)CrossRefMathSciNetGoogle Scholar
  26. 26.
    Saenko, K., Kulis, B., Fritz, M., Darrell, T.: Adapting visual category models to new domains. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 213–226. Springer, Heidelberg (2010). Scholar
  27. 27.
    Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., Lecun, Y.: Overfeat: integrated recognition, localization and detection using convolutional networks. Eprint Arxiv (2013)Google Scholar
  28. 28.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Computer Science (2014)Google Scholar
  29. 29.
    Szegedy, C., Toshev, A., Erhan, D.: Deep neural networks for object detection. In: Advances in Neural Information Processing Systems, vol. 26, pp. 2553–2561 (2013)Google Scholar
  30. 30.
    Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks. In: IEEE International Conference on Computer Vision, pp. 4068–4076 (2017)Google Scholar
  31. 31.
    Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: Maximizing for domain invariance. Computer Science (2014)Google Scholar
  32. 32.
    Yang, J., Yan, R., Hauptmann, A.G.: Adapting SVM classifiers to data with shifted distributions. In: International Conference Data Mining Workshops, pp. 69–76. IEEE (2007)Google Scholar
  33. 33.
    Yi, D., Lei, Z., Liao, S., Li, S.Z.: Learning face representation from scratch. Computer Science (2014)Google Scholar
  34. 34.
    Zhang, K., Schölkopf, B., Muandet, K., Wang, Z.: Domain adaptation under target and conditional shift. In: International Conference on Machine Learning, pp. 819–827 (2013)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Shanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations