Skip to main content
SpringerLink
Log in

Transfer Learning: Domain Adaptation

  • Chapter
  • First Online:
Deep Learning for NLP and Speech Recognition

Abstract

Domain adaptation is a form of transfer learning, in which the task remains the same, but there is a domain shift or a distribution change between the source and the target. As an example, consider a model that has learned to classify reviews on electronic products for positive and negative sentiments, and is used for classifying the reviews for hotel rooms or movies. The task of sentiment analysis remains the same, but the domain (electronics and hotel rooms) has changed. The application of the model to a separate domain poses many problems because of the change between the training data and the unseen testing data, typically known as domain shift. For example, sentences containing phrases such as “loud and clear” will be mostly considered positive in electronics whereas negative in hotel room reviews. Similarly, usage of keywords such as “lengthy” or “boring” which may be prevalent in domains such as book reviews might be completely absent in domains such as kitchen equipment reviews.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 119.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. John Blitzer, Mark Dredze, and Fernando Pereira. “Biographies, Bollywood, boomboxes and blenders: Domain adaptation for sentiment classification”. In: ACL. 2007, pp. 187–205.

    Google Scholar 

  2. Konstantinos Bousmalis et al. “Domain Separation Networks”. In: Advances in Neural Information Processing Systems 29. Ed. by D. D. Lee et al. 2016, pp. 343–351.

    Google Scholar 

  3. Lorenzo Bruzzone and Mattia Marconcini. “Domain Adaptation Problems: A DASVM Classification Technique and a Circular Validation Strategy”. In: IEEE Trans. Pattern Anal. Mach. Intell. 32.5 (May 2010), pp. 770–787.

    Google Scholar 

  4. Minmin Chen et al. “Marginalized Denoising Autoencoders for Domain Adaptation”. In: Proceedings of the 29th International Conference on International Conference on Machine Learning. ICML’12. 2012, pp. 1627–1634.

    Google Scholar 

  5. Sumit Chopra, Suhrid Balakrishnan, and Raghuraman Gopalan. “DLID: Deep learning for domain adaptation by interpolating between domains”. In: in ICML Workshop on Challenges in Representation Learning. 2013.

    Google Scholar 

  6. Gabriela Csurka, ed. Domain Adaptation in Computer Vision Applications. Advances in Computer Vision and Pattern Recognition. Springer, 2017.

    Google Scholar 

  7. Ido Dagan et al. “Zero-Shot Transfer Learning for Event Extraction”. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018. 2018, pp. 2160–2170.

    Google Scholar 

  8. Daniele Falavigna et al. “DNN adaptation by automatic quality estimation of ASR hypotheses”. In: Computer Speech & Language 46 (2017), pp. 585–604.

    Article  Google Scholar 

  9. Li Fei-Fei, Rob Fergus, and Pietro Perona. “One-Shot Learning of Object Categories”. In: IEEE Trans. Pattern Anal. Mach. Intell. 28.4 (Apr. 2006), pp. 594–611.

    Google Scholar 

  10. Dmitry Kalenichenko Florian Schroff and James Philbin. “FaceNet: A unified embedding for face recognition and clustering”. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, 2015, pp. 815–823.

    Google Scholar 

  11. Lisheng Fu et al. “Domain Adaptation for Relation Extraction with Domain Adversarial Neural Network”. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP. 2017, pp. 425–429.

    Google Scholar 

  12. Yaroslav Ganin et al. “Domain-adversarial Training of Neural Networks”. In: J. Mach. Learn. Res. 17.1 (Jan. 2016), pp. 2096–2030.

    Google Scholar 

  13. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. “Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach”. In: Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011. 2011, pp. 513–520.

    Google Scholar 

  14. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. “Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach”. In: Proceedings of the 28th International Conference on Machine Learning, ICML. 2011, pp. 513–520.

    Google Scholar 

  15. Boqing Gong, Kristen Grauman, and Fei Sha. “Connecting the Dots with Landmarks: Discriminatively Learning Domain-invariant Features for Unsupervised Domain Adaptation”. In: Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28. ICML’13. 2013, pp. I–222–I–230.

    Google Scholar 

  16. Ehsan Hosseini-Asl et al. “A Multi-Discriminator CycleGAN for Unsupervised Non-Parallel Speech Domain Adaptation”. In: CoRR abs/1804.00522 (2018).

    Google Scholar 

  17. Koray Kavukcuoglu, Marc’Aurelio Ranzato, and Yann LeCun. “Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition”. In: CoRR abs/1010.3467 (2010).

    Google Scholar 

  18. Gil Keren et al. “Weakly Supervised One-Shot Detection with Attention Siamese Networks”. In: CoRR abs/1801.03329 (2018).

    Google Scholar 

  19. Roger Levy and Lucia Specia, eds. Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada, August 3–4, 2017. Association for Computational Linguistics, 2017.

    Google Scholar 

  20. Ming-Yu Liu and Oncel Tuzel. “Coupled Generative Adversarial Networks”. In: Advances in Neural Information Processing Systems 29. Ed. by D. D. Lee et al. 2016, pp. 469–477.

    Google Scholar 

  21. Mingsheng Long et al. “Learning Transferable Features with Deep Adaptation Networks”. In: Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37. ICML’15. 2015, pp. 97–105.

    Google Scholar 

  22. Yukun Ma, Erik Cambria, and Sa Gao. “Label Embedding for Zero-shot Fine-grained Named Entity Typing”. In: COLING 2016, 26th International Conference on Computational Linguistics. 2016, pp. 171–180.

    Google Scholar 

  23. Zhong Meng et al. “Unsupervised adaptation with domain separation networks for robust speech recognition”. In: 2017 IEEE Automatic Speech Recognition and Understanding Workshop. 2017, pp. 214–221.

    Google Scholar 

  24. Tom M. Mitchell, Shashank Srivastava, and Igor Labutov “Zero-shot Learning of Classifiers from Natural Language Quantification”. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15–20, 2018, Volume 1: Long Papers. 2018, pp. 306–316.

    Google Scholar 

  25. Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. De l’Imprimerie Royale, 1781.

    Google Scholar 

  26. Thien Huu Nguyen and Ralph Grishman. “Employing Word Representations and Regularization for Domain Adaptation of Relation Extraction”. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL. 2014, pp. 68–74.

    Google Scholar 

  27. Thien Huu Nguyen and Ralph Grishman. “Event Detection and Domain Adaptation with Convolutional Neural Networks”. In: Proceedings of the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing. 2015, pp. 365–371.

    Google Scholar 

  28. Thien Huu Nguyen, Barbara Plank, and Ralph Grishman. “Semantic Representations for Domain Adaptation: A Case Study on the Tree Kernel-based Method for Relation Extraction”. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL. 2015, pp. 635–644.

    Google Scholar 

  29. Mark Palatucci et al. “Zero-shot Learning with Semantic Output Codes”. In: NIPS. Curran Associates, Inc., 2009, pp. 1410–1418.

    Google Scholar 

  30. Sinno Jialin Pan et al. “Domain Adaptation via Transfer Component Analysis”. In: IEEE Trans. Neural Networks 22.2 (2011), pp. 199–210.

    Google Scholar 

  31. Pushpankar Kumar Pushp and Muktabh Mayank Srivastava. “Train Once, Test Anywhere: Zero-Shot Learning for Text Classification”. In: CoRR abs/1712.05972 (2017).

    Google Scholar 

  32. Ievgen Redko, Amaury Habrard, and Marc Sebban. “Theoretical Analysis of Domain Adaptation with Optimal Transport”. In: Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18–22, 2017, Proceedings, Part II. 2017, pp. 737–753.

    Google Scholar 

  33. Mengye Ren et al. “Meta-Learning for Semi-Supervised Few-Shot Classification”. In: CoRR abs/1803.00676 (2018).

    Google Scholar 

  34. Lina Maria Rojas-Barahona et al. “Nearly Zero-Shot Learning for Semantic Decoding in Spoken Dialogue Systems”. In: CoRR abs/1806.05484 (2018).

    Google Scholar 

  35. Dino Sejdinovic et al. “Equivalence of distance-based and RKHS-based statistics in hypothesis testing”. In: CoRR abs/1207.6076 (2012).

    Google Scholar 

  36. Dmitriy Serdyuk et al. “Invariant Representations for Noisy Speech Recognition”. In: CoRR abs/1612.01928 (2016).

    Google Scholar 

  37. Baochen Sun and Kate Saenko. “Deep CORAL: Correlation Alignment for Deep Domain Adaptation”. In: ECCV Workshops (3). Vol. 9915. Lecture Notes in Computer Science. 2016, pp. 443–450.

    Google Scholar 

  38. Sining Sun et al. “Domain Adversarial Training for Accented Speech Recognition”. In: CoRR abs/1806.02786 (2018).

    Google Scholar 

  39. Sining Sun et al. “Training Augmentation with Adversarial Examples for Robust Speech Recognition”. In: CoRR abs/1806.02782 (2018).

    Google Scholar 

  40. Eleni Triantafillou, Richard S. Zemel, and Raquel Urtasun. “Few-Shot Learning Through an Information Retrieval Lens”. In: NIPS. 2017, pp. 2252–2262.

    Google Scholar 

  41. Eric Tzeng et al. “Deep Domain Confusion: Maximizing for Domain Invariance”. In: CoRR abs/1412.3474 (2014).

    Google Scholar 

  42. Eric Tzeng et al. “Simultaneous Deep Transfer Across Domains and Tasks”. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV). ICCV ’15. 2015, pp. 4068–4076.

    Google Scholar 

  43. Eric Tzeng et al. “Adversarial Discriminative Domain Adaptation”. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21–26, 2017. 2017, pp. 2962–2971.

    Google Scholar 

  44. Ke Wang et al. “Empirical Evaluation of Speaker Adaptation on DNN based Acoustic Model”. In: CoRR abs/1803.10146 (2018).

    Google Scholar 

  45. Yongqin Xian, Bernt Schiele, and Zeynep Akata. “Zero-Shot Learning - The Good, the Bad and the Ugly”. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR. 2017, pp. 3077–3086.

    Google Scholar 

  46. Leiming Yan, Yuhui Zheng, and Jie Cao. “Few-shot learning for short text classification”. In: Multimedia Tools and Applications (2018), pp. 1–12.

    Google Scholar 

  47. Majid Yazdani and James Henderson. “A Model of Zero-Shot Learning of Spoken Language Understanding”. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015, pp. 244–249.

    Google Scholar 

  48. Dani Yogatama et al. “Generative and Discriminative Text Classification with Recurrent Neural Networks”. In: CoRR abs/1703.01898 (2017).

    Google Scholar 

  49. Donghyun Yoo et al. “Efficient K-Shot Learning With Regularized Deep Networks”. In: AAAI. AAAI Press, 2018, pp. 4382–4389.

    Google Scholar 

  50. Jianfei Yu and Jing Jiang. “Learning Sentence Embeddings with Auxiliary Tasks for Cross-Domain Sentiment Classification”. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1–4, 2016. 2016, pp. 236–246.

    Google Scholar 

  51. Werner Zellinger et al. “Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning”. In: CoRR abs/1702.08811 (2017).

    Google Scholar 

  52. Yuan Zhang, Regina Barzilay, and Tommi S. Jaakkola. “Aspect-augmented Adversarial Networks for Domain Adaptation”. In: TACL 5 (2017), pp. 515–528.

    Article  Google Scholar 

  53. Guangyou Zhou et al. “Bi-Transferring Deep Neural Networks for Domain Adaptation”. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL. 2016.

    Google Scholar 

  54. Jun-Yan Zhu et al. “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”. In: Computer Vision (ICCV), 2017 IEEE International Conference on. 2017.

    Google Scholar 

  55. Yftah Ziser and Roi Reichart. “Neural Structural Correspondence Learning for Domain Adaptation”. In: Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). 2017, pp. 400–410.

    Google Scholar 

  56. Yftah Ziser and Roi Reichart. “Pivot Based Language Modeling for Improved Neural Domain Adaptation”. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT. 2018, pp. 1241–1251.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Digital Reasoning Systems Inc., McLean, VA, USA

    Uday Kamath & James Whitaker

  2. Intelluron Corporation, Nashville, TN, USA

    John Liu

Authors
  1. Uday Kamath
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. James Whitaker
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Kamath, U., Liu, J., Whitaker, J. (2019). Transfer Learning: Domain Adaptation. In: Deep Learning for NLP and Speech Recognition . Springer, Cham. https://doi.org/10.1007/978-3-030-14596-5_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-14596-5_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-14595-8

  • Online ISBN: 978-3-030-14596-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation