Skip to main content
SpringerLink
Log in

Deep Domain Adaptation for Regression

  • Chapter
  • First Online:
Development and Analysis of Deep Learning Architectures

Part of the book series: Studies in Computational Intelligence ((SCI,volume 867))

Abstract

Developing machine learning algorithms under the constraint of limited labeled data has attracted significant attention in the research community in recent years. Domain adaptation or transfer learning algorithms alleviate this challenge by transferring relevant knowledge from a source domain to induce a model for a related target domain, where labeled data are scarce. Further, deep learning algorithms are instrumental in learning informative feature representations from a given dataset and have replaced the need for hand-crafted features. In this chapter, we propose a novel framework, DeepDAR, for domain adaptation for regression applications, using deep convolutional neural networks (CNNs). We formulate a loss function relevant to the research task and exploit the gradient descent algorithm to optimize the loss and train the deep CNN. To the best of our knowledge, domain adaptation for regression applications using deep neural networks has not been explored in the literature. Our extensive empirical studies on two popular regression applications (age estimation and head pose estimation from images) depict the merit of our framework over competing baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aldecoa, R., Marín, I.: Deciphering network community structure by surprise. PLoS ONE 6(9), e24195 (2011)

    Article  Google Scholar 

  2. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN. arXiv:1701.07875 (2017)

  3. Aytar, Y., Zisserman, A.: Tabula rasa: model transfer for object category detection. In: IEEE International Conference on Computer Vision (ICCV), pp. 2252–2259 (2011)

    Google Scholar 

  4. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoderdecoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 39(12), 2481–2495 (2017)

    Article  Google Scholar 

  5. Bay, H., Tuytelaars, T., VanGool, L.: Surf: speeded up robust features. In: European Conference on Computer Vision (ECCV), pp. 404–417 (2006)

    Chapter  Google Scholar 

  6. Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J.: A theory of learning from different domains. Mach. Learn. 79(1–2), 151–175 (2010)

    Article  MathSciNet  Google Scholar 

  7. Bhattarai, B., Sharma, G., Lechervy, A., Jurie, F.: A joint learning approach for cross domain age estimation. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 1901–1905 (2016)

    Google Scholar 

  8. Borghi, G., Fabbri, M., Vezzani, R., Calderara, S., Cucchiara, R.: Face-from-depth for head pose estimation on depth images. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2018)

    Google Scholar 

  9. Borgwardt, K., Gretton, A., Rasch, M., Kriegel, H., Scholkopf, B., Smola, A.: Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22(14), e49–e57 (2006)

    Article  Google Scholar 

  10. Bruzzone, L., Marconcini, M.: Domain adaptation problems: a dasvm classification technique and a circular validation strategy. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 32(5), 770–787 (2010)

    Article  Google Scholar 

  11. Cao, B., Pan, S., Zhang, Y., Yeung, D., Yang, Q.: Adaptive transfer learning. In: Association for the Advancement of Artificial Intelligence (AAAI), pp. 407–412 (2010)

    Google Scholar 

  12. Chapelle, O., Scholkopf, B., Zien, A.: Semi-Supervised Learning. MIT Press (2006)

    Google Scholar 

  13. Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: delving deep into convolutional nets. In: British Machine Vision Conference (BMVC) (2014)

    Google Scholar 

  14. Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 40(4), 834–848 (2018)

    Article  Google Scholar 

  15. Cortes, C., Mohri, M.: Domain adaptation in regression. In: International Conference on Algorithmic Learning Theory, pp. 308–323 (2011)

    MATH  Google Scholar 

  16. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 886–893 (2005)

    Google Scholar 

  17. Dong, X., Thanou, D., Frossard, P., Vandergheynst, P.: Learning laplacian matrix in smooth graph signal representations. IEEE Trans. Signal Process. 64(23), 6160–6173 (2016)

    Article  MathSciNet  Google Scholar 

  18. Duan, L., Tsang, I., Xu, D., Maybank, S.: Domain transfer SVM for video concept detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1375–1381 (2009)

    Google Scholar 

  19. Escalera, S., Fabian, J., et al., P.P.: Chalearn looking at people 2015: apparent age and cultural event recognition datasets and results. In: IEEE International Conference on Computer Vision Workshop (ICCVW), pp. 243–251 (2015)

    Google Scholar 

  20. Fanelli, G., Dantone, M., GallA, J., Fossati, A., Gool, L.V.: Random forests for real time 3D face analysis. Int. J. Comput. Vis. (IJCV) 101(3), 437–458 (2013)

    Article  Google Scholar 

  21. Fernando, B., Habrard, A., Sebban, M., Tuytelaars, T.: Unsupervised visual domain adaptation using subspace alignment. In: IEEE Inrternational Conference on Computer Vision (ICCV), pp. 2960–2967 (2013)

    Google Scholar 

  22. Fu, Y., Huang, T.: Graph embedded analysis for head pose estimation. In: International Conference on Automatic Face and Gesture Recognition (2006)

    Google Scholar 

  23. Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. (JMLR) 17(1), 2096–2130 (2016)

    MathSciNet  MATH  Google Scholar 

  24. Garcke, J., Vanck, T.: Importance weighted inductive transfer learning for regression. In: European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 466–481 (2014)

    Chapter  Google Scholar 

  25. Geng, X., Zhou, Z., Miles, K.: Automatic age estimation based on facial aging patterns. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 29(12), 2234–2240 (2007)

    Article  Google Scholar 

  26. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances of Neural Information Processing Systems (NIPS), pp. 2672–2680 (2014)

    Google Scholar 

  27. Gopalan, R., Li, R., Chellappa, R.: Domain adaptation for object recognition: an unsupervised approach. In: IEEE International Conference on Computer Vision (ICCV), pp. 999–1006 (2011)

    Google Scholar 

  28. Guan, Z., Li, A., Zhu, T.: Local regression transfer learning with applications to users’ psychological characteristics prediction. Brain Inf. 2(3), 145–153 (2015)

    Article  Google Scholar 

  29. Guo, G., Fu, Y., Dyer, C., Huang, T.: Image-based human age estimation by manifold learning and locally adjusted robust regression. IEEE Trans. Image Process. 17(7), 1178–1188 (2008)

    Article  MathSciNet  Google Scholar 

  30. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  31. Hu, L., Kan, M., Shan, S., Chen, X.: Duplex generative adversarial network for unsupervised domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1498–1507 (2018)

    Google Scholar 

  32. Jamal, A., Namboodiri, V., Deodhare, D., Venkatesh, K.: Deep domain adaptation in action space. In: British Machine Vision Conference (BMVC) (2018)

    Google Scholar 

  33. Khosla, A., Sarma, A., Hamid, R.: What makes an image popular? In: International World Wide Web Conference (WWW), pp. 867–876 (2014)

    Google Scholar 

  34. Kingma, D., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  35. Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: Neural Information Processing Systems (NIPS) (2012)

    Google Scholar 

  36. Lai, S., Xu, L., Liu, K., Zhao, J.: Recurrent convolutional neural networks for text classification. In: Association for the Advancement of Artificial Intelligence (AAAI), pp. 2267–2273 (2015)

    Google Scholar 

  37. Le, T., Nguyen, K., Ho, N., Bui, H., Phung, D.: Theoretical perspective of deep domain adaptation. arXiv:1811.06199 (2019)

  38. LeCun, Y., Bottou, L., Orr, G., Muller, K.: Efficient backprop. In: Neural Networks: Tricks of the Trade, pp. 9–50 (1998)

    Google Scholar 

  39. Liang, Z., Zhang, G., Huang, J., Hu, Q.: Deep learning for healthcare decision making with EMRs. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 556–559 (2014)

    Google Scholar 

  40. Liu, M., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Advances of Neural Information Processing Systems (NIPS), pp 700–708 (2017)

    Google Scholar 

  41. Liu, M., Tuzel, O.: Coupled generative adversarial networks. In: Advances of Neural Information Processing Systems (NIPS), pp. 469–477 (2016)

    Google Scholar 

  42. Liu, S., Liu, S., Cai, W., Pujol, S., Kikinis, R., Feng, D.: Early diagnosis of Alzheimer’s disease with deep learning. In: IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1015–1018 (2014)

    Google Scholar 

  43. Long, M., Cao, Y., Wang, J., Jordan, M.: Learning transferable features with deep adaptation networks. In: International Conference on Machine Learning (ICML), pp. 97–105 (2015)

    Google Scholar 

  44. Long, M., Zhu, H., Wang, J., Jordan, M.: Unsupervised domain adaptation with residual transfer networks. In: Advances of Neural Information Processing Systems (NIPS), pp. 136–144 (2016)

    Google Scholar 

  45. Luu, T., Low, K., Qu, X., Lim, H., Hoon, K.: An individual-specific gait pattern prediction model based on generalized regression neural networks. Gait and Posture 39(1), 443–448 (2014)

    Article  Google Scholar 

  46. Motiian, S., Jones, Q., Iranmanesh, S., Doretto, G.: Few-shot adversarial domain adaptation. In: Advances of Neural Information Processing Systems (NIPS) (2017)

    Google Scholar 

  47. Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. International Journal of Computer Vision (IJCV) 42(3), 145–175 (2001)

    Article  Google Scholar 

  48. Pan, S., Tsang, I., Kwok, J., Yang, Q.: Domain adaptation via transfer component analysis. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 1187–1192 (2009)

    Google Scholar 

  49. Pan, S., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. (TKDE) 22(10), 1345–1359 (2010)

    Article  Google Scholar 

  50. Pan, S., Tsang, I., Kwok, J., Yang, Q.: Domain adaptation via transfer component analysis. IEEE Trans. Neural Netw. Learn. Syst. (TNNLS) 22(2), 199–210 (2011)

    Article  Google Scholar 

  51. Pardoe, D., Stone, P.: Boosting for regression transfer. In: International Conference on Machine Learning (ICML), pp. 863–870 (2010)

    Google Scholar 

  52. Peng, K., Wu, Z., Ernst, J.: Zero-shot deep domain adaptation. In: European Conference on Computer Vision (ECCV), pp. 793–810 (2018)

    Chapter  Google Scholar 

  53. Pinto, H., Almeida, J., Goncalves, M.: Using early view patterns to predict the popularity of youtube videos. In: ACM International Conference on Web Search and Data Mining (WSDM), pp. 365–374 (2013)

    Google Scholar 

  54. Puheim, M., Madarasz, L.: Normalization of inputs and outputs of neural network based robotic arm controller in role of inverse kinematic model. In: IEEE International Symposium on Applied Machine Intelligence and Informatics, pp. 35–38 (2014)

    Google Scholar 

  55. Qi, F., Yang, X., Xu, C.: A unified framework for multimodal domain adaptation. In: ACM Multimedia Conference (ACM-MM), pp. 429–437 (2018)

    Google Scholar 

  56. Ranganathan, H., Venkateswara, H., Chakraborty, S., Panchanathan, S.: Deep active learning for image classification. In: IEEE International Conference on Image Processing (ICIP), pp. 3934–3938 (2017)

    Google Scholar 

  57. Ranganathan, H., Venkateswara, H., Chakraborty, S., Panchanathan, S.: Multi-label deep active learning with label correlation. In: IEEE International Conference on Image Processing (ICIP), pp. 3418–3422 (2018)

    Google Scholar 

  58. Rothe, R., Timofte, R., VanGool, L.: Deep expectation of real and apparent age from a single image without facial landmarks. Int. J. Comput. Vis. (IJCV) 126(2–4), 144–157 (2018)

    Article  MathSciNet  Google Scholar 

  59. Rozantsev, A., Salzmann, M., Fua, P.: Residual parameter transfer for deep domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4339–4348 (2018)

    Google Scholar 

  60. Saenko, K., Kulis, B., Fritz, M., Darrell, T.: Adapting visual category models to new domains. In: European Conference on Computer Vision (ECCV), pp. 213–226 (2010)

    Chapter  Google Scholar 

  61. Sankaranarayanan, S., Balaji, Y., Castillo, C., Chellappa, R.: Generate to adapt: aligning domains using generative adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8503–8512 (2018)

    Google Scholar 

  62. Sener, O., Song, H., Saxena, A., Savarese, S.: Learning transferrable representations for unsupervised domain adaptation. In: Advances of Neural Information Processing Systems (NIPS), pp. 2118–2126 (2016)

    Google Scholar 

  63. Shen, J., Qu, Y., Zhang, W., Yu, Y.: Wasserstein distance guided representation learning for domain adaptation. In: Association for the Advancement of Artificial Intelligence (AAAI), pp. 4058–4065 (2018)

    Google Scholar 

  64. Sherrah, J., Gong, S.: Fusion of perceptual cues for robust tracking of head pose and position. Pattern Recogn. 34(8), 1565–1572 (2001)

    Article  Google Scholar 

  65. Smola, A., Kondor, R.: Kernels and regularization on graphs. In: Conference on Computational Learning Theory (COLT), pp. 144–158 (2003)

    Chapter  Google Scholar 

  66. Sriperumbudur, B., Fukumizu, K., Lanckriet, G.: Universality, characteristic kernels and RKHS embedding of measures. J. Mach. Learn. Res. (JMLR) 12, 2389–2410 (2011)

    Google Scholar 

  67. Sriperumbudur, B., Gretton, A., Fukumizu, K., Scholkopf, B., Lanckriet, G.: Hilbert space embeddings and metrics on probability measures. J. Mach. Learn. Res. (JMLR) 11, 1517–1561 (2010)

    Google Scholar 

  68. Steinwart, I.: On the influence of the kernel on the consistency of support vector machines. J. Mach. Learn. Res. (JMLR) 2, 67–93 (2001)

    Google Scholar 

  69. Stevenage, S., Nixon, M., Vince, K.: Visual analysis of gait as a cue to identity. Appl. Cognit. Psychol. 13(6), 513–526 (1999)

    Article  Google Scholar 

  70. Sun, B., Feng, J., Saenko, K.: Return of frustratingly easy domain adaptation. In: Association for the Advancement of Artificial Intelligence (AAAI), pp. 2058–2065 (2016)

    Google Scholar 

  71. Szabo, G., Huberman, B.: Predicting the popularity of online content. Commun. ACM 53(8), 80–88 (2010)

    Article  Google Scholar 

  72. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  73. Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks. In: IEEE International Conference on Computer Vision (ICCV), pp. 4068–4076 (2015)

    Google Scholar 

  74. Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7167–7176 (2017)

    Google Scholar 

  75. Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014)

  76. Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing network for unsupervised domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5018–5027 (2017)

    Google Scholar 

  77. Venkateswara, H., Lade, P., Ye, J., Panchanathan, S.: Coupled support vector machines for supervised domain adaptation. In: ACM Multimedia Conference (ACM-MM), pp. 1295–1298 (2015)

    Google Scholar 

  78. Volpi, R., Morerio, P., Savarese, S., Murino, V.: Adversarial feature augmentation for unsupervised domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5495–5504 (2018)

    Google Scholar 

  79. Winter, D.: The Biomechanics and Motor Control of Human Gait: Normal, Elderly, and Pathological. University of Waterloo Press (1991)

    Google Scholar 

  80. How to improve neural network stability and modeling performance with data scaling. https://machinelearningmastery.com/how-to-improve-neural-network-stabilityand-modeling-performance-with-data-scaling/

  81. Yamada, M., Sigal, L., Chang, Y.: Domain adaptation for structured regression. Int. J. Comput. Vision 109(1–2), 126–145 (2014)

    Article  Google Scholar 

  82. Yan, K., Zheng, W., Cui, Z., Zong, Y., Zhang, T., Tang, C.: Unsupervised facial expression recognition using domain adaptation based dictionary learning approach. Neurocomputing 319, 84–91 (2018)

    Article  Google Scholar 

  83. Yun, Y., Kim, H., Shin, S., Lee, J., Deshpande, A., Kim, C.: Statistical method for prediction of gait kinematics with gaussian process regression. J. Biomech. 47(1), 186–192 (2014)

    Article  Google Scholar 

  84. Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Advances of Neural Information Processing Systems (NIPS), pp. 649–657 (2015)

    Google Scholar 

  85. Zhou, D., Scholkopf, B.: A regularization framework for learning from graph data. In: International Conference on Machine Learning Workshop (2004)

    Google Scholar 

  86. Zhu, R., Sang, G., Zhao, Q.: Discriminative feature adaptation for cross-domain facial expression recognition. In: International Conference on Biometrics (ICB) (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Florida State University, Tallahassee, USA

    Ankita Singh & Shayok Chakraborty

Authors
  1. Ankita Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shayok Chakraborty
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shayok Chakraborty .

Editor information

Editors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB, Canada

    Witold Pedrycz

  2. Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan

    Shyi-Ming Chen

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Singh, A., Chakraborty, S. (2020). Deep Domain Adaptation for Regression. In: Pedrycz, W., Chen, SM. (eds) Development and Analysis of Deep Learning Architectures. Studies in Computational Intelligence, vol 867. Springer, Cham. https://doi.org/10.1007/978-3-030-31764-5_4

Download citation

Publish with us

Policies and ethics

Search

Navigation