Semantic Segmentation via Multi-task, Multi-domain Learning

  • Damien Fourure
  • Rémi Emonet
  • Elisa FromontEmail author
  • Damien Muselet
  • Alain Trémeau
  • Christian Wolf
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10029)


We present an approach that leverages multiple datasets possibly annotated using different classes to improve the semantic segmentation accuracy on each individual dataset. We propose a new selective loss function that can be integrated into deep networks to exploit training data coming from multiple datasets with possibly different tasks (e.g., different label-sets). We show how the gradient-reversal approach for domain adaptation can be used in this setup. Thorought experiments on semantic segmentation applications show the relevance of our approach.


Deep learning Convolutional neural networks Semantic segmentation Domain adaptation Multi-task learning 



Authors acknowledge the support from the ANR project SoLStiCe (ANR-13-BS02-0002-01). They also want to thank Nvidia for providing two Titan X GPU.


  1. 1.
    Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J.W.: A theory of learning from different domains. Mach. Learn. 79(1–2), 151–175 (2010)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene labeling. IEEE TPAMI 35(8), 1915–1929 (2013)CrossRefGoogle Scholar
  3. 3.
    Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: ICML (2015)Google Scholar
  4. 4.
    Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Robot. Res. 34, 727–743 (2013)Google Scholar
  5. 5.
    Germain, P., Habrard, A., Laviolette, F., Morvant, E.: A new PAC-Bayesian perspective on domain adaptation. In: Proceedings of the 33rd International Conference on Machine Learning (ICML 2016), New York, NY, United States (2016).
  6. 6.
    Gould, S., Fulton, R., Koller, D.: Decomposing a scene into geometric and semantically consistent regions. In: ICCV (2009)Google Scholar
  7. 7.
    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS Deep Learning Workshop (2014)Google Scholar
  8. 8.
    Hinton, G., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Improving neural networks by preventing co-adaptation of feature detectors arXiv:1207.0580 (2012)
  9. 9.
    Hu, H., Ben, U.: Nonparametric semantic segmentation for 3D street scenes. In: Intelligent Robots and Systems (IROS) (2013)Google Scholar
  10. 10.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML (2015)Google Scholar
  11. 11.
    Kekec, T., Emonet, R., Fromont, E., Trémeau, A., Wolf, C.: Contextually constrained deep networks for scene labeling. In: BMVC (2014)Google Scholar
  12. 12.
    Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  13. 13.
    Kundu, A., Li, Y., Dellaert, F., Li, F., Rehg, J.M.: Joint semantic segmentation and 3D reconstruction from monocular video. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 703–718. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10599-4_45 Google Scholar
  14. 14.
    Ladicky, L., Shi, J., Pollefeys, M.: Pulling things out of perspective. In: CVPR (2014)Google Scholar
  15. 15.
    Liu, C., Yuen, J., Torralba, A.: Nonparametric scene parsing via label transfer. IEEE TPAMI 33(5), 978–994 (2011)CrossRefGoogle Scholar
  16. 16.
    Mansour, Y., Mohri, M., Rostamizadeh, A.: Domain adaptation: learning bounds and algorithms. In: COLT 2009 - The 22nd Conference on Learning Theory, Montreal, Quebec, Canada, 18–21 June 2009Google Scholar
  17. 17.
    Pinheiro, P., Collobert, R.: Recurrent convolutional neural networks for scene labeling. In: ICML (2014)Google Scholar
  18. 18.
    Ros, G., Ramos, S., Granados, M., Bakhtiary, A., Vazquez, D., Lopez, A.M.: Vision-based offline-online perception paradigm for autonomous driving. In: Winter Conference on Applications of Computer Vision (WACV) (2015)Google Scholar
  19. 19.
    Sengupta, S., Greveson, E., Shahrokni, A., Torr, P.H.: Urban 3D semantic modelling using stereo vision. In: ICRA (2013)Google Scholar
  20. 20.
    Sharma, A., Tuzel, O., Jacobs, D.W.: Deep hierarchical parsing for semantic segmentation. In: CVPR (2015)Google Scholar
  21. 21.
    Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks. In: ICCV (2015)Google Scholar
  22. 22.
    Xu, P., Davoine, F., Bordes, J.B., Zhao, H., Denoeux, T.: Information fusion on oversegmented images: an application for urban scene understanding. In: IAPR MVA (2013)Google Scholar
  23. 23.
    Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: NIPS (2014)Google Scholar
  24. 24.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10590-1_53 Google Scholar
  25. 25.
    Zhang, R., Candra, S.A., Vetter, K., Zakhor, A.: Sensor fusion for semantic segmentation of urban scenes. In: IEEE ICRA (2015)Google Scholar
  26. 26.
    Zhang, X., Yu, F.X., Chang, S., Wang, S.: Deep transfer network: unsupervised domain adaptation. arXiv (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Damien Fourure
    • 1
  • Rémi Emonet
    • 1
  • Elisa Fromont
    • 1
    Email author
  • Damien Muselet
    • 1
  • Alain Trémeau
    • 1
  • Christian Wolf
    • 2
  1. 1.Universite de Lyon, UJM, CNRS, Lab Hubert Curien UMR5516LyonFrance
  2. 2.Universite de Lyon, CNRS, INSA-Lyon, LIRIS, UMR5205LyonFrance

Personalised recommendations