Weakly-Supervised Semantic Segmentation by Redistributing Region Scores Back to the Pixels

  • Josip KrapacEmail author
  • Siniša Šegvić
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9796)


We address the problem of semantic segmentation of objects in weakly supervised setting, when only image-wide labels are available. We describe an image with a set of pre-trained convolutional features and embed this set into a Fisher vector. We apply the learned image classifier on the set of all image regions and propagate the region scores back to the pixels. Compared to the alternatives the proposed method is simple, fast in inference, and especially in training. The method displays very good performance of on two standard semantic segmentation benchmarks.


Convolutional Neural Network Conditional Random Field Segmentation Performance Fisher Vector Convolutional Layer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work has been fully supported by Croatian Science Foundation under the project I-2433-2014.


  1. 1.
    Arbeláez, P., Pont-Tuset, J., Barron, J., Marques, F., Malik, J.: Multiscale combinatorial grouping. In: CVPR (2014)Google Scholar
  2. 2.
    Boix, X., Gonfaus, J.M., van de Weijer, J., Bagdanov, A.D., Gual, J.S., Gonzàlez, J.: Harmony potentials - fusing global and local scale for semantic image segmentation. IJCV (2012)Google Scholar
  3. 3.
    Cimpoi, M., Maji, S., Kokkinos, I., Vedaldi, A.: Deep filter banks for texture recognition, description, and segmentation. IJCV 1–30 (2016)Google Scholar
  4. 4.
    Cinbis, R.G., Verbeek, J., Schmid, C.: Weakly supervised object localization with multi-fold multiple instance learning. In: PAMI (2016)Google Scholar
  5. 5.
    Csurka, G., Perronnin, F.: An efficient approach to semantic segmentation. IJCV (2011)Google Scholar
  6. 6.
    Dai, J., He, K., Sun, J.: BoxSup: exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In: ICCV (2015)Google Scholar
  7. 7.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009)Google Scholar
  8. 8.
    Everingham, M., Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. IJCV (2010)Google Scholar
  9. 9.
    Hong, S., Noh, H., Han, B.: Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation. CoRR abs/1506.04924 (2015)Google Scholar
  10. 10.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  11. 11.
    Lampert, C.H., Blaschko, M.B., Hofmann, T.: Efficient subwindow search: a branch and bound framework for object localization. In: PAMI (2009)Google Scholar
  12. 12.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)Google Scholar
  13. 13.
    Papandreou, G., Chen, L., Murphy, K.P., Yuille, A.L.: Weakly- and semi-supervised learning of a deep convolutional network for semantic image segmentation. In: ICCV (2015)Google Scholar
  14. 14.
    Pathak, D., Krähenbühl, P., Darrell, T.: Constrained convolutional neural networks for weakly supervised segmentation. In: ICCV (2015)Google Scholar
  15. 15.
    Pathak, D., Shelhamer, E., Long, J., Darrell, T.: Fully convolutional multi-class multiple instance learning. In: ICLR Workshop (2015)Google Scholar
  16. 16.
    Perronnin, F., Sánchez, J., Mensink, T.: Improving the Fisher Kernel for large-scale image classification. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part IV. LNCS, vol. 6314, pp. 143–156. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  17. 17.
    Pinheiro, P.O., Collobert, R.: From image-level to pixel-level labeling with convolutional networks. In: CVPR (2015)Google Scholar
  18. 18.
    Ramachandran, S., Mooney, R.J.: Revising Bayesian network parameters using backpropagation. In: ICNN (1996)Google Scholar
  19. 19.
    Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS (2015)Google Scholar
  20. 20.
    Sánchez, J., Perronnin, F., Mensink, T., Verbeek, J.J.: Image classification with the fisher vector: theory and practice. IJCV (2013)Google Scholar
  21. 21.
    Shotton, J., Winn, J.M., Rother, C., Criminisi, A.: Textonboost for image understanding: multi-class object recognition and segmentation by jointly modeling texture, layout, and context. IJCV (2009)Google Scholar
  22. 22.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)Google Scholar
  23. 23.
    Uijlings, J.R.R., van de Sande, K.E.A., Gevers, T., Smeulders, A.W.M.: Selective search for object recognition. IJCV (2013)Google Scholar
  24. 24.
    Xu, J., Schwing, A.G., Urtasun, R.: Learning to segment under various forms of weak supervision. In: CVPR (2015)Google Scholar
  25. 25.
    Zadrija, V., Krapac, J., Verbeek, J.J., Segvic, S.: Patch-level spatial layout for classification and weakly supervised localization. In: GCPR (2015)Google Scholar
  26. 26.
    Zhang, W., Zeng, S., Wang, D., Xue, X.: Weakly supervised semantic segmentation for social images. In: CVPR (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Faculty of Electrical Engineering and ComputingUniversity of ZagrebZagrebCroatia

Personalised recommendations