Segmentation Propagation in ImageNet

  • Daniel Kuettel
  • Matthieu Guillaumin
  • Vittorio Ferrari
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7578)


ImageNet is a large-scale hierarchical database of object classes. We propose to automatically populate it with pixelwise segmentations, by leveraging existing manual annotations in the form of class labels and bounding-boxes. The key idea is to recursively exploit images segmented so far to guide the segmentation of new images. At each stage this propagation process expands into the images which are easiest to segment at that point in time, e.g. by moving to the semantically most related classes to those segmented so far. The propagation of segmentation occurs both (a) at the image level, by transferring existing segmentations to estimate the probability of a pixel to be foreground, and (b) at the class level, by jointly segmenting images of the same class and by importing the appearance models of classes that are already segmented. Through an experiment on 577 classes and 500k images we show that our technique (i) annotates a wide range of classes with accurate segmentations; (ii) effectively exploits the hierarchical structure of ImageNet; (iii) scales efficiently; (iv) outperforms a baseline GrabCut [1] initialized on the image center, as well as our recent segmentation transfer technique [2] on which this paper is based. Moreover, our method also delivers state-of-the-art results on the recent iCoseg dataset for co-segmentation.


Gaussian Mixture Model Appearance Model Related Classis Foreground Object Segmentation Accuracy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Rother, C., Kolmogorov, V., Blake, A.: Grabcut: Interactive foreground extraction using iterated graph cuts (2004)Google Scholar
  2. 2.
    Kuettel, D., Ferrari, V.: Figure-ground segmentation by transferring window masks (2012)Google Scholar
  3. 3.
    Chai, Y., Lempitsky, V., Zisserman, A.: Bicos: A bi-level co-segmentation method for image classification, pp. 2579–2586 (2011)Google Scholar
  4. 4.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation  59, 167–181 (2004)Google Scholar
  5. 5.
    Tu, Z., Chen, X., Yuille, A., Zhu, S.: Image parsing: Unifying segmentation, detection, and recognition  63, 113–140 (2005)Google Scholar
  6. 6.
    Shotton, J., Blake, A., Cipolla, R.: Contour-Based Learning for Object Detection (2005)Google Scholar
  7. 7.
    Jiang, H.: Human pose estimation using consistent max-covering (2009)Google Scholar
  8. 8.
    Gong, Y., Lazebnik, S.: Iterative quantization: A procrustean approach to learning binary codes (2011)Google Scholar
  9. 9.
    Torralba, A., Fergus, R., Weiss, Y.: Small codes and large image databases for recognition (2008)Google Scholar
  10. 10.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-fei, L.: ImageNet: A large-scale hierarchical image database (2009),
  11. 11.
    Rosenfeld, A., Weinshall, D.: Extracting foreground masks towards object recognition (2011)Google Scholar
  12. 12.
    Batra, D., Kowdle, A., Parikh, D., Luo, J., Chen, T.: Interactively co-segmentating topically related images with intelligent scribble guidance (2011)Google Scholar
  13. 13.
    Tsochantaridis, I., Joachims, T., Hofmann, T., Altun, Y.: Large margin methods for structured and interdependent output variables.  6, 1453–1484 (2005)Google Scholar
  14. 14.
    Szummer, M., Kohli, P., Hoiem, D.: Learning CRFs using graph cuts (2008)Google Scholar
  15. 15.
    Vicente, S., Rother, C., Kolmogorov, V.: Object cosegmentation, pp. 2217–2224 (2011)Google Scholar
  16. 16.
    Joulin, A., Bach, F., Ponce, J.: Discriminative clustering for image co-segmentation, pp. 1943–1950 (2010)Google Scholar
  17. 17.
    Borenstein, E., Sharon, E., Ullman, S.: Combining top-down and bottom-up segmentation (2004)Google Scholar
  18. 18.
    Jojic, N., Perina, A., Cristani, M., Murino, V., Frey, B.: Stel component analysis: Modeling spatial correlations in image class structure (2009)Google Scholar
  19. 19.
    Bertelli, L., Yu, T., Vu, D., Gokturk, S.: Kernelized structural SVM learning for supervised object segmentation (2011)Google Scholar
  20. 20.
    Winn, J., Jojic, N.: LOCUS: learning object classes with unsupervised segmentation (2005)Google Scholar
  21. 21.
    Arora, H., Loeff, N., Forsyth, D., Ahuja, N.: Unsupervised segmentation of objects using efficient learning (2007)Google Scholar
  22. 22.
    Verbeek, J., Triggs, B.: Region classification with Markov field aspect models. In: CVPR (2007)Google Scholar
  23. 23.
    Alexe, B., Deselaers, T., Ferrari, V.: What is an object? (2010)Google Scholar
  24. 24.
    Carreira, J., Sminchisescu, C.: Constrained parametric min cuts for automatic object segmentation (2010)Google Scholar
  25. 25.
    Schoenemann, T., Cremers, D.: Introducing curvature into globally optimal image segmentation: Minimum ratio cycles on product graphs (2007)Google Scholar
  26. 26.
    Blake, A., Rother, C., Brown, M., Perez, P., Torr, P.: Interactive image segmentation using an adaptive GMMRF model (2004)Google Scholar
  27. 27.
    Kim, G., Xing, E., Fei-Fei, L., Kanade, T.: Distributed cosegmentation via submodular optimization on anisotropic diffusion, pp. 169–176 (2011)Google Scholar
  28. 28.
    Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In: CVPR Workshop of Generative Model Based Vision (2004)Google Scholar
  29. 29.
    Lampert, C., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer (2009)Google Scholar
  30. 30.
    Ott, P., Everingham, M.: Shared parts for deformable part-based models (2011)Google Scholar
  31. 31.
    Norouzi, M., Punjani, A., Fleet, D.J.: Fast search in hamming space with multi-index hashing (2012)Google Scholar
  32. 32.
    Vicente, S., Kolmogorov, V., Rother, C.: Graph cut based image segmentation with connectivity priors (2008)Google Scholar
  33. 33.
    Shotton, J., Winn, J., Rother, C., Criminisi, A.: TextonBoost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation (2006)Google Scholar
  34. 34.
    Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images (2001)Google Scholar
  35. 35.
    Rother, C., Kolmogorov, V., Minka, T., Blake, A.: Cosegmentation of image pairs by histogram matching -incorporating a global constraint into MRFs (2006)Google Scholar
  36. 36.
    Batra, D., Kowdle, A., Parikh, D., Luo, J., Chen, T.: iCoseg: Interactive co-segmentation with intelligent scribble guidance, pp. 3169–3176 (2010)Google Scholar
  37. 37.
    Wang, J., Cohen, M.: An iterative optimization approach for unified image segmentation and matting (2005)Google Scholar
  38. 38.
    Tommasi, T., Orabona, F., Caputo, B.: Safety in numbers: Learning categories from few examples with multi model knowledge transfer (2010)Google Scholar
  39. 39.
    Guillaumin, M., Ferrari, V.: Large-scale knowledge transfer for object localization in ImageNet (2012)Google Scholar
  40. 40.
    Salakhutdinov, R., Torralba, A., Tenenbaum, J.: Learning to share visual appearance for multiclass object detection (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Daniel Kuettel
    • 1
    • 2
  • Matthieu Guillaumin
    • 2
  • Vittorio Ferrari
    • 1
  1. 1.University of EdinburghUK
  2. 2.ETH ZürichSwitzerland

Personalised recommendations