Learning contextual superpixel similarity for consistent image segmentation

  • Mahaman Sani ChaibouEmail author
  • Pierre-Henri Conze
  • Karim Kalti
  • Mohamed Ali Mahjoub
  • Basel Solaiman


This paper addresses the problem of image segmentation by iterative region aggregations starting from an initial superpixel decomposition. Classical approaches for this task compute superpixel similarity using distance measures between superpixel descriptor vectors. This usually poses the well-known problem of the semantic gap and fails to properly aggregate visually non-homogeneous superpixels that belong to the same high-level object. This work proposes to use random forests to learn the merging probability between adjacent superpixels in order to overcome the aforementioned issues. Compared to existing works, this approach learns the fusion rules without explicit similarity measure computation. We also introduce a new superpixel context descriptor to strengthen the learned characteristics towards better similarity prediction. Image segmentation is then achieved by iteratively merging the most similar superpixel pairs selected using a similarity weighting objective function. Experimental results of our approach on four datasets including DAVIS 2017 and ISIC 2018 show its potential compared to state-of-the-art approaches.


Context description Superpixels similarity Machine learning Random forests Image segmentation Region-growing 



  1. 1.
    Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S (2012) Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(11):2274–2282CrossRefGoogle Scholar
  2. 2.
    Amit Y, Geman D (1997) Shape quantization and recognition with randomized trees. Neural Computation 9(7):1545–1588CrossRefGoogle Scholar
  3. 3.
    Audebert N, Boulch A, Randrianarivo H, Le Saux B, Ferecatu M, Lefèvre S., Marlet R (2017) Deep learning for urban remote sensing. In: Urban Remote Sensing Event (JURSE), 2017 Joint, IEEE, pp 1–4Google Scholar
  4. 4.
    Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13(Feb):281–305MathSciNetzbMATHGoogle Scholar
  5. 5.
    Bhatti AH, Rahman AU, Butt AA (2019) Unsupervised video object segmentation using conditional random fields. Image and Video Processing 13(1):9–16CrossRefGoogle Scholar
  6. 6.
    Bosch A, Zisserman A, Munoz X (2007) Image classification using random forests and ferns. In: 2007. ICCV 2007. IEEE 11th International Conference on Computer Vision, IEEE, pp 1–8Google Scholar
  7. 7.
    Brahim K, Kalboussi R, Abdellaoui M, Douik A (2019) Spatio-temporal saliency detection using objectness measure. Signal, Image and Video Processing, pp 1–8CrossRefGoogle Scholar
  8. 8.
    Breiman L (2001) Random forests. Mach learn 45(1):5–32CrossRefGoogle Scholar
  9. 9.
    Breiman L (2002) Manual on setting up, using, and understanding random forests v3. 1. Statistics Department University of California Berkeley, CA, USAGoogle Scholar
  10. 10.
    Chaibou MS, Conze P-H, Kalti K, Solaiman B, Mahjoub MA (2017) Adaptive strategy for superpixel-based region-growing image segmentation. J Electron Imaging 26(6):061605CrossRefGoogle Scholar
  11. 11.
    Chen C, Li S, Qin H, Pan Z, Yang G (2018) Bilevel feature learning for video saliency detection. IEEE Trans Multimedia 20(12):3324–3336CrossRefGoogle Scholar
  12. 12.
    Chen C, Li S, Wang Y, Qin H, Hao A (2017) Video saliency detection via spatial-temporal fusion and low-rank coherency diffusion. IEEE Trans Image Process 26(7):3156–3170MathSciNetCrossRefGoogle Scholar
  13. 13.
    Conze P-H, Noblet V, Rousseau F, Heitz F, De Blasi V, Memeo R, Pessaux P (2017) Scale-adaptive supervoxel-based random forests for liver tumor segmentation in dynamic contrast-enhanced ct scans. International Journal of Computer Assisted Radiology and Surgery 12(2):223–233CrossRefGoogle Scholar
  14. 14.
    Duffner S, Garcia C (2013) Pixeltrack: a fast adaptive algorithm for tracking non-rigid objects. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2480–2487Google Scholar
  15. 15.
    Freixenet J, Muñoz X, Raba D, Martí J, Cufí X (2002) Yet another survey on image segmentation: Region and boundary information integration. In: European Conference on Computer Vision, Springer, pp 408–422CrossRefGoogle Scholar
  16. 16.
    Fukuchi K, Miyazato K, Kimura A, Takagi S, Yamato J (2009) Saliency-based video segmentation with graph cuts and sequentially updated priors. In: 2009 IEEE International Conference on Multimedia and Expo, IEEE, pp 638–641Google Scholar
  17. 17.
    Godec M, Roth PM, Bischof H (2013) Hough-based tracking of non-rigid objects. Comput Vis Image Underst 117(10):1245–1256CrossRefGoogle Scholar
  18. 18.
    Granitto PM, Furlanello C, Biasioli F, Gasperi F (2006) Recursive feature elimination with random forest for ptr-ms analysis of agroindustrial products. Chemometr Intell Lab Syst 83(2):83–90CrossRefGoogle Scholar
  19. 19.
    Guyon I, Weston J, Barnhill S, Vapnik V (2002) Gene selection for cancer classification using support vector machines. Mach learn 46(1-3):389–422CrossRefGoogle Scholar
  20. 20.
    Haller E, Leordeanu M (2017) Unsupervised object segmentation in video by efficient selection of highly probable positive features. In: Proceedings of the IEEE International Conference on Computer Vision, pp 5085–5093Google Scholar
  21. 21.
    Hamming R (1950) The bell system technical journal. Bell Syst Tech J 26 (2):147–160CrossRefGoogle Scholar
  22. 22.
    Haralick RM, Shapiro LG (1985) Image segmentation techniques. Computer Vision, Graphics, and Image Processing 29(1):100–132CrossRefGoogle Scholar
  23. 23.
    Hsu C-Y, Ding J-J (2013) Efficient image segmentation algorithm using slic superpixels and boundary-focused region merging. In: Communications and Signal Processing (ICICS) 2013 9th international conference on Information, IEEE, pp 1–5Google Scholar
  24. 24.
    Lepetit V, Fua P (2006) Keypoint recognition using randomized trees. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(9):1465–1479CrossRefGoogle Scholar
  25. 25.
    Li F, Kim T, Humayun A, Tsai D, Rehg JM (2013) Video segmentation by tracking many figure-ground segments. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2192– 2199Google Scholar
  26. 26.
    Louppe G, Wehenkel L, Sutera A, Geurts P (2013) Understanding variable importances in forests of randomized trees. In: Advances in Neural Information Processing Systems, pp 431–439Google Scholar
  27. 27.
    Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proc. 8th Int’l Conf Computer Vision 2:416–423CrossRefGoogle Scholar
  28. 28.
    Meilă M (2007) Comparing clusterings — an information based distance. J Multivar Anal 98(5):873–895MathSciNetCrossRefGoogle Scholar
  29. 29.
    Oneata D, Revaud J, Verbeek J, Schmid C (2014) Spatio-temporal object detection proposals. In: European Conference on Computer Vision, Springer, pp 737–752CrossRefGoogle Scholar
  30. 30.
    Ozuysal M, Fua P, Lepetit V (2007) Fast keypoint recognition in ten lines of code. In: 2007. CVPR’07. IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 1–8Google Scholar
  31. 31.
    Pauly O (2012) Random forests for medical applications. PhD thesis, Technische Universität MünchenGoogle Scholar
  32. 32.
    Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830MathSciNetzbMATHGoogle Scholar
  33. 33.
    Pont-Tuset J, Perazzi F, Caelles S, Arbeláez P, Sorkine-Hornung A, Van Gool L. (2017) The 2017 davis challenge on video object segmentation. arXiv:1704.00675
  34. 34.
    Ren X, Malik J (2003) Learning a classification model for segmentation. In: ICCV, vol 1, pp 10–17Google Scholar
  35. 35.
    Sangsefidi N, Foruzan AH, Dolati A (2017) Balancing the data term of graph-cuts algorithm to improve segmentation of hepatic vascular structures. Computers in Biology and MedicineGoogle Scholar
  36. 36.
    Santana TM, Machado AM, Araújo AdA, dos Santos JA (2016) Star: a contextual description of superpixels for remote sensing image classification. In: Iberoamerican Congress on Pattern Recognition, Springer, pp 300–308Google Scholar
  37. 37.
    Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Trans Pattern Anal Machine Intelligence 22(8):888–905CrossRefGoogle Scholar
  38. 38.
    Silva RE (2017) An alternative approach to counting minimum (s; t)-cuts in planar graphsGoogle Scholar
  39. 39.
    Son J, Jung I, Park K, Han B (2015) Tracking-by-segmentation with online gradient boosting decision tree. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3056–3064Google Scholar
  40. 40.
    Stutz D, Hermans A, Leibe B (2016) Superpixels: an evaluation of the state-of-the-art. CoRR, arXiv:
  41. 41.
    Tilquin F, Conze P-H, Pessaux P, Lamard M, Quellec G, Noblet V, Heitz F (2018) Robust supervoxel matching combining mid-level spectral and context-rich features. In: International Workshop on Patch-based Techniques in Medical Imaging, Springer, pp 39–47Google Scholar
  42. 42.
    Unnikrishnan R, Pantofaru C, Hebert M (2007) Toward objective evaluation of image segmentation algorithms. IEEE transactions on pattern analysis and machine intelligence, 29(6)CrossRefGoogle Scholar
  43. 43.
    Vargas JE, Falcão AX, Dos Santos J, Esquerdo JCDM, Coutinho AC, Antunes J (2015) Contextual superpixel description for remote sensing image classification. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS), IEEE 2015, pp 1132–1135Google Scholar
  44. 44.
    Vasconcelos MJM, Tavares JMR (2015) Human motion segmentation using active shape models. In: Computational and Experimental Biomedical Sciences: Methods and Applications, Springer, pp 237–246Google Scholar
  45. 45.
    Wang S, Lu H, Yang F, Yang M-H (2011) Superpixel tracking. In: 2011 International Conference on Computer Vision, IEEE, pp 1323–1330Google Scholar
  46. 46.
    Yang Y, Wang Y, Xue X (2016) A novel spectral clustering method with superpixels for image segmentation. Optik-International Journal for Light and Electron Optics 127(1):161–167CrossRefGoogle Scholar
  47. 47.
    Yeo D, Son J, Han B, Hee Han J (2017) Superpixel-based tracking-by-segmentation using markov chains. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1812–1821Google Scholar
  48. 48.
    Yin P, Criminisi A, Winn J, Essa I (2007) Tree-based classifiers for bilayer video segmentation. In: 2007. CVPR’07. IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 1–8Google Scholar
  49. 49.
    Yin S, Qian Y, Gong M (2017) Unsupervised hierarchical image segmentation through fuzzy entropy maximization. Pattern RecognitionGoogle Scholar
  50. 50.
    Yu H, Zhang X, Wang S, Hou B (2013) Context-based hierarchical unequal merging for sar image segmentation. IEEE Transactions on Geoscience and Remote Sensing 51(2):995–1009CrossRefGoogle Scholar
  51. 51.
    Zhang D, Javed O, Shah M (2013) Video object segmentation through spatially accurate and temporally dense extraction of primary object regions (open access). Technical report University of Central Florida Orlando United StatesGoogle Scholar
  52. 52.
    Zhang Y, He K (2017) Multi-scale gaussian segmentation via graph cuts. DEStech Transactions on Computer Science and Engineering (csae)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Université de Sousse, Ecole Nationale d’Ingénieurs de SousseLATIS - Laboratory of Advanced Technology and Intelligent SystemsSousseTunisie
  2. 2.Institut Supérieur d’Informatique et des Techniques de CommunicationUniversité de SousseHammam SousseTunisie
  3. 3.IMT Atlantique, Technopôle Brest-IroiseBrest Cedex 03France
  4. 4.LaTIM UMR 1101, Inserm, IBRBSBrestFrance
  5. 5.Faculté des sciences de MonatirUniversité de MonastirMonastirTunisie

Personalised recommendations