Salient object segmentation based on depth-aware image layering

  • Huan Du
  • Zhi LiuEmail author
  • Ran Shi


This paper proposes an efficient salient object segmentation method via depth-aware image layering. First, based on the multiscale region segmentation results of an input color image, the depth consistency integration is utilized to generate the image pre-segmentation result. Then, under the guidance of the depth histogram division, the pre-segmented regions are divided into several different layers to differentiate salient object regions and background regions. Finally, an adaptive sample update and selection method based on layered image regions is used to select appropriate training samples for salient object segmentation. The depth information of the image is fully utilized in each step of the entire framework. Experimental results on two public datasets demonstrate that the proposed method achieves the better performance than the state-of-the-art depth-aware salient object segmentation methods.


Depth consistency integration Depth distribution Image layering Depth histogram Adaptive sample update and selection Salient object segmentation 



This work was supported by the National Natural Science Foundation of China under Grants 61771301 and 61801219, and by Shanghai Science and Technology Commission Project under Grant No. 17595800900.


  1. 1.
    Arbelaez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33(5):898–916CrossRefGoogle Scholar
  2. 2.
    Boykov Y, Funka-Lea G (2006) Graph cuts and efficient N-D image segmentation. Int J Comput Vis 70(2):109–131CrossRefGoogle Scholar
  3. 3.
    Boykov Y, Kolmogorov V (2004) An experimental comparison of mincut/max-flow algorithms for energy minimization in vision. IEEE Trans Pattern Anal Machine Intell 26(9):1124–1137CrossRefGoogle Scholar
  4. 4.
    Boykov Y, Veksler O, Zabih R (2001) Fast approximate energy minimization via graph cuts. IEEE Trans Pattern Anal Mach Intell 23(11):1222–1239CrossRefGoogle Scholar
  5. 5.
    Cheng M, Zhang G, Mitra N, Huang X, Hu S (2011) Global contrast based salient region detection. Proc IEEE CVPR: 409–416Google Scholar
  6. 6.
    Cheng M, Mitra N, Huang X, Hu S (Apr. 2014) SalientShape: group saliency in image collections. Vis Comput 30(4):443–453CrossRefGoogle Scholar
  7. 7.
    Du H, Liu Z, Jiang J, Shen L (2013) Stretchability-aware block scaling for image retargeting. J Vis Commun Image Represent 24(4):499–508CrossRefGoogle Scholar
  8. 8.
    Du H, Liu Z, Song H, Mei L, Xu Z (2016) Improving RGBD saliency detection using progressive region classification and saliency fusion. IEEE Access 4:8987–8994CrossRefGoogle Scholar
  9. 9.
    Fan X, Liu Z, Sun G (2014) Salient region detection for stereoscopic images. Proc IEEE DSP: 454–458Google Scholar
  10. 10.
    Fan X, Liu Z, Ye L (2015) Salient object segmentation from stereoscopic images. Proc. Graph-Based Represent Pattern Recogn: 272–281Google Scholar
  11. 11.
    Fu H, Xu D, Lin S (2017) Object-based multiple foreground segmentation in RGBD video. IEEE Trans Image Process 26(3):1418–1427MathSciNetCrossRefGoogle Scholar
  12. 12.
    Han J, Ngan K, Li M, Zhang H (2006) Unsupervised extraction of visual attention objects in color images. IEEE Trans Circuits Syst Video Technol 16(1):141–145CrossRefGoogle Scholar
  13. 13.
    Hu S, Chen T, Xu K, Cheng M, Martin R (2013) Internet visual media processing: a survey with graphics and vision applications. Vis Comput 29(5):393–405CrossRefGoogle Scholar
  14. 14.
    Ju R, Liu Y, Ren T, Ge L, Wu G (2015) Depth-aware salient object detection using anisotropic center-surround difference. Signal Process Image Commun 38(10):115–126CrossRefGoogle Scholar
  15. 15.
    Ko B, Nam J (2006) Object-of-interest image segmentation based on human attention and semantic region clustering. J Opt Soc Am A 23(10):2462–2470CrossRefGoogle Scholar
  16. 16.
    Lei J, Zhang H, You L, Hou C, Wang L (2013) Evaluation and modeling of depth feature incorporated visual attention for salient object segmentation. Neurocomputing 120(9):24–33CrossRefGoogle Scholar
  17. 17.
    Lei J, Zhang C, Fang Y, Gu Z, Ling N, Hou C (2015) Depth sensation enhancement for multiple virtual view rendering. IEEE Trans. Multimed 17(4):457–469CrossRefGoogle Scholar
  18. 18.
    Lin S, Yeh I, Lin C, Lee T (2013) Patch-based image warping for content aware retargeting. IEEE Trans Multimed 15(2):359–368CrossRefGoogle Scholar
  19. 19.
    Liu Z, Shi R, Shen L, Xue Y, Ngan K, Zhang Z (2012) Unsupervised salient object segmentation based on kernel density estimation and two-phase graph cut. IEEE Trans Multimed 14(4):1275–1289CrossRefGoogle Scholar
  20. 20.
    Liu H, Philipose M, Sun M (2014) Automatic objects segmentation with RGB-D cameras. J Vis Commun Image Represent 25(4):709–718CrossRefGoogle Scholar
  21. 21.
    Mishra A, Shrivastava A, Aloimonos Y (2012) Segmenting “simple” objects using RGBD. Proc IEEE ICRA: 4406–4413Google Scholar
  22. 22.
    Niu Y, Geng Y, Li X, Liu F (2012) Leveraging stereopsis for saliency analysis. Proc IEEE CVPR: 454–461Google Scholar
  23. 23.
    Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66CrossRefGoogle Scholar
  24. 24.
    Peng H, Li B, Xiong W, Hu W, Ji R (2014) RGBD salient object detection: a benchmark and algorithms. Proc ECCV: 92–109Google Scholar
  25. 25.
    Rahtu E, Kannala J, Salo M, Heikkilä J (2010) Segmenting salient objects from images and videos. Proc Eur Conf Comput Vis (ECCV): 366–379Google Scholar
  26. 26.
    Rother C, Kolmogorov V, Blake A (2004) Grabcut - interactive foreground extraction using iterated graph cuts. ACM Trans Graph 23(3):309–314CrossRefGoogle Scholar
  27. 27.
    Rutishauser U, Walther D, Koch C, Perona P (2004) Is bottom-up attention useful for object recognition? Proc IEEE Comput Soc Conf Comput Vis Pattern Recog 2:37–44Google Scholar
  28. 28.
    Setlur V, Lechner T, Nienhaus M, Gooch B (2007) Retargeting images and video for preserving information saliency. IEEE Comput Graph Appl 27(5):80–88CrossRefGoogle Scholar
  29. 29.
    Shen L, Liu Z, Zhang Z (Apr. 2013) A novel H.264 rate control algorithm with consideration of visual attention. Multimed Tools Appl 63(3):709–727CrossRefGoogle Scholar
  30. 30.
    Song H, Liu Z, Du H, Sun G, Le Meur O, Ren T (2017) Depth-aware salient object detection and segmentation via multiscale discriminative saliency fusion and bootstrap learning. IEEE Trans Image Process 26(9):4204–4216MathSciNetCrossRefGoogle Scholar
  31. 31.
    Sun D, Roth S, Black M (2010) Secrets of optical flow estimation and their principles. Proc IEEE CVPR: 2432–2439Google Scholar
  32. 32.
    Tang M, Gorelick L, Veksler L, Boykov Y (2013) GrabCut in one cut. Proc IEEE ICCV: 1769–1776Google Scholar
  33. 33.
    Tong N, Lu H, Ruan X, Yang M (2015) Salient object detection via bootstrap learning. Proc IEEE CVPR: 1884–1892Google Scholar
  34. 34.
    Xue J, Li C, Zheng N (2011) Proto-object based rate control for JPEG2000: an approach to content-based scalability. IEEE Trans Image Process 20(4):1177–1184MathSciNetCrossRefGoogle Scholar
  35. 35.
    Yang Y, Yang L, Wu G, Li S (2014) Image relevance prediction using query-context bag-of-object retrieval model. IEEE Trans. Multimedia 16(6):1700–1712CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Shanghai Institute for Advanced Communication and Data ScienceShanghai UniversityShanghaiChina
  2. 2.School of Communication and Information EngineeringShanghai UniversityShanghaiChina
  3. 3.Technology Research and Development Center for the Internet of ThingsThe Third Research Institute of the Ministry of Public SecurityShanghaiChina
  4. 4.School of Computer Science and EngineeringNanjing University of Science and TechnologyNanjingChina

Personalised recommendations