Skip to main content
Log in

Adaptive saliency fusion based on quality assessment

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A variety of saliency models based on different schemes and methods have been proposed in the recent years, and the performance of these models often vary with images and complement each other. Therefore it is a natural idea whether we can elevate saliency detection performance by fusing different saliency models. This paper proposes a novel and general framework to adaptively fuse saliency maps generated using various saliency models based on quality assessment of these saliency maps. Given an input image and its multiple saliency maps, the quality features based on the input image and saliency maps are extracted. Then, a quality assessment model, which is learned using the boosting algorithm with multiple kernels, indicates the quality score of each saliency map. Next, a linear summation method with power-law transformation is exploited to fuse these saliency maps adaptively according to their quality scores. Finally, a graph cut based refinement method is exploited to enhance the spatial coherence of saliency and generate the high-quality final saliency map. Experimental results on three public benchmark datasets with state-of-the-art saliency models demonstrate that our saliency fusion framework consistently outperforms all saliency models and other fusion methods, and effectively elevates saliency detection performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Achanta R, Hemami SS, Estrada FJ, Süsstrunk S (2009) Frequency tuned salient region detection. Proc. of IEEE Conference on Computer Vision Pattern Recognition, In, pp. 1597–1604

    Google Scholar 

  2. Aytekin C, Kiranyaz S, Gabbouj M (2014) Automatic object segmentation by quantum cuts. Proc. of IEEE International Conference on Pattern Recognition, In, pp. 112–117

    Google Scholar 

  3. Borji A, Sihite D.N., Itti L (2012) Salient object detection: a benchmark. In: Proc. of European Conference on Computer Vision, pp. 414–429

  4. Borji A, Cheng MM, Jiang H, Li J (2015) Salient object detection: a benchmark. IEEE Trans Image Process 24(12):5706–5722

    Article  MathSciNet  Google Scholar 

  5. Boykov Y, and Jolly MP (2001) Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images. In: Proc. of IEEE International Conference on Computer Vision, pp. 105–112

  6. Boykov Y, Kolmogorov V (2004) An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans Pattern Anal Mach Intell 26(9):1124–1137

    Article  MATH  Google Scholar 

  7. Cao Z, Qin T, Liu T.Y., Tsai M.F., Li H (2007) Learning to rank: from pairwise approach to listwise approach. In: Proc. of International Conference on Machine Learning, pp.129–136

  8. Cheng MM, Zhang GX, Mitra NJ, Huang X, Hu SM (2011) Global contrast based salient region detection. Proc. of IEEE Conference on Computer Vision Pattern Recognition, In, pp. 409–416

    Google Scholar 

  9. Dollár P, Zitnick C (2013) Structured forests for fast edge detection. In: Proc. of IEEE International Conference on Computer Vision, pp. 1841-1848

  10. Du H, Liu Z, Jiang J, Shen L (2013) Stretchability-aware block scaling for image retargeting. J Vis Commun Image Represent 24(4):499–508

    Article  Google Scholar 

  11. Fang Y, Chen Z, Lin W, and Lin C.-W. (2012) Saliency detection in the compressed domain for adaptive image retargeting. IEEE Trans Image Process, 21(9): 3888–3901

  12. Fang Y, Lin W, Lee B.-S., Lau CT, Chen Z, and Lin C.-W. (2012) Bottom-up saliency detection model based on human visual sensitivity and amplitude spectrum. IEEE Trans Multimed, 12(1): 187–198

  13. Goferman S, Zelnik-Manor L, Tal A (2010) Context-aware saliency detection. Proc. of IEEE Conference on Computer Vision Pattern Recognition, In, pp. 2376–2383

    Google Scholar 

  14. Guo C, Zhang L (2010) A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans Image Process 19(1):85–198

    Article  MATH  MathSciNet  Google Scholar 

  15. Harel J, Koch C, and Perona P (2007) Graph-based visual saliency. In Proc. of Advances in Neural Information Processing Systems, pp. 545–552

  16. Hiremath P, Pujari J (2008) Content based image retrieval using color boosted salient points and shape features of an image. Int J Image Process 2(1):10–17

    Google Scholar 

  17. Hou X, Zhang L (2007) Saliency detection: a spectral residual approach. Proc. of IEEE Conference on Computer Vision Pattern Recognition, In, pp. 1–8

    Google Scholar 

  18. Hu P, Wang W, Zhang C, Lu K (2016) Detecting salient objects via color and texture compactness hypotheses. IEEE Trans Image Process 25(10):4653–4664

    Article  MathSciNet  Google Scholar 

  19. Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259

    Article  Google Scholar 

  20. Jia Y, and Han M (2013) Category-independent object-level saliency detection. In: Proc. of European Conference on Computer Vision, pp. 1761–1768

  21. Jiang B, Zhang L, Lu H, Yang C, and Yang MH (2013) saliency detection via absorbing Markov chain. In: Proc. of IEEE International Conference on Computer Vision, pp. 1665–1672

  22. Jiang H, Wang J, Yuan Z, Wu Y, Zheng N, and Li S (2013) salient object detection: a discriminative regional feature integration approach. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 2083-2090

  23. Kim J, Han D, Tai YW, Kim J (2016) Salient region detection via high-dimensional color transform and local spatial support. IEEE Trans Image Process 25(1):9–23

    Article  MathSciNet  Google Scholar 

  24. Kolmogorov V, Zabin R (2004) What energy functions can be minimized via graph cuts? IEEE Trans Pattern Anal Mach Intell 26(2):147–159

    Article  Google Scholar 

  25. Le Meur O, and Liu Z (2014) Saliency aggregation: does unity make strength? In Proc. of Asia Conference on Computer Vision, pp. 18–32

  26. Li J, Tian Y, Duan L, Huang T (2013) Estimating visual saliency through single image optimization. IEEE Signal Process Lett 20(9):845–848

    Article  Google Scholar 

  27. Li X, Lu H, Zhang L, Ruan X, and Yang MH (2013) Saliency detection via dense and sparse reconstruction. In: Proc. of IEEE International Conference on Computer Vision, pp. 2976–2983

  28. Li Y, Hou X, Koch C, Rehg JM, and Yuille AL (2014) The secrets of salient object segmentation. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 280–287

  29. Liu Z, Xue Y, Shen L, Zhang Z (2010) Nonparametric saliency detection using kernel density estimation. Proc. of IEEE International Conference on Image Processing, In, pp. 253–256

    Google Scholar 

  30. Liu T, Yuan Z, Sun J, Wang J, Zheng N, Tang X, Shum HY (2011) Learning to detect a salient object. IEEE Trans Pattern Anal Mach Intell 33(2):53–367

    Google Scholar 

  31. Liu Z, Xue Y, Yan H, Zhang Z (2011) Efficient saliency detection based on Gaussian models. IET Image Process 5(2):122–131

    Article  Google Scholar 

  32. Liu Z, Shi R, Shen L, Xue Y, Ngan KN, Zhang Z (2012) Unsupervised salient object segmentation based on kernel density estimation and two-phase graph cut. IEEE Trans Multimed 14(4):1275–1289

    Article  Google Scholar 

  33. Liu Z, Le Meur O, Luo S, Shen L (2013) Saliency detection using regional histograms. Opt Lett 38(5):700–702

    Article  Google Scholar 

  34. Liu Z, Le Meur O, Luo S (2013) Superpixel-based saliency detection. Proc. of IEEE International Workshop on Image and Audio Analysis for Multimedia Interactive Services, In, pp. 1–4

    Google Scholar 

  35. Liu Z, Zou W, Le Meur O (2014) Saliency tree: a novel saliency detection framework. IEEE Trans Image Process 23(5):1937–1952

    Article  MathSciNet  Google Scholar 

  36. Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense SIFT. Inf Fusion 23:139–155

    Article  Google Scholar 

  37. Mai L, Liu F (2014) Comparing salient object detection results without ground truth. In: Proc. of European Conference on Computer Vision, pp. 76–91

  38. Mai L, Niu Y, and Liu F (2013) Saliency aggregation: a data-driven approach. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 1131–1138

  39. Margolin R, Zelnik-Manor L, and Tal A (2014) How to evaluate foreground maps? In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 248–255

  40. Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66

    Article  MathSciNet  Google Scholar 

  41. Pan Z, You X, Chen H, Tao D, Pang B (2013) Generalization performance of magnitude-preserving semi-supervised ranking with graph-based regularization. Inf Sci 221:284–296

    Article  MATH  MathSciNet  Google Scholar 

  42. Qin Y, Lu H, Xu Y, Wang H (2015) Saliency detection via cellular automata. Proc. of IEEE Conference on Computer Vision and Pattern Recognition, In, pp. 110–119

    Google Scholar 

  43. Rother C, Kolmogorov V, Blake A (2004) GrabCut: interactive foreground extraction using iterated graph cuts. ACM Trans Graph 23(3):309–314

    Article  Google Scholar 

  44. Shamir A, Avidan S (2009) Seam carving for media retargeting. Commun ACM 52(1):77–85

    Article  Google Scholar 

  45. Sharma G, Jurie F, Schmid C (2012) Discriminative spatial saliency for image classification. Proc. of IEEE Conference on Computer Vision Pattern Recognition, In, pp. 3506–3513

    Google Scholar 

  46. Shen L, Liu Z, Zhang Z (2013) A novel H.264 rate control algorithm with consideration of visual attention. Multimed Tools Appl 63(3):709–727

    Article  Google Scholar 

  47. Shi J, Malik J (1997) Normalized cuts and image segmentation. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 731–737

  48. Shi R, Liu Z, Du H, Zhang X, Shen L (2012) Region diversity maximization for salient object detection. IEEE Signal Process Lett 19(4):215–218

    Article  Google Scholar 

  49. THUR15000 dataset [Online] (2015) Available: http://mmcheng.net/gsal/

  50. Wei Y, Wen F, Zhu W, Sun J (2012) Geodesic saliency using background priors. Proc. of European Conference on Computer Vision, In, pp. 29–42

    Google Scholar 

  51. Xie Y, Lu H, Yang MH (2013) Bayesian saliency via low and mid-level cues. IEEE Trans Image Process 22(5):1689–1698

    Article  MathSciNet  MATH  Google Scholar 

  52. Xu C, Tao D, Xu C (2015) Multi-view intact space learning. IEEE Trans Pattern Anal Mach Intell 37(12):2531–2544

    Article  Google Scholar 

  53. Yan Q, Xu L, Shi J, and Jia J (2013) Hierarchical saliency detection. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 1155–1162

  54. Yang F, Lu H, and Chen YW (2010) Human tracking by multiple kernel boosting with locality affinity constraints. In Proc. of Asia Conference on Computer Vision, pp. 39–50.

  55. Zhang L, Tong M, Marks T, Shan H, Cottrell G (2008) SUN: a Bayesian framework for saliency using natural statistics. J Vis 8(7):1–20

    Article  Google Scholar 

  56. Zhou X, Liu Z, Sun G, Ye L, Wang X (2016) Improving saliency detection via multiple kernel boosting and adaptive fusion. IEEE Signal Process Lett 23(4):517–521

    Article  Google Scholar 

  57. Zhu W, Liang S, Wei Y, and Sun J (2014) Saliency optimization from robust background detection. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 2814–2821.

  58. Zou W, Kpalma K, Liu Z, Ronsin J (2013) Segmentation driven low-rank matrix recovery for saliency detection. Proc. of British Machine Vision Conference, In, pp. 1–13

    Google Scholar 

Download references

Acknowledgments

This work was supported by National Natural Science Foundation of China under Grant No. 61471230 and No. 61171144, and by the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zhi Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, X., Liu, Z., Sun, G. et al. Adaptive saliency fusion based on quality assessment. Multimed Tools Appl 76, 23187–23211 (2017). https://doi.org/10.1007/s11042-016-4093-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-4093-8

Keywords

Navigation