Abstract
A variety of saliency models based on different schemes and methods have been proposed in the recent years, and the performance of these models often vary with images and complement each other. Therefore it is a natural idea whether we can elevate saliency detection performance by fusing different saliency models. This paper proposes a novel and general framework to adaptively fuse saliency maps generated using various saliency models based on quality assessment of these saliency maps. Given an input image and its multiple saliency maps, the quality features based on the input image and saliency maps are extracted. Then, a quality assessment model, which is learned using the boosting algorithm with multiple kernels, indicates the quality score of each saliency map. Next, a linear summation method with power-law transformation is exploited to fuse these saliency maps adaptively according to their quality scores. Finally, a graph cut based refinement method is exploited to enhance the spatial coherence of saliency and generate the high-quality final saliency map. Experimental results on three public benchmark datasets with state-of-the-art saliency models demonstrate that our saliency fusion framework consistently outperforms all saliency models and other fusion methods, and effectively elevates saliency detection performance.
Similar content being viewed by others
References
Achanta R, Hemami SS, Estrada FJ, Süsstrunk S (2009) Frequency tuned salient region detection. Proc. of IEEE Conference on Computer Vision Pattern Recognition, In, pp. 1597–1604
Aytekin C, Kiranyaz S, Gabbouj M (2014) Automatic object segmentation by quantum cuts. Proc. of IEEE International Conference on Pattern Recognition, In, pp. 112–117
Borji A, Sihite D.N., Itti L (2012) Salient object detection: a benchmark. In: Proc. of European Conference on Computer Vision, pp. 414–429
Borji A, Cheng MM, Jiang H, Li J (2015) Salient object detection: a benchmark. IEEE Trans Image Process 24(12):5706–5722
Boykov Y, and Jolly MP (2001) Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images. In: Proc. of IEEE International Conference on Computer Vision, pp. 105–112
Boykov Y, Kolmogorov V (2004) An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans Pattern Anal Mach Intell 26(9):1124–1137
Cao Z, Qin T, Liu T.Y., Tsai M.F., Li H (2007) Learning to rank: from pairwise approach to listwise approach. In: Proc. of International Conference on Machine Learning, pp.129–136
Cheng MM, Zhang GX, Mitra NJ, Huang X, Hu SM (2011) Global contrast based salient region detection. Proc. of IEEE Conference on Computer Vision Pattern Recognition, In, pp. 409–416
Dollár P, Zitnick C (2013) Structured forests for fast edge detection. In: Proc. of IEEE International Conference on Computer Vision, pp. 1841-1848
Du H, Liu Z, Jiang J, Shen L (2013) Stretchability-aware block scaling for image retargeting. J Vis Commun Image Represent 24(4):499–508
Fang Y, Chen Z, Lin W, and Lin C.-W. (2012) Saliency detection in the compressed domain for adaptive image retargeting. IEEE Trans Image Process, 21(9): 3888–3901
Fang Y, Lin W, Lee B.-S., Lau CT, Chen Z, and Lin C.-W. (2012) Bottom-up saliency detection model based on human visual sensitivity and amplitude spectrum. IEEE Trans Multimed, 12(1): 187–198
Goferman S, Zelnik-Manor L, Tal A (2010) Context-aware saliency detection. Proc. of IEEE Conference on Computer Vision Pattern Recognition, In, pp. 2376–2383
Guo C, Zhang L (2010) A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans Image Process 19(1):85–198
Harel J, Koch C, and Perona P (2007) Graph-based visual saliency. In Proc. of Advances in Neural Information Processing Systems, pp. 545–552
Hiremath P, Pujari J (2008) Content based image retrieval using color boosted salient points and shape features of an image. Int J Image Process 2(1):10–17
Hou X, Zhang L (2007) Saliency detection: a spectral residual approach. Proc. of IEEE Conference on Computer Vision Pattern Recognition, In, pp. 1–8
Hu P, Wang W, Zhang C, Lu K (2016) Detecting salient objects via color and texture compactness hypotheses. IEEE Trans Image Process 25(10):4653–4664
Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259
Jia Y, and Han M (2013) Category-independent object-level saliency detection. In: Proc. of European Conference on Computer Vision, pp. 1761–1768
Jiang B, Zhang L, Lu H, Yang C, and Yang MH (2013) saliency detection via absorbing Markov chain. In: Proc. of IEEE International Conference on Computer Vision, pp. 1665–1672
Jiang H, Wang J, Yuan Z, Wu Y, Zheng N, and Li S (2013) salient object detection: a discriminative regional feature integration approach. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 2083-2090
Kim J, Han D, Tai YW, Kim J (2016) Salient region detection via high-dimensional color transform and local spatial support. IEEE Trans Image Process 25(1):9–23
Kolmogorov V, Zabin R (2004) What energy functions can be minimized via graph cuts? IEEE Trans Pattern Anal Mach Intell 26(2):147–159
Le Meur O, and Liu Z (2014) Saliency aggregation: does unity make strength? In Proc. of Asia Conference on Computer Vision, pp. 18–32
Li J, Tian Y, Duan L, Huang T (2013) Estimating visual saliency through single image optimization. IEEE Signal Process Lett 20(9):845–848
Li X, Lu H, Zhang L, Ruan X, and Yang MH (2013) Saliency detection via dense and sparse reconstruction. In: Proc. of IEEE International Conference on Computer Vision, pp. 2976–2983
Li Y, Hou X, Koch C, Rehg JM, and Yuille AL (2014) The secrets of salient object segmentation. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 280–287
Liu Z, Xue Y, Shen L, Zhang Z (2010) Nonparametric saliency detection using kernel density estimation. Proc. of IEEE International Conference on Image Processing, In, pp. 253–256
Liu T, Yuan Z, Sun J, Wang J, Zheng N, Tang X, Shum HY (2011) Learning to detect a salient object. IEEE Trans Pattern Anal Mach Intell 33(2):53–367
Liu Z, Xue Y, Yan H, Zhang Z (2011) Efficient saliency detection based on Gaussian models. IET Image Process 5(2):122–131
Liu Z, Shi R, Shen L, Xue Y, Ngan KN, Zhang Z (2012) Unsupervised salient object segmentation based on kernel density estimation and two-phase graph cut. IEEE Trans Multimed 14(4):1275–1289
Liu Z, Le Meur O, Luo S, Shen L (2013) Saliency detection using regional histograms. Opt Lett 38(5):700–702
Liu Z, Le Meur O, Luo S (2013) Superpixel-based saliency detection. Proc. of IEEE International Workshop on Image and Audio Analysis for Multimedia Interactive Services, In, pp. 1–4
Liu Z, Zou W, Le Meur O (2014) Saliency tree: a novel saliency detection framework. IEEE Trans Image Process 23(5):1937–1952
Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense SIFT. Inf Fusion 23:139–155
Mai L, Liu F (2014) Comparing salient object detection results without ground truth. In: Proc. of European Conference on Computer Vision, pp. 76–91
Mai L, Niu Y, and Liu F (2013) Saliency aggregation: a data-driven approach. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 1131–1138
Margolin R, Zelnik-Manor L, and Tal A (2014) How to evaluate foreground maps? In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 248–255
Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66
Pan Z, You X, Chen H, Tao D, Pang B (2013) Generalization performance of magnitude-preserving semi-supervised ranking with graph-based regularization. Inf Sci 221:284–296
Qin Y, Lu H, Xu Y, Wang H (2015) Saliency detection via cellular automata. Proc. of IEEE Conference on Computer Vision and Pattern Recognition, In, pp. 110–119
Rother C, Kolmogorov V, Blake A (2004) GrabCut: interactive foreground extraction using iterated graph cuts. ACM Trans Graph 23(3):309–314
Shamir A, Avidan S (2009) Seam carving for media retargeting. Commun ACM 52(1):77–85
Sharma G, Jurie F, Schmid C (2012) Discriminative spatial saliency for image classification. Proc. of IEEE Conference on Computer Vision Pattern Recognition, In, pp. 3506–3513
Shen L, Liu Z, Zhang Z (2013) A novel H.264 rate control algorithm with consideration of visual attention. Multimed Tools Appl 63(3):709–727
Shi J, Malik J (1997) Normalized cuts and image segmentation. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 731–737
Shi R, Liu Z, Du H, Zhang X, Shen L (2012) Region diversity maximization for salient object detection. IEEE Signal Process Lett 19(4):215–218
THUR15000 dataset [Online] (2015) Available: http://mmcheng.net/gsal/
Wei Y, Wen F, Zhu W, Sun J (2012) Geodesic saliency using background priors. Proc. of European Conference on Computer Vision, In, pp. 29–42
Xie Y, Lu H, Yang MH (2013) Bayesian saliency via low and mid-level cues. IEEE Trans Image Process 22(5):1689–1698
Xu C, Tao D, Xu C (2015) Multi-view intact space learning. IEEE Trans Pattern Anal Mach Intell 37(12):2531–2544
Yan Q, Xu L, Shi J, and Jia J (2013) Hierarchical saliency detection. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 1155–1162
Yang F, Lu H, and Chen YW (2010) Human tracking by multiple kernel boosting with locality affinity constraints. In Proc. of Asia Conference on Computer Vision, pp. 39–50.
Zhang L, Tong M, Marks T, Shan H, Cottrell G (2008) SUN: a Bayesian framework for saliency using natural statistics. J Vis 8(7):1–20
Zhou X, Liu Z, Sun G, Ye L, Wang X (2016) Improving saliency detection via multiple kernel boosting and adaptive fusion. IEEE Signal Process Lett 23(4):517–521
Zhu W, Liang S, Wei Y, and Sun J (2014) Saliency optimization from robust background detection. In: Proc. of IEEE Conference on Computer Vision Pattern Recognition, pp. 2814–2821.
Zou W, Kpalma K, Liu Z, Ronsin J (2013) Segmentation driven low-rank matrix recovery for saliency detection. Proc. of British Machine Vision Conference, In, pp. 1–13
Acknowledgments
This work was supported by National Natural Science Foundation of China under Grant No. 61471230 and No. 61171144, and by the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhou, X., Liu, Z., Sun, G. et al. Adaptive saliency fusion based on quality assessment. Multimed Tools Appl 76, 23187–23211 (2017). https://doi.org/10.1007/s11042-016-4093-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-016-4093-8