Real-time saliency detection for greyscale and colour images

Abstract

Unsupervised salient image generation without the aid of prior assumptions has many applications in computer vision. We present three unique real-time saliency generation algorithms that provide state-of-the-art performance for greyscale and colour images. Our fastest method run under 50 ms per frame on average. Our algorithm introduces a novel weighted histogram of orientation feature to supplement image intensity for monochromatic image manifold ranking. We also provide a method of dimensional reduction for the non-normalized optimal affinity matrix (OAM) using principal components analysis; this novel technique allows faster computation and stabilization of the OAM inversion process. We compare our methods with 18 traditional and recent techniques using three standard and custom datasets including ECSSD, DUT-OMRON and MSRA10K totalling 32,536 images for colour and greyscale variations. The results show our method to be more than \(10{\times }\) faster than the RC and GMR models and having similar or better precision performances.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Notes

  1. 1.

    https://mmcheng.net/salobjbenchmark/.

  2. 2.

    Discounting the discontinuous subpixels.

  3. 3.

    http://ai-automata.ca/research/hisafe.html.

  4. 4.

    AMD FX-8350 8-4.0GHz.

  5. 5.

    AMD A4-5000-APU 4-1.5GHz.

  6. 6.

    Intel 2-Quad-Q6600 4-2.4GHz.

References

  1. 1.

    Achanta, R., Süsstrunk, S.: Saliency detection using maximum symmetric surround. In: 17th IEEE International Conference on Image Processing, Hong Kong, China (2010)

  2. 2.

    Achanta, R., Estrada, F., Wils, P., Süsstrunk, S.: Salient region detection and segmentation. In: IEEE International Conference on Computer Vision (2008)

  3. 3.

    Achanta, R., Hemami, S., Estrada, F., Süsstrunk, S.: Frequency-tuned salient region detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1597–1604 (2009)

  4. 4.

    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2281 (2012)

    Article  Google Scholar 

  5. 5.

    Alahi, A., Ortiz, R., Vandergheynst, P.: Freak: Fast retina keypoint. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 510–517 (2012)

  6. 6.

    Alcantarilla, P., Nuevo, J., Bartoli, A.: Fast explicit diffusion for accelerated features in nonlinear scale spaces. In: British Machine Vision Conference, Bristol, UK (2013)

  7. 7.

    Assens, M., Giro-i Nieto, X., McGuinness, K., Ne, O.: Saltinet: scan-path prediction on 360 degree images using saliency volumes. In: IEEE International Conference on Computer Vision Workshop (2017)

  8. 8.

    Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. ACM Trans. Graph. 26(3), 10–19 (2007)

    Article  Google Scholar 

  9. 9.

    Aye, H., Zaw, S.: Salient object based action recognition using histogram of changing edge orientation. In: IEEE International Conference on Software Engineering Research, Management and Applications (2017)

  10. 10.

    Bay, H., Ess, A., Tuytelaars, T., Gool, L.: Speeded-up robust features. Comput. Vis. Image Underst. 110(3), 346–359 (2008)

    Article  Google Scholar 

  11. 11.

    Borenstein, E., Malik, J.: Shape guided object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 969–976 (2006)

  12. 12.

    Borji, A.: Boosting bottom-up and top-down visual features for saliency estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 438–445 (2012)

  13. 13.

    Borji, A., Cheng, M., Jiang, H., Li, J.: Salient object detection: a benchmark. IEEE Trans. Image Process. 24(12), 5706–5723 (2015)

    MathSciNet  MATH  Article  Google Scholar 

  14. 14.

    Calonder, M., Lepetit, V., Strecha, C., Fua, P.: Brief: binary robust independent elementary features. In: European Conference on Computer Vision, Heraklion, Crete (2010)

  15. 15.

    Chan, K.: Saliency/non-saliency segregation in video sequences using perception-based local ternary pattern features. In: IEEE International Conference on Machine Vision Applications (2017)

  16. 16.

    Chen, T., Lin, L., Liu, L., Luo, X., Li, X.: Disc: deep image saliency computing via progressive representation learning. IEEE Trans. Neural Netw. Learn. Syst. 27(6), 1135–1149 (2016)

    MathSciNet  Article  Google Scholar 

  17. 17.

    Cheng, M., Zhang, G., Mitra, N., Huang, X., Hu, S.: Global contrast based salient region detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 409–416 (2011)

  18. 18.

    Cheng, M., Warrell, J., Lin, W., Zheng, S., Vineet, V., Crook, N.: Efficient salient region detection with soft image abstraction. In: IEEE International Conference on Computer Vision, pp. 1529–1536 (2013)

  19. 19.

    Cheng, M., Zhang, Z., Lin, W., Torr, P.: Bing: binarized normed gradients for objectness estimates at 300 fps. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3286–3293 (2014)

  20. 20.

    Cheng, M., Mitra, N., Huang, X., Torr, P., Hu, S.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)

    Article  Google Scholar 

  21. 21.

    Christopoulos, C., Skodras, A., Ebrahimi, T.: The JPEG2000 still image coding system: an overview. IEEE Trans. Consum. Electron. 46(4), 1103–1127 (2002)

    Article  Google Scholar 

  22. 22.

    Fattal, A.K., Karg, M., Scharfenberger, C., Adamy, J.: Saliency-guided region proposal network for CNN based object detection. In: IEEE International Conference on Intelligent Transportation System (2017)

  23. 23.

    Fawcett, T.: An introduction to ROC analysis. Pattern Recognit. Lett. 27, 861–874 (2006)

    Article  Google Scholar 

  24. 24.

    Felzenszwalb, P., Huttenlocher, D.: Efficient graph-based image segmentation. Int. J. Comput. Vis. 59(2), 167–181 (2004)

    Article  Google Scholar 

  25. 25.

    Fujiwara, Y., Irie, G., Kuroyama, S., Onizuka, M.: Scaling manifold ranking based image retrieval. In: Proceedings of the VLDB Endowment, vol. 8 (2014)

  26. 26.

    Guo, C., Ma, Q., Zhang, L.: Spatio-temporal saliency detection using phase spectrum of quaternion Fourier transform. In: IEEE Conference on Computer Vision and Pattern Recognition (2008)

  27. 27.

    Hou, X., Zhang, L.: Saliency detection: a spectral residual approach. In: IEEE Conference on Computer Vision and Pattern Recognition (2007)

  28. 28.

    Howard, R., Heaton, A., Pinson, R., Carrington, C.: Orbital express advanced video guidance sensor. In: IEEE Aerospace Conference, Big Sky, MT (2008)

  29. 29.

    Hu, P., Shuai, B., Liu, J., Wang, G.: Deep levelsets for salient object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)

  30. 30.

    Huang, K., Gao, S.: Image saliency detection via multi-scale iterative CNN. Vis. Comput. J. (2019). https://doi.org/10.1007/s00371-019-01734-2

    Article  Google Scholar 

  31. 31.

    Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention, for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Article  Google Scholar 

  32. 32.

    Julesz, B.: Textons, the elements of texture perception, and their interactions. Nature 290(5802), 91–97 (1981)

    Article  Google Scholar 

  33. 33.

    Jung, C., Kim, W., Yoo, S., Kim, C.: A novel monochromatic cue for detecting regions of visual interest. J. Image Vis. Comput. 32, 405–413 (2014)

    Article  Google Scholar 

  34. 34.

    Koch, C., Ullman, S.: Shifts in selective visual attention: towards the underlying neural circuitry. Hum. Neurobiol. 4, 219–227 (1985)

    Google Scholar 

  35. 35.

    Leutenegger, S., Chli, M., Siegwart, R.: Brisk: binary robust invariant scalable keypoints. In: IEEE International Conference on Computer Vision, pp. 2548–2555 (2011)

  36. 36.

    Li, C., Xu, C., Gui, C., Fox, M.: Distance regularized level set evolution and its application to image segmentation. IEEE Trans. Image Process. 19(12), 3243–3254 (2010)

    MathSciNet  MATH  Article  Google Scholar 

  37. 37.

    Li, C., Zhang, B., Zhang, S., Sheng, H.: Saliency detection with relative location measure in light field image. In: IEEE International Conference on Image, Vision and Computing, Chengdu, China (2017)

  38. 38.

    Li, G., Xie, Y., Lin, L., Yu, Y.: Instance-level salient object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 247–256 (2017)

  39. 39.

    Li, J., Tian, Y., Huang, T., Gao, W.: Probabilistic multi-task learning for visual saliency estimation in video. Int. J. Comput. Vis. 90(2), 150–165 (2010)

    Article  Google Scholar 

  40. 40.

    Li, J., Tian, Y., Chen, X., Huang, T.: Measuring visual surprise jointly from intrinsic and extrinsic contexts for image saliency estimation. Int. J. Comput. Vis. 120(1), 44–60 (2016)

    MathSciNet  Article  Google Scholar 

  41. 41.

    Li, L., Zhou, F., Zheng, Y., Bai, X.: Saliency detection based on foreground appearance and background-prior. Neurocomputing 301, 46–61 (2018)

    Article  Google Scholar 

  42. 42.

    Li, S., Zeng, C., Fu, Y., Liu, S.: Optimizing multi-graph learning based salient object detection. Signal Process. Image Commun. 55, 93–105 (2017)

    Article  Google Scholar 

  43. 43.

    Li, Z., Chen, J.: Superpixel segmentation using linear spectral clustering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1356–1363 (2015)

  44. 44.

    Liu, T., Sun, J., Zheng, N., Tang, X., Shum, H.: Learning to detect a salient object. In: IEEE Conference on Computer Vision and Pattern Recognition (2007)

  45. 45.

    Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., Shum, H.: Learning to detect a salient object. IEEE Trans. Pattern Anal. Mach. Intell. 33(2), 353–367 (2011)

    Article  Google Scholar 

  46. 46.

    Liu, Z., Tang, J., Zhao, P.: Salient object detection via hybrid upsampling and hybrid loss computing. Vis. Comput. J. (2019). https://doi.org/10.1007/s00371-019-01659-w

    Article  Google Scholar 

  47. 47.

    Liu, Z., Xiang, Q., Tang, J., Wang, Y., Zhao, P.: Robust salient object detection for RGB images. Vis. Comput. J. (2019). https://doi.org/10.1007/s00371-019-01778-4

    Article  Google Scholar 

  48. 48.

    Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  49. 49.

    Lu, Y., Zhou, K., Wu, X., Gong, P.: A novel multi-graph framework for salient object detection. Vis. Comput. J. 35(11), 1683–1699 (2019)

    Article  Google Scholar 

  50. 50.

    Luo, Z., Mishra, A., Achkar, A., Eichel, S., Li, J., Jodoin, P.: Non-local deep features for salient object detection. In: IEEE International Conference on Computer Vision (2017)

  51. 51.

    Malik, J., Perona, P.: Preattentive texture discrimination with early vision mechanisms. J. Opt. Soc. Am. A 7(5), 923–932 (1990)

    Article  Google Scholar 

  52. 52.

    Margolin, R., Tal, A., Zelnik-Manor, L.: What makes a patch distinct? In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1139–1146 (2013)

  53. 53.

    Meyer, F.: Color image segmentation. In: IET International Conference on Image Processing and Its Applications (1992)

  54. 54.

    Otsu, N.: A threshold selection method from gray-level histograms. Automatica 11, 23–27 (1975)

    Google Scholar 

  55. 55.

    Qi, W., Cheng, M., Borji, A., Lu, H., Bai, L.: Saliencyrank: two-stage manifold ranking for salient object detection. Comput. Vis. Media 1(4), 309–320 (2015)

    MATH  Article  Google Scholar 

  56. 56.

    Rahtu, E., Kannala, J., Salo, M., Heikkilä, J.: Segmenting salient objects from images and videos. In: European Conference on Computer Vision, pp. 366–379 (2010)

  57. 57.

    Rother, C., Kolmogorov, V., Blake, A.: "grabcut"-interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)

    Article  Google Scholar 

  58. 58.

    Seo, H., Milanfar, P.: Static and space-time visual saliency detection by self-resemblance. J. Vis. 9(12), 15 (2009)

    Article  Google Scholar 

  59. 59.

    Shi, J., Yan, Q., Xu, L., Jia, J.: Hierarchical image saliency detection on extended CSSD. IEEE Trans. Pattern Anal. Mach. Intell. 38(4), 717–729 (2016)

    Article  Google Scholar 

  60. 60.

    Shi, J., Ulrich, S., Ruel, S.: Regional method for monocular infrared image spacecraft pose estimation. In: Proceedings of the AIAA Space Conference and Exhibit, Orlando, FL (2018)

  61. 61.

    Shigematsu, R., Feng, D., You, S., Barnes, N.: Learning RGB-D salient object detection using background enclosure, depth contrast, and top-down features. In: IEEE International Conference on Computer Vision (2017)

  62. 62.

    Strand, R., Ciesielski, K., Malmberg, F., Saha, P.: The minimum barrier distance. Comput. Vis. Image Underst. 117(4), 429–437 (2013)

    MATH  Article  Google Scholar 

  63. 63.

    Tan, Z., Wan, L., Feng, W., Pun, C.: Image co-saliency detection by propagating superpixel affinities. In: IEEE International Conference on Acoustics, Speech and Signal Processing (2013)

  64. 64.

    Tian, Z., Zheng, N., Xue, J., Lan, X., Li, C., Zhou, G.: Video object segmentation with shape cue based on spatiotemporal superpixel neighbourhood. IET Comput. Vis. 8(1), 16–25 (2014)

    Article  Google Scholar 

  65. 65.

    Van den Bergh, M., Boix, X., Roig, G., Van Gool, L.: Seeds: superpixels extracted via energy-driven sampling. Int. J. Comput. Vis. 111(3), 298–314 (2015)

    MathSciNet  Article  Google Scholar 

  66. 66.

    Varma, M., Zisserman, A.: A statistical approach to texture classification from single images. Int. J. Comput. Vis. 62(1), 61–81 (2005)

    Article  Google Scholar 

  67. 67.

    Wan, X., Yang, J., Xiao, J.: Manifold-ranking based topic-focused multi-document summarization. In: Proceedings of International Joint Conference on Artificial Intelligence, pp. 2903–2908 (2007)

  68. 68.

    Wang, J., Lu, H., Li, X., Tong, N., Liu, W.: Saliency detection via background and foreground seed selection. Neurocomputing 152, 359–368 (2015)

    Article  Google Scholar 

  69. 69.

    Wang, J., Jiang, H., Yuan, Z., Cheng, M., Hu, X., Zheng, N.: Salient object detection: a discriminative regional feature integration approach. Int. J. Comput. Vis. 123(2), 251–268 (2017)

    Article  Google Scholar 

  70. 70.

    Wang, W., Shen, J., Yang, R., Porikli, F.: Saliency-aware video object segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 40(1), 20–33 (2018)

    Article  Google Scholar 

  71. 71.

    Wang, Y., Wei, X., Ding, L., Tang, X., Zhang, H.: A robust visual tracking method via local feature extraction and saliency detection. Vis. Comput. J. (2019). https://doi.org/10.1007/s00371-019-01646-1

    Article  Google Scholar 

  72. 72.

    Wang, Z., Wu, X.: Salient object detection using biogeography-based optimization to combine features. Appl. Intell. 45(1), 1–17 (2016)

    Article  Google Scholar 

  73. 73.

    Wei, Y., Wen, F., Zhu, W., Sun, J.: Geodesic saliency using background priors. In: European Conference on Computer Vision, pp. 29–42 (2012)

  74. 74.

    Wu, X., Lin, X., Jiang, L., Zhao, D.: An improved manifold ranking based method for saliency detection. In: IEEE International Conference on Systems and Informatics (2017)

  75. 75.

    Xia, C., Li, J., Chen, X., Zheng, A., Zhang, Y.: What is and what is not a salient object, learning salient object detector by ensembling linear exemplar regressors. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4321–4329 (2017)

  76. 76.

    Xu, B., Bu, J., Chen, C., Cai, D., He, X., Liu, W., Luo, J.: Efficient manifold ranking for image retrieval. In: International ACM Conference on Research and Development in Information Retrieval (2011)

  77. 77.

    Xu, Y., Li, J., Chen, J., Shen, G., Gao, Y.: A novel approach for visual saliency detection and segmentation based on objectness and top-down attention. In: IEEE International Conference on Image, Vision and Computing, pp. 4321–4329 (2017)

  78. 78.

    Yacoob, Y., Davis, L.: Segmentation using meta-texture saliency. In: IEEE International Conference on Computer Vision (2007)

  79. 79.

    Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: IEEE Conference on Computer Vision and Pattern Recognition (2013)

  80. 80.

    Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.: Saliency detection via graph-based manifold ranking. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3166–3173 (2013)

  81. 81.

    Yang, C., Pu, J., Dong, Y., Xie, G., Si, Y., Liu, Z.: Scene classification-oriented saliency detection via the modularized prescription. Vis. Comput. J. 35(4), 473–488 (2019)

    Article  Google Scholar 

  82. 82.

    Yang, J., Yang, M.: Top-down visual saliency via joint CRF and dictionary learning. IEEE Trans. Pattern Anal. Mach. Intell. 39(3), 576–588 (2017)

    Article  Google Scholar 

  83. 83.

    Ye, L., Liu, Z., Zhou, X., Shen, L., Zhang, J.: Saliency detection via similar image retrieval. IEEE Signal Process. Lett. 23(6), 838–842 (2016)

    Article  Google Scholar 

  84. 84.

    Ye, R., Chen, Z.: Universal enhancement of salient object detection. In: IEEE International Conference on Multimedia and Expo (2017)

  85. 85.

    Zhai, Y., Shah, M.: Visual attention detection in video sequences using spatiotemporal cues. In: ACM International Conference on Multimedia (2006)

  86. 86.

    Zhang, D., Han, J., Zhang, Y.: Supervision by fusion: Towards unsupervised learning of deep salient object detector. In: IEEE International Conference on Computer Vision (2017)

  87. 87.

    Zhang, J., Shen, Y.: Spectral segmentation via minimum barrier distance. Multimed. Tools Appl. 76(24), 25713–25729 (2017)

    Article  Google Scholar 

  88. 88.

    Zhang, J., Sclaroff, S., Lin, Z., Shen, X., Price, B., Mĕch, R.: Minimum barrier salient object detection at 80 fps. In: IEEE International Conference on Computer Vision (2015)

  89. 89.

    Zhang, L., Yang, C., Lu, H., Ruan, X., Yang, M.: Ranking saliency. IEEE Trans. Pattern Anal. Mach. Intell. 39(9), 1892–1904 (2017)

    Article  Google Scholar 

  90. 90.

    Zhao, J., Gao, X., Chen, Y., Feng, H.: Optical imaging system-based real-time image saliency extraction method. Opt. Eng. 54(4), 43101-1–43101-8 (2015)

    Google Scholar 

  91. 91.

    Zhou, D., Weston, J., Gretton, A.: Ranking on data manifolds. In: Conference on Neural Information Processing Systems (2004)

Download references

Acknowledgements

The first author wishes to thank Dr. Gerhard Roth of Carleton University for his continuing guidance and support.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jian-Feng Shi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was jointly funded by the NSERC Scholarship CGSD3-453738-2014, CSA STDP and OCE VIP II Award 24053.

Proofs

Proofs

The proof of Lemma 1 is as follows,

Proof

If the alternative form of \(\bar{\mathbf {A}}\) from Lemma 1 is equivalent, then when multiplied by the inverse it will result in the identity.

$$\begin{aligned} \begin{aligned}&\left( \mathbf {1}-\alpha \mathbf {E}\mathbf {F}\right) \left( \mathbf {1}-\mathbf {E} \left( \mathbf {F}\mathbf {E}-\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1} \mathbf {F}\right) \\&\quad = \mathbf {1} -\alpha \mathbf {E}\mathbf {F} -\mathbf {E}\left( \mathbf {F}\mathbf {E}-\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1}\mathbf {F} +\alpha \mathbf {E}\mathbf {F}\mathbf {E}\left( \mathbf {F}\mathbf {E} -\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1}\mathbf {F} \\&\quad = \mathbf {1} -\alpha \mathbf {E}\mathbf {F} +\alpha \mathbf {E}\left( -\frac{1}{\alpha }\mathbf {1}_{d}+\mathbf {F}\mathbf {E}\right) \left( \mathbf {F}\mathbf {E}-\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1}\mathbf {F} \\&\quad = \mathbf {1} \end{aligned} \end{aligned}$$

\(\square \)

The proof of Corollary 1 is as follows,

Proof

Substituting Lemma 1 into Eq. (6), and using the previous definitions for \(\mathbf {E}\) and \(\mathbf {F}\),

$$\begin{aligned} \begin{aligned} \mathbf {A}&=\left( \mathbf {D}-\alpha \mathbf {W}\right) ^{-1} \\&=\mathbf {D}^{-\frac{1}{2}}\left( \mathbf {1}-\mathbf {E}\left( \mathbf {F}\mathbf {E} -\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1}\mathbf {F} \right) \mathbf {D}^{-\frac{1}{2}} \\&=\mathbf {D}^{-1} \left( \mathbf {1}-\mathbf {P}\left( \mathbf {Q}\mathbf {D}^{-1}\mathbf {P} -\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1}\mathbf {Q}\mathbf {D}^{-1}\right) \end{aligned} \end{aligned}$$

\(\square \)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Shi, J., Ulrich, S. & Ruel, S. Real-time saliency detection for greyscale and colour images. Vis Comput (2020). https://doi.org/10.1007/s00371-020-01865-x

Download citation

Keywords

  • Image saliency
  • Image segmentation
  • Image features