Skip to main content
SpringerLink
Log in

Real-time saliency detection for greyscale and colour images

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Unsupervised salient image generation without the aid of prior assumptions has many applications in computer vision. We present three unique real-time saliency generation algorithms that provide state-of-the-art performance for greyscale and colour images. Our fastest method run under 50 ms per frame on average. Our algorithm introduces a novel weighted histogram of orientation feature to supplement image intensity for monochromatic image manifold ranking. We also provide a method of dimensional reduction for the non-normalized optimal affinity matrix (OAM) using principal components analysis; this novel technique allows faster computation and stabilization of the OAM inversion process. We compare our methods with 18 traditional and recent techniques using three standard and custom datasets including ECSSD, DUT-OMRON and MSRA10K totalling 32,536 images for colour and greyscale variations. The results show our method to be more than \(10{\times }\) faster than the RC and GMR models and having similar or better precision performances.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. https://mmcheng.net/salobjbenchmark/.

  2. Discounting the discontinuous subpixels.

  3. http://ai-automata.ca/research/hisafe.html.

  4. AMD FX-8350 8-4.0GHz.

  5. AMD A4-5000-APU 4-1.5GHz.

  6. Intel 2-Quad-Q6600 4-2.4GHz.

References

  1. Achanta, R., Süsstrunk, S.: Saliency detection using maximum symmetric surround. In: 17th IEEE International Conference on Image Processing, Hong Kong, China (2010)

  2. Achanta, R., Estrada, F., Wils, P., Süsstrunk, S.: Salient region detection and segmentation. In: IEEE International Conference on Computer Vision (2008)

  3. Achanta, R., Hemami, S., Estrada, F., Süsstrunk, S.: Frequency-tuned salient region detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1597–1604 (2009)

  4. Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2281 (2012)

    Google Scholar 

  5. Alahi, A., Ortiz, R., Vandergheynst, P.: Freak: Fast retina keypoint. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 510–517 (2012)

  6. Alcantarilla, P., Nuevo, J., Bartoli, A.: Fast explicit diffusion for accelerated features in nonlinear scale spaces. In: British Machine Vision Conference, Bristol, UK (2013)

  7. Assens, M., Giro-i Nieto, X., McGuinness, K., Ne, O.: Saltinet: scan-path prediction on 360 degree images using saliency volumes. In: IEEE International Conference on Computer Vision Workshop (2017)

  8. Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. ACM Trans. Graph. 26(3), 10–19 (2007)

    Google Scholar 

  9. Aye, H., Zaw, S.: Salient object based action recognition using histogram of changing edge orientation. In: IEEE International Conference on Software Engineering Research, Management and Applications (2017)

  10. Bay, H., Ess, A., Tuytelaars, T., Gool, L.: Speeded-up robust features. Comput. Vis. Image Underst. 110(3), 346–359 (2008)

    Google Scholar 

  11. Borenstein, E., Malik, J.: Shape guided object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 969–976 (2006)

  12. Borji, A.: Boosting bottom-up and top-down visual features for saliency estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 438–445 (2012)

  13. Borji, A., Cheng, M., Jiang, H., Li, J.: Salient object detection: a benchmark. IEEE Trans. Image Process. 24(12), 5706–5723 (2015)

    MathSciNet  MATH  Google Scholar 

  14. Calonder, M., Lepetit, V., Strecha, C., Fua, P.: Brief: binary robust independent elementary features. In: European Conference on Computer Vision, Heraklion, Crete (2010)

  15. Chan, K.: Saliency/non-saliency segregation in video sequences using perception-based local ternary pattern features. In: IEEE International Conference on Machine Vision Applications (2017)

  16. Chen, T., Lin, L., Liu, L., Luo, X., Li, X.: Disc: deep image saliency computing via progressive representation learning. IEEE Trans. Neural Netw. Learn. Syst. 27(6), 1135–1149 (2016)

    MathSciNet  Google Scholar 

  17. Cheng, M., Zhang, G., Mitra, N., Huang, X., Hu, S.: Global contrast based salient region detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 409–416 (2011)

  18. Cheng, M., Warrell, J., Lin, W., Zheng, S., Vineet, V., Crook, N.: Efficient salient region detection with soft image abstraction. In: IEEE International Conference on Computer Vision, pp. 1529–1536 (2013)

  19. Cheng, M., Zhang, Z., Lin, W., Torr, P.: Bing: binarized normed gradients for objectness estimates at 300 fps. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3286–3293 (2014)

  20. Cheng, M., Mitra, N., Huang, X., Torr, P., Hu, S.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)

    Google Scholar 

  21. Christopoulos, C., Skodras, A., Ebrahimi, T.: The JPEG2000 still image coding system: an overview. IEEE Trans. Consum. Electron. 46(4), 1103–1127 (2002)

    Google Scholar 

  22. Fattal, A.K., Karg, M., Scharfenberger, C., Adamy, J.: Saliency-guided region proposal network for CNN based object detection. In: IEEE International Conference on Intelligent Transportation System (2017)

  23. Fawcett, T.: An introduction to ROC analysis. Pattern Recognit. Lett. 27, 861–874 (2006)

    Google Scholar 

  24. Felzenszwalb, P., Huttenlocher, D.: Efficient graph-based image segmentation. Int. J. Comput. Vis. 59(2), 167–181 (2004)

    Google Scholar 

  25. Fujiwara, Y., Irie, G., Kuroyama, S., Onizuka, M.: Scaling manifold ranking based image retrieval. In: Proceedings of the VLDB Endowment, vol. 8 (2014)

  26. Guo, C., Ma, Q., Zhang, L.: Spatio-temporal saliency detection using phase spectrum of quaternion Fourier transform. In: IEEE Conference on Computer Vision and Pattern Recognition (2008)

  27. Hou, X., Zhang, L.: Saliency detection: a spectral residual approach. In: IEEE Conference on Computer Vision and Pattern Recognition (2007)

  28. Howard, R., Heaton, A., Pinson, R., Carrington, C.: Orbital express advanced video guidance sensor. In: IEEE Aerospace Conference, Big Sky, MT (2008)

  29. Hu, P., Shuai, B., Liu, J., Wang, G.: Deep levelsets for salient object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)

  30. Huang, K., Gao, S.: Image saliency detection via multi-scale iterative CNN. Vis. Comput. J. (2019). https://doi.org/10.1007/s00371-019-01734-2

    Article  Google Scholar 

  31. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention, for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Google Scholar 

  32. Julesz, B.: Textons, the elements of texture perception, and their interactions. Nature 290(5802), 91–97 (1981)

    Google Scholar 

  33. Jung, C., Kim, W., Yoo, S., Kim, C.: A novel monochromatic cue for detecting regions of visual interest. J. Image Vis. Comput. 32, 405–413 (2014)

    Google Scholar 

  34. Koch, C., Ullman, S.: Shifts in selective visual attention: towards the underlying neural circuitry. Hum. Neurobiol. 4, 219–227 (1985)

    Google Scholar 

  35. Leutenegger, S., Chli, M., Siegwart, R.: Brisk: binary robust invariant scalable keypoints. In: IEEE International Conference on Computer Vision, pp. 2548–2555 (2011)

  36. Li, C., Xu, C., Gui, C., Fox, M.: Distance regularized level set evolution and its application to image segmentation. IEEE Trans. Image Process. 19(12), 3243–3254 (2010)

    MathSciNet  MATH  Google Scholar 

  37. Li, C., Zhang, B., Zhang, S., Sheng, H.: Saliency detection with relative location measure in light field image. In: IEEE International Conference on Image, Vision and Computing, Chengdu, China (2017)

  38. Li, G., Xie, Y., Lin, L., Yu, Y.: Instance-level salient object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 247–256 (2017)

  39. Li, J., Tian, Y., Huang, T., Gao, W.: Probabilistic multi-task learning for visual saliency estimation in video. Int. J. Comput. Vis. 90(2), 150–165 (2010)

    Google Scholar 

  40. Li, J., Tian, Y., Chen, X., Huang, T.: Measuring visual surprise jointly from intrinsic and extrinsic contexts for image saliency estimation. Int. J. Comput. Vis. 120(1), 44–60 (2016)

    MathSciNet  Google Scholar 

  41. Li, L., Zhou, F., Zheng, Y., Bai, X.: Saliency detection based on foreground appearance and background-prior. Neurocomputing 301, 46–61 (2018)

    Google Scholar 

  42. Li, S., Zeng, C., Fu, Y., Liu, S.: Optimizing multi-graph learning based salient object detection. Signal Process. Image Commun. 55, 93–105 (2017)

    Google Scholar 

  43. Li, Z., Chen, J.: Superpixel segmentation using linear spectral clustering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1356–1363 (2015)

  44. Liu, T., Sun, J., Zheng, N., Tang, X., Shum, H.: Learning to detect a salient object. In: IEEE Conference on Computer Vision and Pattern Recognition (2007)

  45. Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., Shum, H.: Learning to detect a salient object. IEEE Trans. Pattern Anal. Mach. Intell. 33(2), 353–367 (2011)

    Google Scholar 

  46. Liu, Z., Tang, J., Zhao, P.: Salient object detection via hybrid upsampling and hybrid loss computing. Vis. Comput. J. (2019). https://doi.org/10.1007/s00371-019-01659-w

    Article  Google Scholar 

  47. Liu, Z., Xiang, Q., Tang, J., Wang, Y., Zhao, P.: Robust salient object detection for RGB images. Vis. Comput. J. (2019). https://doi.org/10.1007/s00371-019-01778-4

    Article  Google Scholar 

  48. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Google Scholar 

  49. Lu, Y., Zhou, K., Wu, X., Gong, P.: A novel multi-graph framework for salient object detection. Vis. Comput. J. 35(11), 1683–1699 (2019)

    Google Scholar 

  50. Luo, Z., Mishra, A., Achkar, A., Eichel, S., Li, J., Jodoin, P.: Non-local deep features for salient object detection. In: IEEE International Conference on Computer Vision (2017)

  51. Malik, J., Perona, P.: Preattentive texture discrimination with early vision mechanisms. J. Opt. Soc. Am. A 7(5), 923–932 (1990)

    Google Scholar 

  52. Margolin, R., Tal, A., Zelnik-Manor, L.: What makes a patch distinct? In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1139–1146 (2013)

  53. Meyer, F.: Color image segmentation. In: IET International Conference on Image Processing and Its Applications (1992)

  54. Otsu, N.: A threshold selection method from gray-level histograms. Automatica 11, 23–27 (1975)

    Google Scholar 

  55. Qi, W., Cheng, M., Borji, A., Lu, H., Bai, L.: Saliencyrank: two-stage manifold ranking for salient object detection. Comput. Vis. Media 1(4), 309–320 (2015)

    MATH  Google Scholar 

  56. Rahtu, E., Kannala, J., Salo, M., Heikkilä, J.: Segmenting salient objects from images and videos. In: European Conference on Computer Vision, pp. 366–379 (2010)

  57. Rother, C., Kolmogorov, V., Blake, A.: "grabcut"-interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)

    Google Scholar 

  58. Seo, H., Milanfar, P.: Static and space-time visual saliency detection by self-resemblance. J. Vis. 9(12), 15 (2009)

    Google Scholar 

  59. Shi, J., Yan, Q., Xu, L., Jia, J.: Hierarchical image saliency detection on extended CSSD. IEEE Trans. Pattern Anal. Mach. Intell. 38(4), 717–729 (2016)

    Google Scholar 

  60. Shi, J., Ulrich, S., Ruel, S.: Regional method for monocular infrared image spacecraft pose estimation. In: Proceedings of the AIAA Space Conference and Exhibit, Orlando, FL (2018)

  61. Shigematsu, R., Feng, D., You, S., Barnes, N.: Learning RGB-D salient object detection using background enclosure, depth contrast, and top-down features. In: IEEE International Conference on Computer Vision (2017)

  62. Strand, R., Ciesielski, K., Malmberg, F., Saha, P.: The minimum barrier distance. Comput. Vis. Image Underst. 117(4), 429–437 (2013)

    MATH  Google Scholar 

  63. Tan, Z., Wan, L., Feng, W., Pun, C.: Image co-saliency detection by propagating superpixel affinities. In: IEEE International Conference on Acoustics, Speech and Signal Processing (2013)

  64. Tian, Z., Zheng, N., Xue, J., Lan, X., Li, C., Zhou, G.: Video object segmentation with shape cue based on spatiotemporal superpixel neighbourhood. IET Comput. Vis. 8(1), 16–25 (2014)

    Google Scholar 

  65. Van den Bergh, M., Boix, X., Roig, G., Van Gool, L.: Seeds: superpixels extracted via energy-driven sampling. Int. J. Comput. Vis. 111(3), 298–314 (2015)

    MathSciNet  Google Scholar 

  66. Varma, M., Zisserman, A.: A statistical approach to texture classification from single images. Int. J. Comput. Vis. 62(1), 61–81 (2005)

    Google Scholar 

  67. Wan, X., Yang, J., Xiao, J.: Manifold-ranking based topic-focused multi-document summarization. In: Proceedings of International Joint Conference on Artificial Intelligence, pp. 2903–2908 (2007)

  68. Wang, J., Lu, H., Li, X., Tong, N., Liu, W.: Saliency detection via background and foreground seed selection. Neurocomputing 152, 359–368 (2015)

    Google Scholar 

  69. Wang, J., Jiang, H., Yuan, Z., Cheng, M., Hu, X., Zheng, N.: Salient object detection: a discriminative regional feature integration approach. Int. J. Comput. Vis. 123(2), 251–268 (2017)

    Google Scholar 

  70. Wang, W., Shen, J., Yang, R., Porikli, F.: Saliency-aware video object segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 40(1), 20–33 (2018)

    Google Scholar 

  71. Wang, Y., Wei, X., Ding, L., Tang, X., Zhang, H.: A robust visual tracking method via local feature extraction and saliency detection. Vis. Comput. J. (2019). https://doi.org/10.1007/s00371-019-01646-1

    Article  Google Scholar 

  72. Wang, Z., Wu, X.: Salient object detection using biogeography-based optimization to combine features. Appl. Intell. 45(1), 1–17 (2016)

    Google Scholar 

  73. Wei, Y., Wen, F., Zhu, W., Sun, J.: Geodesic saliency using background priors. In: European Conference on Computer Vision, pp. 29–42 (2012)

  74. Wu, X., Lin, X., Jiang, L., Zhao, D.: An improved manifold ranking based method for saliency detection. In: IEEE International Conference on Systems and Informatics (2017)

  75. Xia, C., Li, J., Chen, X., Zheng, A., Zhang, Y.: What is and what is not a salient object, learning salient object detector by ensembling linear exemplar regressors. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4321–4329 (2017)

  76. Xu, B., Bu, J., Chen, C., Cai, D., He, X., Liu, W., Luo, J.: Efficient manifold ranking for image retrieval. In: International ACM Conference on Research and Development in Information Retrieval (2011)

  77. Xu, Y., Li, J., Chen, J., Shen, G., Gao, Y.: A novel approach for visual saliency detection and segmentation based on objectness and top-down attention. In: IEEE International Conference on Image, Vision and Computing, pp. 4321–4329 (2017)

  78. Yacoob, Y., Davis, L.: Segmentation using meta-texture saliency. In: IEEE International Conference on Computer Vision (2007)

  79. Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: IEEE Conference on Computer Vision and Pattern Recognition (2013)

  80. Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.: Saliency detection via graph-based manifold ranking. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3166–3173 (2013)

  81. Yang, C., Pu, J., Dong, Y., Xie, G., Si, Y., Liu, Z.: Scene classification-oriented saliency detection via the modularized prescription. Vis. Comput. J. 35(4), 473–488 (2019)

    Google Scholar 

  82. Yang, J., Yang, M.: Top-down visual saliency via joint CRF and dictionary learning. IEEE Trans. Pattern Anal. Mach. Intell. 39(3), 576–588 (2017)

    Google Scholar 

  83. Ye, L., Liu, Z., Zhou, X., Shen, L., Zhang, J.: Saliency detection via similar image retrieval. IEEE Signal Process. Lett. 23(6), 838–842 (2016)

    Google Scholar 

  84. Ye, R., Chen, Z.: Universal enhancement of salient object detection. In: IEEE International Conference on Multimedia and Expo (2017)

  85. Zhai, Y., Shah, M.: Visual attention detection in video sequences using spatiotemporal cues. In: ACM International Conference on Multimedia (2006)

  86. Zhang, D., Han, J., Zhang, Y.: Supervision by fusion: Towards unsupervised learning of deep salient object detector. In: IEEE International Conference on Computer Vision (2017)

  87. Zhang, J., Shen, Y.: Spectral segmentation via minimum barrier distance. Multimed. Tools Appl. 76(24), 25713–25729 (2017)

    Google Scholar 

  88. Zhang, J., Sclaroff, S., Lin, Z., Shen, X., Price, B., Mĕch, R.: Minimum barrier salient object detection at 80 fps. In: IEEE International Conference on Computer Vision (2015)

  89. Zhang, L., Yang, C., Lu, H., Ruan, X., Yang, M.: Ranking saliency. IEEE Trans. Pattern Anal. Mach. Intell. 39(9), 1892–1904 (2017)

    Google Scholar 

  90. Zhao, J., Gao, X., Chen, Y., Feng, H.: Optical imaging system-based real-time image saliency extraction method. Opt. Eng. 54(4), 43101-1–43101-8 (2015)

    Google Scholar 

  91. Zhou, D., Weston, J., Gretton, A.: Ranking on data manifolds. In: Conference on Neural Information Processing Systems (2004)

Download references

Acknowledgements

The first author wishes to thank Dr. Gerhard Roth of Carleton University for his continuing guidance and support.

Author information

Authors and Affiliations

  1. Mechanical and Aerospace Engineering Department, Carleton University, Ottawa, Canada

    Jian-Feng Shi & Steve Ulrich

  2. MDA Neptec Design Group Ltd., Ottawa, Canada

    Stéphane Ruel

Authors
  1. Jian-Feng Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steve Ulrich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stéphane Ruel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian-Feng Shi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was jointly funded by the NSERC Scholarship CGSD3-453738-2014, CSA STDP and OCE VIP II Award 24053.

Proofs

Proofs

The proof of Lemma 1 is as follows,

Proof

If the alternative form of \(\bar{\mathbf {A}}\) from Lemma 1 is equivalent, then when multiplied by the inverse it will result in the identity.

$$\begin{aligned} \begin{aligned}&\left( \mathbf {1}-\alpha \mathbf {E}\mathbf {F}\right) \left( \mathbf {1}-\mathbf {E} \left( \mathbf {F}\mathbf {E}-\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1} \mathbf {F}\right) \\&\quad = \mathbf {1} -\alpha \mathbf {E}\mathbf {F} -\mathbf {E}\left( \mathbf {F}\mathbf {E}-\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1}\mathbf {F} +\alpha \mathbf {E}\mathbf {F}\mathbf {E}\left( \mathbf {F}\mathbf {E} -\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1}\mathbf {F} \\&\quad = \mathbf {1} -\alpha \mathbf {E}\mathbf {F} +\alpha \mathbf {E}\left( -\frac{1}{\alpha }\mathbf {1}_{d}+\mathbf {F}\mathbf {E}\right) \left( \mathbf {F}\mathbf {E}-\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1}\mathbf {F} \\&\quad = \mathbf {1} \end{aligned} \end{aligned}$$

\(\square \)

The proof of Corollary 1 is as follows,

Proof

Substituting Lemma 1 into Eq. (6), and using the previous definitions for \(\mathbf {E}\) and \(\mathbf {F}\),

$$\begin{aligned} \begin{aligned} \mathbf {A}&=\left( \mathbf {D}-\alpha \mathbf {W}\right) ^{-1} \\&=\mathbf {D}^{-\frac{1}{2}}\left( \mathbf {1}-\mathbf {E}\left( \mathbf {F}\mathbf {E} -\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1}\mathbf {F} \right) \mathbf {D}^{-\frac{1}{2}} \\&=\mathbf {D}^{-1} \left( \mathbf {1}-\mathbf {P}\left( \mathbf {Q}\mathbf {D}^{-1}\mathbf {P} -\frac{1}{\alpha }\mathbf {1}_{d}\right) ^{-1}\mathbf {Q}\mathbf {D}^{-1}\right) \end{aligned} \end{aligned}$$

\(\square \)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi, JF., Ulrich, S. & Ruel, S. Real-time saliency detection for greyscale and colour images. Vis Comput 37, 1277–1296 (2021). https://doi.org/10.1007/s00371-020-01865-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01865-x

Keywords

Search

Navigation