Multimedia Tools and Applications

, Volume 78, Issue 22, pp 31807–31821 | Cite as

Detecting salient objects in non-stationary video image sequence for analyzing user perceptions of digital video contents

  • Timothy AdeliyiEmail author
  • Oludayo Olugbara


The ubiquitous utilization of video applications in recent years has made research on video quality of experience paramount. Lack of sufficient bandwidth deters the effective transmission of raw video contents to users. This bandwidth challenge has given rise to encoders for compressing digital video contents for transmission over an internet protocol infrastructure. However, transmitting compressed video color images still has an intrinsic limitation of high bandwidth consumption. Simple linear iterative clustering algorithm was applied for binary segmentation of video color images to circumvent the challenge of efficiently transmitting video contents. Compressed binary segmented images are generally fast to transmit and require lower bandwidth consumption as opposed to compressed video color images. However, since color images contain more useful information than binary image counterparts, evaluation of binary segmentation results was performed using the mean opinion score metric to determine user quality of experience of the transmitted video contents. The practical application of our method will lead to the development of a novel encoder that can deliver binary video contents faster, hence solving the bandwidth hiccup.


Object detection Salient object Sparse reconstruction User perception Video image Video sequence 



  1. 1.
    Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S (2010) Slic superpixelsGoogle Scholar
  2. 2.
    Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S (2012) SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34(11):2274–2282CrossRefGoogle Scholar
  3. 3.
    Achanta R, Süsstrunk S (2010) Saliency detection using maximum symmetric surround. In: Proceedings of Image processing (ICIP), 2010 17th IEEE international conference on. IEEE, 2653–2656Google Scholar
  4. 4.
    Ahn E, Bi L, Jung YH, Kim J, Li C, Fulham M, Feng DD (2015) Automated saliency-based lesion segmentation in dermoscopic images. In: Proceedings of Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE. IEEE, 3009–3012Google Scholar
  5. 5.
    Ahn E, Kim J, Bi L, Kumar A, Li C, Fulham M, Feng DD (2017) Saliency-based lesion segmentation via background detection in dermoscopic images. IEEE Journal of Biomedical and Health Informatics 21(6):1685–1693CrossRefGoogle Scholar
  6. 6.
    Al-azawi RJ, Abdulhameed AA, Ahmed HM (2017) A Robustness Segmentation Approach for Skin Cancer Image Detection Based on an Adaptive Automatic Thresholding Technique. American Journal of Intelligent Systems 7(4):107–112Google Scholar
  7. 7.
    Bhandari AK, Kumar A, Singh GK (2015) Tsallis entropy based multilevel thresholding for colored satellite image segmentation using evolutionary algorithms. Expert Syst Appl 42(22):8707–8730CrossRefGoogle Scholar
  8. 8.
    Borji A, Cheng M-M, Jiang H, Li J (2015) Salient object detection: A benchmark. IEEE Trans Image Process 24(12):5706–5722MathSciNetCrossRefGoogle Scholar
  9. 9.
    Brooks P, Hestnes B (2010) User measures of quality of experience: why being objective and quantitative is important. IEEE Netw 24(2)Google Scholar
  10. 10.
    Chen C, Li S, Wang Y, Qin H, Hao A (2017) Video Saliency Detection via SpatialTemporal Fusion and Low-Rank Coherency Diffusion. IEEE Trans Image Process 26(7):3156–3170MathSciNetCrossRefGoogle Scholar
  11. 11.
    Correa G, Assuncao P, Agostini L, da Silva Cruz LA (2012) Performance and computational complexity assessment of high-efficiency video encoders. IEEE Transactions on Circuits and Systems for Video Technology 22(12):1899–1909CrossRefGoogle Scholar
  12. 12.
    Dollár P, Appel R, Belongie S, Perona P (2014) Fast feature pyramids for object detection. IEEE Trans Pattern Anal Mach Intell 36(8):15321545CrossRefGoogle Scholar
  13. 13.
    Duanmu Z, Zeng K, Ma K, Rehman A, Wang Z (2016) A quality-of-experience index for streaming video. IEEE Journal of Selected Topics in Signal Processing 11(1):154–166CrossRefGoogle Scholar
  14. 14.
    El Abbadi NK, Miry AH (2014) Automatic segmentation of skin lesions using histogram thresholding. J Comput Sci 10(4):632–639Google Scholar
  15. 15.
    Eng ET, Kopylov M, Negro CJ, Dallaykan S, Rice WJ, Jordan KD, Kelley K, Carragher B, Potter C (2018) The impact of data reduction and lossy image formats on electron microscope images. bioRxiv, p.451427Google Scholar
  16. 16.
    Engelke U, Barkowsky M, Le Callet P, Zepernick H-J (2010) Modelling saliency awareness for objective video quality assessment. In: Proceedings of Quality of Multimedia Experience (QoMEX), 2010 Second International Workshop on. IEEE, 212–217Google Scholar
  17. 17.
    Fang Y, Lin W, Chen Z, Tsai C-M, Lin C-W (2014) A video saliency detection model in compressed domain. IEEE Transactions on Circuits and Systems for Video Technology 24(1):27–38CrossRefGoogle Scholar
  18. 18.
    Fei L, Deng Y, Mahadevan S (2015) Which is the best belief entropy. Journal of Latex Class Files 13(9):1–4Google Scholar
  19. 19.
    Fu H, Cao X, Tu Z (2013) Cluster-based co-saliency detection. IEEE Trans Image Process 22(10):3766–3778MathSciNetCrossRefGoogle Scholar
  20. 20.
    Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of Proceedings of the IEEE conference on computer vision and pattern recognition. 580–587Google Scholar
  21. 21.
    Goferman S, Zelnik-Manor L, Tal A (2012) Context-aware saliency detection. IEEE Trans Pattern Anal Mach Intell 34(10):1915–1926CrossRefGoogle Scholar
  22. 22.
    Gong B, Chao W-L, Grauman K, Sha F (2014) Diverse sequential subset selection for supervised video summarization. In: Proceedings of Advances in Neural Information Processing Systems. 2069–2077Google Scholar
  23. 23.
    Guo L, Cheng T, Huang Y, Zhao J, Zhang R (2017) Unsupervised video object segmentation by spatiotemporal graphical model. Multimed Tools Appl 76(1):1037–1053CrossRefGoogle Scholar
  24. 24.
    Guo C, Zhang L (2010) A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans Image Process 19(1):185–198MathSciNetCrossRefGoogle Scholar
  25. 25.
    Hadizadeh H, Bajic IV (2014) Saliency-aware video compression. IEEE Trans Image Process 23(1):19–33MathSciNetCrossRefGoogle Scholar
  26. 26.
    Han J, Zhou P, Zhang D, Cheng G, Guo L, Liu Z, Bu S, Wu J (2014) Efficient, simultaneous detection of multi-class geospatial targets based on visual saliency modeling and discriminative learning of sparse coding. ISPRS J Photogramm Remote Sens 89:37–48CrossRefGoogle Scholar
  27. 27.
    Helin H, Tolonen T, Ylinen O, Tolonen P, Näpänkangas J, Isola J (2018) Optimized JPEG 2000 compression for efficient storage of histopathological whole-Slide images. Journal of Pathology Informatics 9Google Scholar
  28. 28.
    Huang L-K, Wang M-JJ (1995) Image thresholding by minimizing the measures of fuzziness. Pattern Recogn 28(1):41–51CrossRefGoogle Scholar
  29. 29.
    Itti L (2004) Automatic foveation for video compression using a neurobiological model of visual attention. IEEE Trans Image Process 13(10):1304–1318CrossRefGoogle Scholar
  30. 30.
    Jiang B, Zhang L, Lu H, Yang C, Yang M-H (2013) Saliency detection via absorbing markov chain. In: Proceedings of Proceedings of the IEEE International Conference on Computer Vision. 1665–1672Google Scholar
  31. 31.
    Kou G, Peng Y, Wang G (2014) Evaluation of clustering algorithms for financial risk analysis using MCDM methods. Inf Sci 275:1–12CrossRefGoogle Scholar
  32. 32.
    Kumar V, Barthwal S, Kishore R, Saklani R, Sharma A, Sharma S (2016) Lossy Data Compression Using Logarithm. arXiv preprint arXiv:1604.02035Google Scholar
  33. 33.
    Lee S-H, Kang J-W, Kim C-S (2016) Compressed domain video saliency detection using global and local spatiotemporal features. J Vis Commun Image Represent 35:169–183CrossRefGoogle Scholar
  34. 34.
    Li Y, Fu K, Liu Z, Yang J (2015) Efficient saliency-model-guided visual co-saliency detection. IEEE Signal Processing Letters 22(5):588–592CrossRefGoogle Scholar
  35. 35.
    Li F, Kim T, Humayun A, Tsai D, Rehg JM (2013a) Video segmentation by tracking many figure-ground segments. In: Proceedings of Proceedings of the IEEE International Conference on Computer Vision. 2192–2199Google Scholar
  36. 36.
    Li J, Levine MD, An X, Xu X, He H (2013b) Visual saliency based on scale-space analysis in the frequency domain. IEEE Trans Pattern Anal Mach Intell 35(4):996–1010CrossRefGoogle Scholar
  37. 37.
    Li B, Sullivan GJ, Xu J (2012) Comparison of compression performance of HEVC working draft 5 with AVC high profile. document JCTVC-H0360Google Scholar
  38. 38.
    Lu H, Li X, Zhang L, Ruan X, Yang M-H (2016) Dense and sparse reconstruction error based saliency descriptor. IEEE Trans Image Process 25(4):1592–1603MathSciNetCrossRefGoogle Scholar
  39. 39.
    Luo Q, Geng Y, Liu J, Li W (2014) Saliency and texture information based full reference quality metrics for video QoE assessment. In: Proceedings of Network Operations and Management Symposium (NOMS), 2014 IEEE. IEEE, 1–6Google Scholar
  40. 40.
    Margolin R, Tal A, Zelnik-Manor L (2013) What makes a patch distinct? In: Proceedings of Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1139–1146Google Scholar
  41. 41.
    Perazzi F, Pont-Tuset J, McWilliams B, Van Gool L, Gross M, Sorkine-Hornung A (2016) A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 724–732Google Scholar
  42. 42.
    Podder PK, Paul M, Murshed M (2016) Fast mode decision in the HEVC video coding standard by exploiting region with dominated motion and saliency features. PLoS One 11(3):e0150673CrossRefGoogle Scholar
  43. 43.
    Pont-Tuset J, Marques F (2016) Supervised evaluation of image segmentation and object proposal techniques. IEEE Trans Pattern Anal Mach Intell 38(7):1465–1478CrossRefGoogle Scholar
  44. 44.
    Popovic A, De la Fuente M, Engelhardt M, Radermacher K (2007) Statistical validation metric for accuracy assessment in medical image segmentation. Int J Comput Assist Radiol Surg 2(3):169–181CrossRefGoogle Scholar
  45. 45.
    Potapov D, Douze M, Harchaoui Z, Schmid C (2014) Category-specific video summarization. In: Proceedings of European conference on computer vision. Springer, 540555Google Scholar
  46. 46.
    Ren CY, Reid I (2011) gSLIC: a real-time implementation of SLIC superpixel segmentation. University of Oxford, Department of Engineering, Technical ReportGoogle Scholar
  47. 47.
    Samanthula BK, Jiang W (2016) Secure multiset intersection cardinality and its application to jaccard coefficient. IEEE Transactions on Dependable and Secure Computing 13(5):591–604CrossRefGoogle Scholar
  48. 48.
    Scharr H, Minervini M, French AP, Klukas C, Kramer DM, Liu X, Luengo I, Pape J-M, Polder G, Vukadinovic D (2016) Leaf segmentation in plant phenotyping: a collation study. Mach Vis Appl 27(4):585–606CrossRefGoogle Scholar
  49. 49.
    Shattuck DW, Prasad G, Mirza M, Narr KL, Toga AW (2009) Online resource for validation of brain segmentation methods. NeuroImage 45(2):431–439CrossRefGoogle Scholar
  50. 50.
    Shen J, Du Y, Wang W, Li X (2014) Lazy random walks for superpixel segmentation. IEEE Trans Image Process 23(4):1451–1462MathSciNetCrossRefGoogle Scholar
  51. 51.
    Souly N, Shah M (2016) Visual saliency detection using group lasso regularization in videos of natural scenes. Int J Comput Vis 117(1):93–110MathSciNetCrossRefGoogle Scholar
  52. 52.
    Taha AA, Hanbury A (2015) Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med Imaging 15(1):29CrossRefGoogle Scholar
  53. 53.
    Tasse FP, Kosinka J, Dodgson NA (2016) Quantitative analysis of saliency models. In: Proceedings of SIGGRAPH ASIA 2016 Technical Briefs. ACM, 19Google Scholar
  54. 54.
    Tsai, D, Flagg M, Rehg J (2010) Motion coherent tracking with multi-label mrf optimization, algorithmsGoogle Scholar
  55. 55.
    Udupa JK, LeBlanc VR, Zhuge Y, Imielinska C, Schmidt H, Currie LM, Hirsch BE, Woodburn J (2006) A framework for evaluating image segmentation algorithms. Comput Med Imaging Graph 30(2):75–87CrossRefGoogle Scholar
  56. 56.
    Wang W, Shen J, Shao L (2015) Consistent video saliency using local gradient flow optimization and global refinement. IEEE Trans Image Process 24(11):41854196MathSciNetzbMATHGoogle Scholar
  57. 57.
    Wright J, Yang AY, Ganesh A, Sastry SS, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31(2):210–227CrossRefGoogle Scholar
  58. 58.
    Zhang D, Meng D, Han J (2017a) Co-saliency detection via a self-paced multiple-instance learning framework. IEEE Trans Pattern Anal Mach Intell 39(5):865–878CrossRefGoogle Scholar
  59. 59.
    Zhang J, Sclaroff S (2016) Exploiting surroundedness for saliency detection: a Boolean map approach. IEEE Trans Pattern Anal Mach Intell 38(5):889902Google Scholar
  60. 60.
    Zhang L, Yang C, Lu H, Ruan X, Yang M-H (2017b) Ranking saliency. IEEE Trans Pattern Anal Mach Intell 39(9):1892–1904CrossRefGoogle Scholar
  61. 61.
    Zheng K, Zhang X, Zheng Q, Xiang W, Hanzo L (2015) Quality-of-experience assessment and its application to video services in LTE networks. IEEE Wirel Commun 22(1):70–78CrossRefGoogle Scholar
  62. 62.
    Qin C, Zhang G, Zhou Y, Tao W, Cao Z (2014) Integration of the saliency-based seed extraction and random walks for image segmentation. Neurocomputing 129:378–391Google Scholar
  63. 63.
    Bylinskii Z, Recasens A, Borji A, Oliva A, Torralba A, Durand F (2016) October. Where should saliency models look next?. In European Conference on Computer Vision (pp. 809–824). Springer, ChamGoogle Scholar
  64. 64.
    Gygli M, Grabner H, Riemenschneider H, Nater F, Van Gool L (2013) The interestingness of images. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1633–1640)Google Scholar
  65. 65.
    Wei L, Wang F, Li X, Wu F, Xiao J (2017) Graph-theoretic spatiotemporal context modeling for video saliency detection. arXiv preprint arXiv:1707.07815, 1–5Google Scholar
  66. 66.
    El Abbadi NK, Miry AH (2014) Automatic segmentation of skin lesions using histogram thresholding. Journal of Computer Science 10(4):632–639Google Scholar
  67. 67.
    Rabbani, T., Van Den Heuvel, F. and Vosselmann, G. 2006. Segmentation of point clouds using smoothness constraint. International archives of photogrammetry, remote sensing and spatial information sciences 36(5):248–253Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.ICT and Society Research GroupDurban University of TechnologyDurbanSouth Africa

Personalised recommendations