Sencogi Spatio-Temporal Saliency: A New Metric for Predicting Subjective Video Quality on Mobile Devices

  • Maria Laura MeleEmail author
  • Damon Millar
  • Christiaan Erik Rijnders
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10902)


Objective Video Quality Assessment (VQA) is often used to predict users visual perception of video quality. In the literature, the performance evaluation of objective measures is based on benchmark subjective scores of perceived quality. This paper shows the evaluation of an algorithmic measure on videos presented on mobile devices. The VQA measure is called Sencogi Spatio-Temporal Saliency Metric (Sencogi-STSM), and it uses a spatio-temporal saliency to model subjective perception of video quality. Since STSM was previously validated with a subjective test conducted on laptop computers, the goal of this work was to verify whether the measure is able to significantly predict users’ perception of video quality also on mobile devices. Results show that, compared to the standard VQA metrics, only Sencogi-STSM is able to significantly predict subjective DMOS. This paper describes Sencogi-STSM’s biologically plausible model, its performance evaluation and the comparison with the most commonly used objective VQA metrics.


Video quality perception Computer vision Spatio-temporal saliency Objective video quality assessment 


  1. 1.
    Chikkerur, S., Sundaram, V., Reisslein, M., Karam, L.J.: Objective video quality assessment methods: a classification, review, and performance comparison. IEEE Trans. Broadcast. 57, 165–182 (2011)CrossRefGoogle Scholar
  2. 2.
    BT.500: Methodology for the subjective assessment of the quality of television pictures (n.d.). Accessed 9 Oct 2017
  3. 3.
    Brunnstrom, K., Hands, D., Speranza, F., Webster, A.: VQEG validation and ITU standardization of objective perceptual video quality metrics [Standards in a Nutshell]. IEEE Sign. Process. Mag. 26, 96–101 (2009)CrossRefGoogle Scholar
  4. 4.
    Staelens, N., Moens, S., Van den Broeck, W., Marien, I., Vermeulen, B., Lambert, P., Van de Walle, R., Demeester, P.: Assessing quality of experience of IPTV and video on demand services in real-life environments. IEEE Trans. Broadcast. 56(4), 458–466 (2010)CrossRefGoogle Scholar
  5. 5.
    Seshadrinathan, K., Soundararajan, R., Bovik, A.C., Cormack, L.K.: A subjective study to evaluate video quality assessment algorithms. In: Human Vision and Electronic Imaging, vol. 7527, p. 75270 (2010)Google Scholar
  6. 6.
    Wang, Z., Bovik, A.C.: Modern image quality assessment. Synth. Lect. Image Video Multimedia Process. 2, 1–156 (2006)CrossRefGoogle Scholar
  7. 7.
    Duchowski, A.: Eye Tracking Methodology, Theory and Practice, vol. 373. Springer Science & Business Media, New York (2007). Scholar
  8. 8.
    Itti, L., Koch, C.: A saliency-based search mechanism for overt and covert shifts of visual attention. Vis. Res. 40, 1489–1506 (2000)CrossRefGoogle Scholar
  9. 9.
    Wolfe, J.M.: Visual search in continuous, naturalistic stimuli. Vis. Res. 34, 1187–1195 (1994)CrossRefGoogle Scholar
  10. 10.
    Mitchell, R.K., Agle, B.R., Wood, D.J.: Toward a theory of stakeholder identification and salience: defining the principle of who and what really counts. Acad. Manag. Rev. 22(4), 853–886 (1997)CrossRefGoogle Scholar
  11. 11.
    Zhang, L., Shen, Y., Li, H.: VSI: a visual saliency-induced index for perceptual image quality assessment. IEEE Trans. Image Process. 23, 4270–4281 (2014)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Larson, E.C., Chandler, D.M.: Unveiling relationships between regions of interest and image fidelity metrics. In: Visual Communications and Image Processing 2008 (2008).
  13. 13.
    Zhang, W., Borji, A., Wang, Z., Le Callet, P., Liu, H.: The application of visual saliency models in objective image quality assessment: a statistical evaluation. IEEE Trans. Neural Netw. Learn. Syst. 27, 1266–1278 (2016)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Achanta, R., Estrada, F., Wils, P., Süsstrunk, S.: Salient region detection and segmentation. In: Gasteratos, A., Vincze, M., Tsotsos, John K. (eds.) ICVS 2008. LNCS, vol. 5008, pp. 66–75. Springer, Heidelberg (2008). Scholar
  15. 15.
    Hou, X., Zhang, L.: Saliency detection: a spectral residual approach. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition (2007).
  16. 16.
    Harel, J., Koch, C., Perona, P.: Graph-based visual saliency. In: Advances in Neural Information Processing Systems, pp. 545–552 (2007)Google Scholar
  17. 17.
    Itti, L., Koch, C.: A saliency-based search mechanism for overt and covert shifts of visual attention. Vis. Res. 40, 1489–1506 (2000)CrossRefGoogle Scholar
  18. 18.
    Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)CrossRefGoogle Scholar
  19. 19.
    Li, J., Levine, M., An, X., He, H.: Saliency detection based on frequency and spatial domain analyses. In: Proceedings of the British Machine Vision Conference 2011 (2011).
  20. 20.
    Ma, Y.-F., Zhang, H.-J.: Contrast-based image attention analysis by using fuzzy growing. In: Proceedings of the Eleventh ACM International Conference on Multimedia, MULTIMEDIA 2003 (2003).
  21. 21.
    Bian, P., Zhang, L.: Biological plausibility of spectral domain approach for spatiotemporal visual saliency. In: Köppen, M., Kasabov, N., Coghill, G. (eds.) ICONIP 2008. LNCS, vol. 5506, pp. 251–258. Springer, Heidelberg (2009). Scholar
  22. 22.
    Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient region detection. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition (2009)Google Scholar
  23. 23.
    Yubing, T., Cheikh, F.A., Guraya, F.F.E., Konik, H., Trémeau, A.: A spatiotemporal saliency model for video surveillance. Cogn. Comput. 3, 241–263 (2011)CrossRefGoogle Scholar
  24. 24.
    Arvanitidou, M.G., Sikora, T.: Motion saliency for spatial pooling of objective video quality metrics (2012)Google Scholar
  25. 25.
    Cogisen Homepage. Accessed 22 Jan 2018
  26. 26.
    Mele, M.L., Millar, D., Rijnders, C.E.: The web-based subjective quality assessment of an adaptive image compression plug-in. In: 1st International Conference on Human Computer Interaction Theory and Applications, HUCAPP, Porto, Portugal (2017)Google Scholar
  27. 27.
    Mele, M.L., Millar, D., Rijnders, C.E.: Using spatio-temporal saliency to predict subjective video quality: a new high-speed objective assessment metric. In: Kurosu, M. (ed.) HCI 2017. LNCS, vol. 10271, pp. 353–368. Springer, Cham (2017). Scholar
  28. 28.
    Redi, J., Liu, H., Zunino, R., Heynderickx, I.: Interactions of visual attention and quality perception. In: Human Vision and Electronic Imaging XVI (2011).
  29. 29.
    SurveyGizmo Homepage. Accessed 22 Jan 2018

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Maria Laura Mele
    • 1
    • 2
    • 3
    Email author
  • Damon Millar
    • 1
  • Christiaan Erik Rijnders
    • 1
  1. 1.COGISEN Engineering CompanyRomeItaly
  2. 2.Department of Philosophy, Social and Human Sciences and EducationUniversity of PerugiaPerugiaItaly
  3. 3.ECONA, Interuniversity Centre for Research on Cognitive Processing in Natural and Artificial SystemsSapienza University of RomeRomeItaly

Personalised recommendations