Advertisement

Subjective QoE Models

  • Vlado MenkovskiEmail author
Chapter
  • 274 Downloads
Part of the Springer Theses book series (Springer Theses)

Abstract

The subjective QoE methods are concerned with quantifying the experienced quality of the users. Because these methods measure the subjective quality in an unmediated manner, their measurements are commonly used as ‘ground truth’ for evaluation of other methods [1]. In an end-to-end approach for QoE management, subjective estimation and subjective models are of key importance. It allows for evaluating the performance of the system and enables the loop back signal that closes the control loop. In this chapter these subjective QoE methods are discussed. QoE estimation methods based on rating, Just Noticable Differences and difference scalling are presented as well as their computational counterparts that enable building of the QoE models.

Keywords

Psychometric Curve Difference Scaling Method Differential MOS (DMOS) Video Quality Experts Group (VQEG) 2AFC Test 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    S. Winkler, P. Mohandas, The evolution of video quality measurement: from PSNR to hybrid metrics. IEEE Trans. Broadcast. 54(3), 660668 (2008). http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4550731
  2. 2.
    G. Exarchakos, V. Menkovski, L. Druda, A. Liotta, Network analysis on Skype end-to-end video quality. Int. J. Pervasive Comput. Commun. 11(1) (2015). http://www.emeraldinsight.com/doi/abs/10.1108/IJPCC-08-2014-0044
  3. 3.
    G. Exarchakos, L. Druda, V. Menkovski, P. Bellavista, A. Liotta, Skype resilience to high motion videos. Int. J. Wavelets Multiresolut. Inf. Process. 11(3) (2013). http://dx.doi.org/10.1142/S021969131350029X
  4. 4.
    F. Agboma, A. Liotta, Quality of experience management in mobile content delivery systems. J. Telecommun. Syst. special issue on the Quality of Experience issues in Multimedia Provision, 49(1), 85-98 (2012). doi: 10.1007/s11235-010-9355-6
  5. 5.
    V. Menkovski, G. Exarchakos, A. Liotta, A. Cuadra Sánchez, Quality of experience models for multimedia streaming. Int. J. Mobile Comput. Multimedia Commun. 2(4), 1–20 (2010). http://www.igi-global.com/ijmcmc/. doi: 10.4018/jmcmc.2010100101. ISSN: 1937-9412
  6. 6.
    M. Alhaisoni, A. Liotta, Characterization of signalling and traffic in Joost. J. P2P Networking Appl. special issue on Modelling and Applications of Computational P2P, 2, 75–83 (2009). doi: 10.1007/s12083-008-0015-5, ISSN: 1936-6450
  7. 7.
    M. Torres Vega, E. Giordano, D.C. Mocanu, D. Tjondronegoro, A. Liotta, Cognitive no-reference video quality assessment for mobile streaming services, in Proceedings of the 7th International Workshop on Quality of Multimedia Experience, Messinia, Greece, 26–29 May 2015 (IEEE). http://www.qomex.org
  8. 8.
    A. Liotta, L. Druda, G. Exarchakos, V. Menkovski, Quality of experience management for video streams: the case of Skype, in Proceedings of the 10th International Conference on Advances in Mobile Computing and Multimedia, Bali, Indonesia, 3–5 Dec 2012 (ACM). http://dx.doi.org/10.1145/2428955.2428977
  9. 9.
    V. Menkovski, A. Liotta, Intelligent control for adaptive video streaming, in Proceedings of the International Conference on Consumer Electronics, Las Vegas, US, 11–14 Jan 2013 (IEEE). http://dx.doi.org/10.1109/ICCE.2013.6486825
  10. 10.
    A. Liotta, D. Constantin Mocanu, V. Menkovski, L. Cagnetta, G. Exarchakos, Instantaneous video quality assessment for lightweight devices, in Proceedings of the 11th International Conference on Advances in Mobile Computing and Multimedia, Vienna, Austria, 2–4 Dec 2013 (ACM). http://dx.doi.org/10.1145/2536853.2536903
  11. 11.
    D.C. Mocanu, A. Liotta, A. Ricci, M. Torres Vega, G. Exarchakos, When does lower bitrate give higher quality in modern video services?, in Proceedings of the 2nd IEEE/IFIP International Workshop on Quality of Experience Centric Management, Krakow, Poland, 9 May 2014 (IEEE). http://dx.doi.org/10.1109/NOMS.2014.6838400
  12. 12.
    D.C. Mocanu, G. Exarchakos, A. Liotta, Deep learning for objective quality assessment of 3D images, in Proceedings of IEEE International Conference on Image Processing, Paris, France, 27–30 Oct 2014 (IEEE)Google Scholar
  13. 13.
    M. Torres Vega, S. Zou, D. Constantin Mocanu, E. Tangdiongga, A.M.J. Koonen, A. Liotta, End-to-end performance evaluation in high-speed wireless networks, in Proceedings of the 10th International Conference on Network and Service Management, Rio de Janeiro, Brazil, 17–21 Nov 2014 (IEEE)Google Scholar
  14. 14.
    D. Constantin Mocanu, G. Santandrea, W. Cerroni, F. Callegati, A. Liotta, Network performance assessment with quality of experience benchmarks, in Proceedings of the 10th International Conference on Network and Service Management, Rio de Janeiro, Brazil, 17–21 Nov 2014 (IEEE)Google Scholar
  15. 15.
    D. Constantin Mocanu, G. Exarchakos, H.B. Ammar, A. Liotta, Reduced reference image quality assessment via Boltzmann machines, in Proceedings of the 3rd IEEE/IFIP IM 2015 International Workshop on Quality of Experience Centric Management, Ottawa, Canada, 11–15 May 2015 (IEEE)Google Scholar
  16. 16.
    M. Torres Vega, D. Constantin Mocanu, R. Barresi, G. Fortino, A. Liotta, Cognitive streaming on android devices, in Proceedings of the 1st IEEE/IFIP IM 2015 International Workshop on Cognitive Network and Service Management, Ottawa, Canada, 11–15 May 2015 (IEEE). http://www.cogman.org
  17. 17.
    J. Okyere-Benya, M. Aldiabat, V. Menkovski, G. Exarchakos, A. Liotta, Video quality degradation on IPTV networks, in Proceedings of International Conference on Computing, Networking and Communications, Maui, Hawaii, USA, 30 Jan– 2 Feb 2012 (IEEE)Google Scholar
  18. 18.
    G. Exarchakos, V. Menkovski, A. Liotta, Can Skype be used beyond video calling?, in Proceedings of the 9th International Conference on Advances in Mobile Computing and Multimedia, Ho Chi Minh City, Vietnam, 5–7 Dec 2011 (ACM)Google Scholar
  19. 19.
    V. Menkovski, G. Exarchakos, A. Cuadra-Sanchez, A. Liotta, Estimations and remedies for quality of experience in multimedia streaming, in Proceedings of the 3rd International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services, Nice, France, 22–27 Aug 2010 (IEEE)Google Scholar
  20. 20.
    V. Menkovski, G. Exarchakos, A. Liotta, Machine learning approach for quality of experience aware networks, in Proceedings of Computational Intelligence in Networks and Systems, Thessaloniki, Greece, 24–26 Nov 2010 (IEEE)Google Scholar
  21. 21.
    F. Agboma, A. Liotta, User-centric assessment of mobile content delivery, in Proceedings of the 4th International Conference on Advances in Mobile Computing and Multimedia, Yogyakarta, Indonesia, 4–6 Dec 2006Google Scholar
  22. 22.
    F. Agboma, A. Liotta, Managing the user’s quality of experience, in Proceedings of the second IEEE/IFIP International Workshop on Business-driven IT Management (BDIM 2007), Munich, Germany, 21 May 2007 (IEEE)Google Scholar
  23. 23.
    F. Agboma, M. Smy, A. Liotta, QoE analysis of a peer-to-peer television system, in Proceedings of the International Conference on Telecommunications, Networks and Systems, Amsterdam, Netherlands, 22–24 July 2008Google Scholar
  24. 24.
    F. Agboma, A. Liotta, QoE-aware QoS management, in Proceedings of the 6th International Conference on Advances in Mobile Computing and Multimedia, Linz, Austria, 24–26 Nov 2008Google Scholar
  25. 25.
    V. Menkovski, A. Oredope, A. Liotta, A. Cuadra-Sanchez, Optimized online learning for QoE prediction, in Proceedings of the 21st Benelux Conference on Artificial Intelligence, Eindhoven, The Netherlands, 29–30 Oct 2009, pp. 169–176. http://wwwis.win.tue.nl/bnaic2009/proc.html. ISSN: 1568-7805
  26. 26.
    V. Menkovski, A. Oredope, A. Liotta, A. Cuadra-Sanchez, Predicting quality of experience in multimedia streaming, in Proceedings of the 7th International Conference on Advances in Mobile Computing and Multimedia, Dec 2009 (ACM). http://dl.acm.org/citation.cfm?id=1821766. ISBN: 978-1-60558-659-5
  27. 27.
    V. Menkovski, G. Exarchakos, A. Cuadra-Sanchez, A. Liotta, Measuring quality of experience on a commercial mobile TV platform, in Proceedings of the 2nd International Conference on Advances in Multimedia, Athens, Greece, 13–19 June 2010 (IEEE)Google Scholar
  28. 28.
    V. Menkovski, G. Exarchakos, A. Liotta, Online QoE prediction, in Proceedings of the 2nd IEEE International Workshop on Quality of Multimedia Experience, Trondheim, Norway, 21–23 June 2010 (IEEE)Google Scholar
  29. 29.
    V. Menkovski, G. Exarchakos, A. Liotta, Online learning for quality of experience management, in Proceedings of the Annual Machine Learning Conference of Belgium and The Netherlands, Leuven, Belgium, 27–28 May 2010. http://dtai.cs.kuleuven.be/events/Benelearn2010/submissions/benelearn2010_submission_20.pdf
  30. 30.
    V. Menkovski, G. Exarchakos, A. Liotta, The value of relative quality in video delivery. J. Mobile Multimedia 7(3), 151162 (2011). http://dl.acm.org/citation.cfm?id=2230537
  31. 31.
    F. Agboma, A. Liotta, Addressing user expectations in mobile content delivery. Mob. Inf. Syst. 3(3, 4), 153164 (2007). http://dl.acm.org/citation.cfm?id=1376820.1376823
  32. 32.
    W.H. Ehrenstein, A. Ehrenstein, Psychophysical methods, in Modern Techniques in Neuroscience Research, 1999, pp. 1211–1241. http://uni-leipzig.de/~isp/isp/history/texts/PSYPHY-M.PDF
  33. 33.
    V. Menkovski, G. Exarchakos, A. Liotta, Tackling the sheer scale of subjective QoE, in Proceedings of 7th International ICST Mobile Multimedia Communications Conference, Cagliari, Italy, 5–7 Sept 2011, vol. 29, pp. 1–15 (Springer, Lecture Notes of ICST, 2012). http://www.springerlink.com/content/p1443m265r25756x/. doi: 10.1007/978-3-642-30419-4_1
  34. 34.
    ITU, 500-11, Methodology for the subjective assessment of the quality of television pictures, recommendation ITU-R BT. 500-11, ITU Telecom. Standardization Sector of ITU (2002)Google Scholar
  35. 35.
    F. De Simone, M. Naccari, M. Tagliasacchi, F. Dufaux, S. Tubaro, T. Ebrahimi, Subjective assessment of H.264/AVC video sequences transmitted over a noisy channel, in International Workshop on Quality of Multimedia Experience, QoMEx 2009. IEEE 2009, pp. 204–209 (2009)Google Scholar
  36. 36.
    Q. Huynh-Thu, M. Ghanbari, Modelling of spatio-temporal interaction for video quality assessment. Signal Process.: Image Commun. 25(7), 535–546 (2010)Google Scholar
  37. 37.
    K. Seshadrinathan, R. Soundararajan, A. Bovik, L. Cormack, Study of subjective and objective quality assessment of video. IEEE Trans. Image Process. 19(6), 1427–1441 (2010)MathSciNetCrossRefGoogle Scholar
  38. 38.
    I. Recommendation, 910, Subjective video quality assessment methods for multimedia applications, recommendation ITU-T P. 910, ITU Telecom. Standardization Sector of ITU (1999)Google Scholar
  39. 39.
    M. Barkowsky, M. Pinson, R. Pépion, P. Le Callet, Analysis of freely available subjective dataset for HDTV including coding and transmission distortions, in Fifth International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM-10) (2010)Google Scholar
  40. 40.
    S. Stevens, On the psychophysical law. Psychol. Rev. 64(3), 153 (1957)CrossRefGoogle Scholar
  41. 41.
    D. Krantz, A theory of context effects based on cross-context matching. J. Math. Psychol. 5(1), 1–48 (1968)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    D. Krantz, A theory of magnitude estimation and cross-modality matching. J. Math. Psychol. 9(2), 168–199 (1972)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    D. Krantz, R. Luce, P. Suppes, A. Tversky, Foundations of Measurement Volume I: Additive and Polynomial Representations, vol. 1 (Dover Publications, New York, 2006)Google Scholar
  44. 44.
    R. Shepard, On the status of ‘direct’ psychophysical measurement. Minn. Stud. Philos. Sci. 9, 441–490 (1978)Google Scholar
  45. 45.
    R. Shepard, Psychological relations and psychophysical scales: on the status of direct psychophysical measurement. J. Math. Psychol. 24(1), 21–57 (1981)MathSciNetCrossRefzbMATHGoogle Scholar
  46. 46.
    A. Watson, Proposal: measurement of a JND scale for video quality, IEEE G-2.1. 6 Subcommittee on Video Compression Measurements (2000)Google Scholar
  47. 47.
    S. Winkler, On the properties of subjective ratings in video quality experiments, in International Workshop on Quality of Multimedia Experience, QoMEx 2009. IEEE 2009, pp. 139–144 (2009)Google Scholar
  48. 48.
    V. Menkovski, A. Liotta, Adaptive psychometric scaling for video quality assessment. Signal Process.: Image Commun. 27(8), 788–799 (2012)Google Scholar
  49. 49.
    V.Q.E. Group et al., Report on the validation of video quality models for high definition video content, Technical report (2010)Google Scholar
  50. 50.
    P. Corriveau, C. Gojmerac, B. Hughes, L. Stelmach, All subjective scales are not created equal: the effects of context on different scales. Signal Process. 77(1), 1–9 (1999)CrossRefzbMATHGoogle Scholar
  51. 51.
    P. Brooks, B. Hestnes, User measures of quality of experience: why being objective and quantitative is important. IEEE Network 24(2), 8–13 (2010)CrossRefGoogle Scholar
  52. 52.
    G.T. Fechner, Elements of Psychophysics (Holt, Rinehart and Winston, New York, 1966)Google Scholar
  53. 53.
    J.D. McCarthy, M.A. Sasse, D. Miras, Sharp or smooth?: comparing the effects of quantization vs. frame rate for streamed video, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI’04 (ACM, New York, NY, USA, 2004), p. 535542. http://doi.acm.org/10.1145/985692.985760
  54. 54.
    E.H. Weber, E.H. Weber On the Tactile Senses (Psychology Press, Hove, 1996)Google Scholar
  55. 55.
    A. Watson, L. Kreslake, Measurement of visual impairment scales for digital video, in Human Vision, Visual Processing, and Digital Display. Proceedings. SPIE 4299 (2001)Google Scholar
  56. 56.
    L. L. Thurstone, The Measurement of Values, vol. vii (University of Chicago Press, Oxford, England, 1959)Google Scholar
  57. 57.
    L. Maloney, J. Yang, Maximum likelihood difference scaling. J. Vision 3(8) (2003)Google Scholar
  58. 58.
    C. Charrier, L. Maloney, H. Cherifi, K. Knoblauch, Maximum likelihood difference scaling of image quality in compression-degraded images. JOSA A 24(11), 3418–3426 (2007)CrossRefGoogle Scholar
  59. 59.
    K. Knoblauch, L. Maloney et al., MLDS: Maximum likelihood difference scaling in R. J. Stat. Softw. 25(2), 1–26 (2008)CrossRefGoogle Scholar
  60. 60.
    D. Green, J. Swets et al., Signal Detection Theory and Psychophysics, vol. 1974 (Wiley, New York, 1966)Google Scholar
  61. 61.
    H. Bergsten, JavaServer Pages, 3rd edn. (O’Reilly Media Inc., Sebastopol, 2003)Google Scholar
  62. 62.
    JW player: Overview | LongTail video | home of the JW player. http://www.longtailvideo.com/players
  63. 63.
    L. Aimar, L. Merritt, E. Petit, M. Chen, J. Clay, M. Rullgrd, C. Heine, A. Izvorski, VideoLAN—x264, the best H.264/AVC encoder. http://www.videolan.org/developers/x264.html
  64. 64.
    R. Team et al., R: A language and environment for statistical computing (R Foundation Statistical Computing, Vienna, 2008)Google Scholar
  65. 65.
    K. Knoblauch, psyphy: Functions for analyzing psychophysical data in R, R package version 0.0-5. http://CRAN.R-project.org/package=psyphy (2007)
  66. 66.
    B. Efron, R. Tibshirani, An Introduction to the Bootstrap (Chapman & Hall/CRC Monographs on Statistics and Applied Probability) (Chapman and Hall/CRC, New York, 1994)Google Scholar
  67. 67.
    F. Wichmann, N. Hill, The psychometric function: I. Fitting, sampling, and goodness of fit. Attention Percept. Psychophys. 63(8), 1293–1313 (2001)CrossRefGoogle Scholar
  68. 68.
    V. Menkovski, G. Exarchakos, A. Liotta, Adaptive testing for video quality assessment, in Proceedings of Quality of Experience for Multimedia Content Sharing, Lisbon, Portugal, 29 June 2011 (ACM)Google Scholar
  69. 69.
    S. Kullback, R.A. Leibler, On information and sufficiency. Ann. Math. Stat. 22(1), 79–86 (1951). http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoms/1177729694
  70. 70.
    K. Eckschlager, K. Danzer, Information Theory in Analytical Chemistry (Wiley, New York, 1994)Google Scholar
  71. 71.
    M. Alhaisoni, A. Liotta, M. Ghanbari, Scalable P2P video streaming. Int. J. Bus. Data Commun. Networking 6(3), 49–65 (2010). doi: 10.4018/jbdcn.2010070103. ISSN: 1548-0631
  72. 72.
    M. Alhaisoni, M. Ghanbari, A. Liotta, Localized multistreams for P2P streaming. Int. J. Dig. Multimedia Broadcast. Article ID 843574, 12 pp. Hindawi 2010 (2010). doi: 10.1155/2010/843574
  73. 73.
    M. Alhaisoni, A. Liotta, M. Ghanbari, Resource-awareness and trade-off optimization in P2P video streaming. Int. J. Adv. Media Commun. special issue on High-Quality Multimedia Streaming in P2P Environments 4(1), 59–77 (2010). doi: 10.1504/IJAMC.2010.030005. ISSN: 1741-8003
  74. 74.
    A. Liotta, Farewell to deterministic networks, in Proceedings of the 19th IEEE Symposium on Communications and Vehicular Technology in the Benelux, Eindhoven, The Netherlands, 16 Nov 2012 (IEEE). http://dx.doi.org/10.1109/SCVT.2012.6399413

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Philips ResearchEindhovenThe Netherlands

Personalised recommendations