Multimedia Tools and Applications

, Volume 55, Issue 2, pp 185–225 | Cite as

A Hybrid Method for Quality Evaluation in the Context of Use for Mobile (3D) Television



Controlled psychoperceptual quality evaluation experiments are used to assess the excellence of produced audiovisual quality from fundamental signal processing algorithms to consumer services. When compromising produced quality for consumer services, used in dynamic and heterogeneous mobile contexts, the ecological validity of conventional quality evaluation methods can be questioned. The goal of this paper is to develop a method for evaluating the experienced multimedia quality in the context of use. We conducted three studies where the quality of mobile 2D and 3D television was assessed in three different field contexts, one simulated context and one controlled laboratory situation when audio-video compression and transmission parameters were varied. We propose a hybrid method for the design, data-collection and analysis of the experiments in the contexts of use. Its novelty is to complement conventional quantitative quality evaluation with concrete tools to identify factors that surround the assessment in the context. The methodological framework is part of our long-term aim to measure and understand the user-centered quality of experience.


Multimedia Experienced Audiovisual quality Context Context of use Evaluation 



This work is supported by the European Commission within the ICT program of FP7 under Grant 216503 with the acronym MOBILE3DTV ( and the UCIT graduate school. The authors wish to thank Dr. Miska Hannuksela and Dr. Atanas Gotchev for their comments and Cinovent, Red Star Studio, Stereoscape and the Centre of Computer Graphics and Visualization from the University of West Bohemia for providing content for the experiments.


  1. 1.
    Barnard L et al (2007) Capturing the effects of context on human performance in mobile computing systems. Pers Ubiquit Comput 11:81–96CrossRefGoogle Scholar
  2. 2.
    Beresford K, et al. (2006) Contextual effects on sound quality judgements: listening room and automotive environments. AES 120th Convention:6648Google Scholar
  3. 3.
    Bernhaupt R, Mihalic K, Obrist M (2008) Usability evaluation methods for mobile applications. In: Lumsden J (ed) Handbook Res User Interface Des Evaluation for Mobile Technology XLIV:745–758Google Scholar
  4. 4.
    Boev A et al (2009) Classification and simulation of stereoscopic artifacts in mobile 3DTV content. Proc SPIE Stereoscopic Displays and Applications XX:7237(72371F)Google Scholar
  5. 5.
    Bradley NA, Dunlop MD (2005) Toward a multidisciplinary model of context to support context-aware computing. Hum Comput Interact 20(4):403–446CrossRefGoogle Scholar
  6. 6.
    Brewster S (2002) Overcoming the lack of screen space on mobile computers. Pers Ubiq Comput 6:188–205CrossRefGoogle Scholar
  7. 7.
    Buchinger S, Kriglstein S, Hlavacs H (2009) A comprehensive view on user studies: survey and open issues for mobile TV. Proc EuroITV '09:179-188Google Scholar
  8. 8.
    Chen T, Yesilada Y, Harper S (2008) RIAM D2.6: How do people use their mobile phones while they are walking? A field study of real-world small device usage. Research Report School of Computer Science, Univ of Manchester.
  9. 9.
    Consolvo S et al (2007) Conducting in situ evaluations for and with ubiquitous computing technologies. Int J Hum Comput Interact 22(1):107–122Google Scholar
  10. 10.
    Cook T, Campbell D (1979) Quasi-experimentation: design & analysis issues for field settings. Houghton Mifflin, New YorkGoogle Scholar
  11. 11.
    Coolican H (2004) Research methods and statistics in psychology, 4th edn. Arrowsmith, LondonGoogle Scholar
  12. 12.
    Cui Y, Chipchase J, Jung Y (2006) Personal television: a qualitative study of mobile TV users. Lect Notes Comput Sci 4471:195–204CrossRefGoogle Scholar
  13. 13. Accessed 15 December 2009
  14. 14.
    European Broadcasting Union (2003) Subjective listening tests on low-bitrate audio codecs. Tech 3296Google Scholar
  15. 15.
    European Telecommunications Standards Institute (2005) Digital Video Broadcasting (DVB): DVB-H implementation guidelines. ETSI TR 102 377 V1.2.1 (2005-11)Google Scholar
  16. 16.
    Fiske ST, Taylor SE (1991) Social cognition. McGrow-Hill, SingaporeGoogle Scholar
  17. 17.
    Flack J, Harrold J, Woodgate GJ (2007) A prototype 3D mobile phone equipped with a next generation autostereoscopic display. Proc SPIE 6490(64900 M)Google Scholar
  18. 18.
    Fredrickson BL (2000) Extracting meaning from past affective experiences: the importance of peaks, ends and specific emotions. Cogn Emotion 14(4):577–606CrossRefGoogle Scholar
  19. 19.
    Ghinea G, Chen SY (2008) Measuring quality of perception in distributed multimedia: verbalizers vs. imagers. Comput Hum Behav 24(4):1317–1329CrossRefGoogle Scholar
  20. 20.
    Gibson JJ (1979) The ecological approach to visual perception. Houghton Mifflin, Lawrence Eribaum, BostonGoogle Scholar
  21. 21.
    Gochev A, et al (2009) Mobile 3D television: development of core technological elements and user-centered evaluation methods toward an optimized system. In: Creutzburg R, Akopian D (ed) Multimedia on Mobile Devices 2009:7256(1)Google Scholar
  22. 22.
    Goldsmith RE (2001) Using the domain specific innovativeness scale to identify innovative internet consumers. Internet Res Electron Netw Appl Policy 11(2):149–158CrossRefGoogle Scholar
  23. 23.
    Goodman J, Brewster S, Gray P (2004) Using field experiments to evaluate mobile guides. In: Schmidt-Belz B, Cheverst K (ed) Proc HCI 2004Google Scholar
  24. 24.
    Grill-Spector K, Malach R (2004) The human visual cortex. Ann Rev Neurosci 27:649–677CrossRefGoogle Scholar
  25. 25.
    Gulliver SR, Ghinea G (2006) Defining user perception of distributed multimedia quality. ACM Trans Multimed Comput Commun Appl 2(4):241–257CrossRefGoogle Scholar
  26. 26.
    Hands DS et al (2007) Price-dependent quality: examining the effects of price on multimedia quality requirements. Proc SPIE 6492(62920 N)Google Scholar
  27. 27.
    Harrison GW, List JA (2004) Field experiments. J Econ Lit 42(4):1009–1055CrossRefGoogle Scholar
  28. 28.
    Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload. North-Holland, Amsterdam, pp 139–183CrossRefGoogle Scholar
  29. 29.
    ISO 13407 (1999) Human-centered design processes for interactive systems. Int Org StandGoogle Scholar
  30. 30.
    ITU-R BT.500-11 (2002) Methodology for the subjective assessment of the quality of television pictures. Int Telecommun Union—Radiocommun sectorGoogle Scholar
  31. 31.
    ITU-T P.911 Recommendation P.911 (1998) Subjective audiovisual quality assessment methods for multimedia application. Int Telecommun Union—Telecommun sectorGoogle Scholar
  32. 32.
    ITU-T Recommendation P.10 Amendment 1 (2008) Vocabulary for performance and quality of service, New Appendix I Definition of Quality of Experience (QoE). Int Telecommun UnionGoogle Scholar
  33. 33.
    ITU-T. Recommendation E.800 (1994) Terms and definitions related to quality of service and network performance including dependability. Int Telecommun UnionGoogle Scholar
  34. 34.
    Jambon F (2009) User evaluation of mobile devices: in-situ versus laboratory experiments. Int J Mobile Comput Hum Interact 1(2):56–71CrossRefGoogle Scholar
  35. 35.
    Jumisko-Pyykkö S (2008) I would like to see the subtitles and the face or at least hear the voice: effects of picture ratio and audio-video bitrate ratio on perception of quality in mobile television. Multimed Tools Appl 36(1–2):167–184CrossRefGoogle Scholar
  36. 36.
    Jumisko-Pyykkö S, Hannuksela MM (2008) Does context matter in quality evaluation of mobile television? Proc MobileHCI'08:63–72Google Scholar
  37. 37.
    Jumisko-Pyykkö S, Strohmeier D (2008) Report on research methodologies for the experiments. MOBILE3DTV Technical report.
  38. 38.
    Jumisko-Pyykkö S, Utriainen T (2010) Experienced audiovisual quality for mobile 3D television. Proc 3DTV Conference 2010Google Scholar
  39. 39.
    Jumisko-Pyykkö S, Vainio T (2010) Framing the context of use for mobile HCI. Int J Mobile-Human-Computer-Interaction IJMHCI 2(4)Google Scholar
  40. 40.
    Jumisko-Pyykkö S, Häkkinen J, Nyman G (2007) Experienced quality factors—qualitative evaluation approach to audiovisual quality. Proc SPIE Multimedia on Mobile Devices 2007:6507(65070 M)Google Scholar
  41. 41.
    Jumisko-Pyykkö S, Malamal Vadakital VK, Hannuksela MM (2008) Acceptance threshold: bidimensional research method for user-oriented quality evaluation studies. Int J Digit Multimed Broadcast 2008:712380Google Scholar
  42. 42.
    Jumisko-Pyykkö S, Weitzel M, Strohmeier D (2008) Designing for user experience: what to expect from mobile 3D TV and video? Proc 1st Int Conf Designing Interact User Experiences TV and video 2008:183-192Google Scholar
  43. 43.
    Kaikkonen A et al (2008) Will laboratory test results be valid in mobile contexts? In: Lumsden J (ed) Handbook Res User Interface Design Evaluation for Mobile Tech. Information Science Reference, pp 897–909Google Scholar
  44. 44.
    Kennedy RS et al (1993) Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int J Aviation Psychol 3(3):203–220CrossRefGoogle Scholar
  45. 45.
    Kim H, et al. (2002) An empirical study of the use contexts and usability problems in mobile internet. Proc 35th Ann Hawaii Int Conf System Sciences HICSS'02 5(5):132Google Scholar
  46. 46.
    Kjeldskov J et al (2004) Is it worth the hassle? Exploring the added value of evaluating the usability of context-aware mobile systems in the field. Proc MobileHCI 2004:61–73Google Scholar
  47. 47.
    Knoche H, Sasse MA (2009) The big picture on small screens delivering acceptable video quality in mobile TV. ACM Trans Multimed Comput Commun Appl TOMCCAP 5(3):1–27CrossRefGoogle Scholar
  48. 48.
    Knoche H, McCarthy J, Sasse MA (2006) Reading the fine print: the effect of text legibility on perceived video quality in mobile TV. Proc ACM Multimed 2006:727–730Google Scholar
  49. 49.
    Knoche H, McCarthy J, Sasse MA (2008) How low can you go? The effect of low resolutions on shot types. Multimed Tools Appl 36(1–2):145–166CrossRefGoogle Scholar
  50. 50.
    Lewicki MS (2002) Efficient coding of natural sounds. Nat Neurosci 5(4):292–294CrossRefGoogle Scholar
  51. 51.
    McCarthy JD, Sasse MA, Miras D (2004) Sharp or smooth?: Comparing the effect of quantization vs. framerate for streamed video. Proc CHI 2004:535-542Google Scholar
  52. 52.
    Mizobuchi S, Chignell M, Newton D (2005) Mobile text entry: relationship between walking speed and text input task difficulty. Proc MobileHCI'05 111:122-128Google Scholar
  53. 53.
    Mustonen T, Olkkonen M, Häkkinen J (2004) Examining mobile phone text legibility while walking. Ext Abstr Human Factors in Computing Systems CHI '04:1243-1246Google Scholar
  54. 54.
    Nahrstedt K, Steinmetz R (1995) Resource management in networked multimedia systems. IEEE Comput 28(5):52–63Google Scholar
  55. 55.
    Neisser U (1976) Cognition and reality, principles and implications of cognitive psychology. Freeman, San FranciscoGoogle Scholar
  56. 56.
    Oatley K, Jenkins JM (2003) Understanding emotions. Blackwell, OxfordGoogle Scholar
  57. 57.
    O'Hara K, Mitchell AS, Vorbau A (2007) Consuming video on mobile devices. Proc CHI '07:857-866Google Scholar
  58. 58.
    Oksman V, et al (2007) News in mobiles, Comparing text, audio and video. VTT 2007.
  59. 59.
    Oksman V et al (2008) ‘Podracing’: experimenting with mobile TV content consumption and delivery methods. Multimed Syst 14(2):105–114CrossRefGoogle Scholar
  60. 60.
    Oulasvirta A (2009) Field experiments in HCI: promises and challenges. In: Saariluoma P, Isomaki H (eds) Future Interaction Design II, SpringerGoogle Scholar
  61. 61.
    Oulasvirta A, Nyyssönen T (2009) Flexible hardware configurations for studying mobile usability. J Usability Stud 4(2):93–105Google Scholar
  62. 62.
    Oulasvirta A, et al. (2005) Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI. Proc SIGCHI Conf Hum Factors Computing Syst CHI '05:919-928Google Scholar
  63. 63.
    Pirhonen A, Brewster SA, Holguin C (2002) Gestural and audio metaphors as a means of control for mobile devices. Proc CHI’2002:291-298Google Scholar
  64. 64.
    Reiter U, Jumisko-Pyykkö S (2007) Watch, press and catch—impact of divided attention on requirements of audiovisual quality. Proc 12th HCI Int 2007:943-952Google Scholar
  65. 65.
    Roto V (2006) Web browsing on mobile phones—characteristics of user experience. Dissertation, Helsinki University of TechnologyGoogle Scholar
  66. 66.
    Sasse MA, Knoche H (2006) Quality in context—an ecological approach to assessing QoS for mobile TV. Proc ISCA/DEGA Tutor & Res Workshop Percept Quality of Systems 2006Google Scholar
  67. 67.
    Schwarz H, Marpe D, Wiegand T (2007) Overview of the scalable video coding extension of the H.264/AVC standard. IEEE Trans Circ Syst Video Tech 17(9):1103–1120CrossRefGoogle Scholar
  68. 68.
    Shadish W, Cook T, Campbell D (2002) Experimental and quasi-experimental designs. Houghton Mifflin, BostonGoogle Scholar
  69. 69.
    Södergård C (ed) (2003) Mobile television—technology and user experiences. Report on the Mobile TV Project. Espoo: VTT Publications 506Google Scholar
  70. 70.
    Strauss A, Corbin J (1998) Basics of qualitative research: techniques and procedures for developing grounded theory, 2nd edn. Sage, Thousand OaksGoogle Scholar
  71. 71.
    Strohmeier D, Tech G (2010) Sharp, bright, three-dimensional: open profiling of quality for mobile 3DTV coding methods. Proc SPIE Multimedia on Mobile Devices 2010Google Scholar
  72. 72.
    Strohmeier D, Jumisko-Pyykkö S, Kunze K (2010) New, lively, and exciting or just artificial, straining, and distracting—a sensory profiling approach to understand mobile 3D audiovisual quality. Proc VPQM 2010Google Scholar
  73. 73.
    Tamminen S et al (2004) Understanding mobile contexts. Pers Ubiquit Comput 8(2):135–143CrossRefGoogle Scholar
  74. 74.
    Tashakkori A, Teddlie C (2008) Quality of inferences in mixed methods research: calling for an intergrative framework. In: Bergman MM (ed) Advances in Mixed Method Research. Age PublicationsGoogle Scholar
  75. 75.
    Winkler S, Faller C (2005) Audiovisual quality evaluation of low-bitrate video. Proc SPIE IS&T Hum Vis Electron Imaging 5666:139–148Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  1. 1.Human-Centered TechnologyTampere University of TechnologyTampereFinland

Personalised recommendations