Multimedia Tools and Applications

, Volume 75, Issue 4, pp 2367–2391 | Cite as

A new method for evaluating the subjective image quality of photographs: dynamic reference

  • Mikko Nuutinen
  • Toni Virtanen
  • Tuomas Leisti
  • Terhi Mustonen
  • Jenni Radun
  • Jukka Häkkinen


The Dynamic Reference (DR) method has been developed for subjective image quality experiments in which original or undistorted images are unavailable. The DR method creates reference image series from test images. Reference images are presented to observers as a slide show prior to evaluating their quality. As the observers view the set of reference images, they determine the overall variation in quality within the set of test images. This study compared the performance of the DR method to that of the standardized absolute category rating (ACR) and paired comparison (PC) methods. We measured the performance of each method in terms of time effort and discriminability. The results showed that the DR method is faster than the PC method and more accurate than the ACR method. The DR method is especially suitable for experiments that require highly accurate results in a short time.


Image quality Subjective evaluation method Performance measure 


  1. 1.
    Bagiella E, Sloan R, Heitjan D (2000) Mixed-effects models in psychophysiology. Psychophysiology 37(1):13–20CrossRefGoogle Scholar
  2. 2.
    Bland M (1995) An introduction to medical statistics, 2nd edn. Oxford Medical PublicationsGoogle Scholar
  3. 3.
    Blin J (2006) New quality evaluation method suited to multimedia context: samviq. In: Proceeding of the 2nd International Workshop on Video Processing and Quality Metrics, (VPQM)Google Scholar
  4. 4.
    Engelke U, Maeder A, Zepernick HJ (2012) Human observer confidence in image quality assessment. Signal Process Image Commun 27(9):935–947CrossRefGoogle Scholar
  5. 5.
    Greenwood P, Nikulin M (1996) A guide to chi-squared testing. A Wiley-Interscience Publication. Wiley.
  6. 6.
    I3A (2007) CPIQ initiative phase 1 white paper fundamentals and review of considered test methodsGoogle Scholar
  7. 7.
    ISO 20462 (2012) Psychophysical experimental methods for estimating image quality – Part 3: Quality ruler methodGoogle Scholar
  8. 8.
    ITU-R, Rec.BT.1788 (2007) Methodology for the subjective assessment of video quality in multimedia applicationsGoogle Scholar
  9. 9.
    ITU-R BT.500 (2012) Methodology for the subjective assessment of the quality of television picturesGoogle Scholar
  10. 10.
    ITU-T Rec. P. 910 (2008) Subjective video quality assessment methods for multimedia applicationsGoogle Scholar
  11. 11.
    Jianping Z, Glotzbach J (2007) Image pipeline tuning for digital cameras. In: Proceedings of IEEE International Symposium on Consumer Electronics, (ISCE), pp 1–4Google Scholar
  12. 12.
    Jin E, Keelan B (2010) Slider-adjusted softcopy ruler for calibrated image quality assessment. J Electron Imaging 19(1):011,009CrossRefGoogle Scholar
  13. 13.
    Kuang J, Yamaguchi H, Liu C, Johnson G., Fairchild M (2007) Evaluating HDR rendering algorithms. ACM Trans Appl Percept 4(2)Google Scholar
  14. 14.
    Leisti T, Radun J, Virtanen T, Halonen R, Nyman G (2009) Subjective experience of image quality: attributes, definitions, and decision making of subjective image quality. In: Proceedings of the SPIE 7242, Image Quality and System Performance VI, pp 72,420D–72,420D–9Google Scholar
  15. 15.
    Mantiuk R, Tomaszewska A, Mantiuk R (2012) Comparison of four subjective methods for image quality assessment. Comput Graph Forum 31(8):2478–2491CrossRefGoogle Scholar
  16. 16.
    Nikkanen J, Gerasimow T, Kong L (2008) Subjective effects of white-balancing errors in digital photography. Opt Eng 47(11):113,201. doi: 10.1117/1.3013232 CrossRefGoogle Scholar
  17. 17.
    Nuutinen M, Orenius O, Säämänen T, Oittinen P (2012) A framework for measuring sharpness in natural images captured by digital cameras base on reference image and local areas. EURASIP J Image Video Process 2012(13640):1–15Google Scholar
  18. 18.
    Nuutinen M, Virtanen T, Oittinen P (2012) Features for predicting quality of images captured by digital cameras. In: Proceedings of IEEE International Symposium on Multimedia, pp 165–168Google Scholar
  19. 19.
    Petit J, Mantiuk R (2013) Assessment of video tone-mapping: are cameras’ s-shaped tone-curves good enough?. J Vis Commun Image Represent 24(7):1020–1030CrossRefGoogle Scholar
  20. 20.
    Ponomarenko V, Lukin V, Zelensky A, Egiazarian K, Carli M, Battisti F (2009) Tid2008 a database for evaluating of full-reference visual quality assessment metrics. Adv Modern Radioelectron 10:30–45Google Scholar
  21. 21.
    Radun J, Leisti T, Virtanen T, Häkkinen J, Vuori T, Nyman G (2010) Evaluating the multivariate visual quality performance of image-processing components. ACM Trans Appl Percept 7(3):16:1–16:16CrossRefGoogle Scholar
  22. 22.
    Ramanath R, Snyder W, Yoo Y, Drew M (2005) Color image processing pipeline. IEEE Signal Proc Mag 22(1):34–43CrossRefGoogle Scholar
  23. 23.
    Redi J, Liu H, Alers H, Zunino R, Heynderickx I (2010) Comparing subjective image quality measurement methods for the creation of public databases. In: Proceedings of the SPIE 7529, Image Quality and System Performance VII, pp 752,903–752,903–11Google Scholar
  24. 24.
    Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15(11):3440–3451CrossRefGoogle Scholar
  25. 25.
    Silverstein D, Farrell J (2001) Efficient method for paired comparison. Jof Electron Imaging 10(2):394–398CrossRefGoogle Scholar
  26. 26.
    Taichi K., Kazuhisa Y., Takanori H. (2012) Performance comparison of subjective assessment methods for 3d video quality. In: Proceedings of International Workshop on Quality of Multimedia Experience, (QoMEx), pp 218–223Google Scholar
  27. 27.
    Teunissen K (1996) The validity of CCIR quality indicators along a graphical scale. SMPTE Motion Imaging J 105(3):144–149Google Scholar
  28. 28.
    Ween B, Kristoffersen D, Hamilton G, Olsen D (2005) Image quality preferences among radiographers and radiologists. a conjoint analysis. Radiography 11(3):191–197CrossRefGoogle Scholar
  29. 29.
    Winkler S (2009) On the properties of subjective ratings in video quality experiments. In: Proceedings of international workshop on quality of multimedia experience, (QoMEx), pp 139–144Google Scholar
  30. 30.
    Winkler S (2012) Analysis of public image and video databases for quality assessment. IEEE J Select Topics Signal Process 6(6):616–625CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Mikko Nuutinen
    • 1
  • Toni Virtanen
    • 1
  • Tuomas Leisti
    • 1
  • Terhi Mustonen
    • 1
  • Jenni Radun
    • 1
  • Jukka Häkkinen
    • 1
  1. 1.Institute of Behavioural Sciences, University of HelsinkiHelsinkiFinland

Personalised recommendations