Advertisement

Journal of Digital Imaging

, Volume 10, Issue 3, pp 103–107 | Cite as

Forced choice and ordinal discrete rating assessment of image quality: A comparison

  • David Gur
  • David A. Rubin
  • Barry H. Kart
  • Arleen M. Peterson
  • Carl R. Fuhrman
  • Howard E. Rockette
  • Jill L. King
Article

Abstract

This study compared a five-category ordinal scale and a two-alternative forced-choice subjective rating of image quality preferences in a multiabnormality environment. 140 pairs of laser-printed posteroanterior (PA) chest images were evaluated twice by three radiologists who were asked to select during a side-by-side review which image in each pair was the “better” one for the determination of the presence or absence of specific abnormalities. Each pair included one image (the digitized film at 100 μm pixel resolution and laser printed onto film) and a highly compressed (∼60∶1) and decompressed version of the digitized film that was laser printed onto film. Ratings were performed once with a five-category ordinal scale and once with a two-alternative forced-choice scale. The selection process was significantly affected by the rating scale used. The “comparable” or “equivalent for diagnosis” category was used in 88.5% of the ratings with the ordinal scale. When using the two-alternative forced-choice approach, noncompressed images were selected 66.8% of the time as being the “better” images. This resulted in a significantly lower ability to detect small differences in perceived image quality between the noncompressed and compressed images when the ordinal rating scale is used. Observer behavior can be affected by the type of question asked and the rating scale used. Observers are highly sensitive to small differences in image presentation during a side-by-side review.

Key Words

image quality observer performance study methodology ratings 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Rosenthal MS, Good WF, Costa-Greco MA, et al: The effect of image processing on chest radiograph interpretations in a PACS environment. Invest Radiol 25:897–901, 1990CrossRefPubMedGoogle Scholar
  2. 2.
    Metz CE: Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 24:234–245 (1989)CrossRefPubMedGoogle Scholar
  3. 3.
    Rockette HE, Obuchowski NA, Gur D, et al: Effect of experimental design on sample size. Proc SPIE 1446:276–286, 1991CrossRefGoogle Scholar
  4. 4.
    Obuchowski NA, Zepp RC: Simple steps for improving multiple-reader studies in radiology. AJR 166:517–521, 1996PubMedGoogle Scholar
  5. 5.
    Holbert JM, Staiger M, Chang TS, et al: Selection of processing algorithms for digital image compression: A rank order study. Acad Radiol 2:273–276, 1995CrossRefPubMedGoogle Scholar
  6. 6.
    Good WF, Gur D, Feist JH, et al: Subjective and objective assessment of image quality—a comparison. J Digit Imag 7:77–78, 1994Google Scholar
  7. 7.
    Good WF, Maitz GS, Gur D: Joint photographic expert group compatible data compression of mammograms. J Digit Imag 7:123–132, 1994Google Scholar
  8. 8.
    Britton CA, Gabriele OF, Chang TS, et al: Subjective quality assessment of computed radiography hand images. J Digit Imag 9:21–24, 1996CrossRefGoogle Scholar
  9. 9.
    Nill NB: A visual model weighted cosine transform for image compression and quality assessment. IEEE Trans Commun COM-33:551–557, 1985CrossRefGoogle Scholar
  10. 10.
    Ngan KN, Leong KS, Singh H: Cosine transform coding incorporating human visual system model. Proc SPIE 707:165–171, 1986Google Scholar
  11. 11.
    Kelley DH: Visual contrast sensitivity. Opt Acta 24:107–129, 1977Google Scholar
  12. 12.
    Thaete FL, Fuhrman CR, Oliver JH, et al: Digital radiography and conventional imaging of the chest: A comparison of observer performance. AJR 162:575–581, 1994PubMedGoogle Scholar
  13. 13.
    Hamdan MA, Jensen DR: A bivariate binomial distribution and some applications. Austral J Statist 18:163–169, 1976CrossRefGoogle Scholar
  14. 14.
    Hamdan MA, Nasro MO: Maximum likelihood estimation of the parameters of the bivariate binomial distribution. Commun Statist Theor Method 15:747–754, 1986CrossRefGoogle Scholar
  15. 15.
    Rockette HE, Gur D, Metz C: The use of continuous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques. Invest Radiol 27:169–172, 1992CrossRefPubMedGoogle Scholar
  16. 16.
    Gur D, Rockette HE, Good WF, et al: Effect of observer instruction on ROC study of chest images. Invest Radiol 25:230–234, 1990PubMedCrossRefGoogle Scholar

Copyright information

© W.B. Saunders Company 1997

Authors and Affiliations

  • David Gur
    • 1
  • David A. Rubin
    • 1
  • Barry H. Kart
    • 1
  • Arleen M. Peterson
    • 1
  • Carl R. Fuhrman
    • 1
  • Howard E. Rockette
    • 1
  • Jill L. King
    • 1
  1. 1.Department of RadiologyUniversity of PittsburghPittsburgh

Personalised recommendations