Abstract
From a practical point of view, image quality in medicine must be defined in terms of the decisons that physicians can make by reading the images. Image-based decisions concerning the actual state of an object or patient are evaluated most meaningfully by ROC analysis, which has been used to quantify detection performance in sensory psychophysics since the early 1960s and more recently has been applied to evaluate diagnostic systems. After surveying briefly some considerations that motivate the use of ROC analysis in medical image evauation, this manuscript addresses various practical issues of experimental design and data analysis, including the selection of cases and observers, the need for standards of truth, reading-order effects, data collection, curve fitting, statistical testing, and possible generalizations of conventional ROC methodology.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Begg, C.B., and McNeil, B.J. (1988). Assessment of radiologic tests: control of bias and other design considerations, Radiology 167, pp. 565–569.
Bunch, P.C., Hamilton, J.F., Sanderson, G.K., and Simmons, A.H. (1977). A free response approach to the measurement and characterization of radiographic observer performance. Proc. SPIE 127, pp. 124–135.
Chakraborty, D.P. (1988). Maximum likelihood analysis of free response receiver operating characteristic (FROC) data. Personal communication.
Dorfman, D.D., and Alf, E. (1969). Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals — rating method data, J. Math. Psychol. 6, pp. 487–496.
Egan, J.P. (1975). Signal detection theory and ROC analysis, Academic Press, New York.
Grey, D.R., and Morgan, B.J.T. (1972). Some aspects of ROC curve-fitting: normal and logistic models, J. Math. Psychol. 9, pp. 128–139.
Getty, D.J., Pickett, R.M., D’Orsi, C.J., and Swets, J.A. (1988). Enhanced interpretation of diagnostic images, Invest. Radiol. 23, pp. 240–252.
Goodenough, D.J., and Metz, C.E. (1977). Implications of a “noisy” observer to data processing techniques. In: Information processing in medical imaging, C. Raynaud and A.E. Todd-Pokropek (eds.), CEA, Orsay, pp. 400–419.
Gray, R., Begg, C.B., and Greenes, R.A. (1984). Construction of receiver operating characteristic curves when disease verification is subject to selection bias, Med. Decis. Making 4, pp. 151–164.
Green, D.M., and Swets, J.A. (1966). Signal detection theory and psychophysics, Wiley, New York. Reprint with corrections: (1974). Krieger, Huntington NY.
Hanley, J.A., and McNeil, B.J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve, Radiology 143, pp. 29–36.
Hanley, J.A., and McNeil, B.J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases, Radiology 148, pp. 839–843.
International Atomic Energy Agency (1977). IAEA co-ordinated research programme on the intercomparison of computer-assisted scinigraphic techniques. In: Medical radionuclide imaging, IAEA, Vienna, 1, pp. 585–615.
Kendall, M., and Stewart, A. (1976). The advanced thoery of statistics, Hafner, New York, 3, pp. 182–243.
Lusted, L.B. (1978). General problems in medical decision making, with comments on ROC analysis, Seminars Nucl. Med. 8, pp. 299–306.
Loo, L.-N., Doi, K., Ishida, M., Metz, C.E., Chan, H.-P., Higashida, Y., and Kodera, Y. (1983). An empirical investigation of variability in contrast-detail diagram measurements, Proc. SPIE 419, pp. 68–76.
MacMillan, N.A., and Kaplan, H.L. (1985). Detection theory analysis of group data: estimating sensitivity from average hit and false alarm rates, Psychol. Bull. 98, pp. 185–199.
McNeil, B.J., Keeler, E., and Adelstein, S.J. (1975). Primer on certain elements of medical decision making, N. Engl. J. Med. 293, pp. 211–215.
McNeil, B.J., and Hanley, J.A. (1984). Statistical approaches to the analysis of receiver operating characteristic (ROC) curves, Med. Decis. Making 4, pp. 137–150.
Metz, C.E. (1978). Basic principles of ROC analysis, Seminars Nucl. Med. 8, pp. 283–298.
Metz, C.E. (1979). Applications of ROC analysis in diagnostic image evaluation. In: The physics of medical imaging: recording system measurements and techniques, A. Haus (ed.), Am. Inst. Phys., New York, pp. 546–572.
Metz, C.E. (1986a). ROC methodology in radiologic imaging, Invest. Radiol. 21, pp. 720–733.
Metz, C.E. (1986b). Statistical analysis of ROC data in evaluating diagnostic performance. In: Multiple regression analysis: applications in the health sciences, D. Herbert and R. Myers (eds.), Am. Inst. Phys., New York, pp. 365–384.
Metz, C.E. (1988). Some practical issues of experimental design and data analysis in radiological ROC syudies, Invest. Radiol. (in press).
Metz, C.E., and Kronman, H.B. (1980). Statistical significance tests for binormal ROC curves, J. Math. Psychol. 22, pp. 218–243.
Metz, C.E., Kronman, H.B., Wang, P.-L., and Shen, J.-H. (1985). ROCFIT: a modified maximum likelihood algorithm for estimating a binormal ROC curve from confidence-rating data, Dept. Radiology, Univ. Chicago, Chicago. Mini-and micro-computer versions of this and other software for ROC analysis are available from the author at no cost.
Metz, C.E., Starr, S.J., Lusted, L.B., and Rossmann, K. (1975). Progress in evaluation of human observer visual detection performance using the ROC curve approach. In: Information processing in medical imaging, C. Raynaud and A.E. Todd-Pokropek (eds.), CEA, Orsay, pp. 420–436.
Metz, C.E., Wang, P.-L., and Kronman, H.B. (1984). A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Information processing in medical imaging, F. Deconinck (ed.), Nijhoff, The Hague, pp. 432–445.
Ransohoff, D.F., and Feinstein, A.R., (1978). Problems of spectrum and bias in evaluating the efficacy of diagnostic tests, N. Engl. J. Med. 299, pp. 926–930.
Starr, S.J., Metz, C.E., Lusted, L.B., and Goodenough, D.J. (1975). Visual detection and localization of radiographic images, Radiology 116, pp. 533–538.
Swets, J.A. (1979). ROC analysis applied to the evaluation of medical imaging techniques, Invest. Radiol. 14, pp. 109–121.
Swets, J.A. (1986a). Indices of discrimination or diagnostic accuracy: their ROCs and implied models, Psychol. Bull. 99, pp. 100–117.
Swets, J.A. (1986b). Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurementof performance, Psychol. Bull. 99, pp. 181–198.
Swets, J.A., and Pickett, R.M. (1982). Evaluation of diagnostic systems: methods from signal detection theory, Academic Press, New York.
Weinstein, M.C., and Feinberg, H.V. (1980). Clinical decision analysis, Saunders, Philadelphia.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1992 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Metz, C.E. (1992). Evaluation of Medical Images. In: Todd-Pokropek, A.E., Viergever, M.A. (eds) Medical Images: Formation, Handling and Evaluation. NATO ASI Series, vol 98. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-77888-9_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-77888-9_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-77890-2
Online ISBN: 978-3-642-77888-9
eBook Packages: Springer Book Archive