Advertisement

Charles Edgar Metz, Ph.D. (1942–2012): pioneer in receiver operating characteristic (ROC) analysis

  • Junji Shiraishi
  • Kunio Doi
Editorial
  • 149 Downloads

Charles Edgar Metz was recognized as a pioneer in receiver operating characteristic (ROC) analysis, as well as a leader in using mathematics to assess and improve the accuracy of diagnostic tests. He made extensive contributions to various medical imaging fields, including general radiography, nuclear medicine, and computer-aided diagnosis.

Charles Metz was born on September 11, 1942, in Bay Shore, New York. He died on July 4, 2012, in Burr Ridge, Illinois, at the age of 69 years., His birthday later became the anniversary of the September 11 US attacks in which almost 3,000 people were killed by terrorists. Furthermore, the day of his death is Independence Day in the United States, with people celebrating it with millions of fireworks. Apart from these unusual coincidences colleagues and friends were saddened by Dr. Metz’s early death and the loss of his brilliant talent.

When Charles Metz was 4 years old, his family moved to Freeport, Long Island. In 1964, he graduated with honors from Bowdoin College in Brunswick, Maine, with a Bachelor’s degree in physics. He obtained a Master’s degree and a Ph.D. in radiological physics from the University of Pennsylvania in 1966 and 1969, respectively.

Dr. Metz started his career in medical image-processing with the work on his Ph.D. dissertation, which was entitled “A mathematical investigation of radioisotope scan image processing” [1]. In this dissertation, he mathematically evaluated various stationary linear image-processing techniques with use of a single digital radionuclide image. He also proposed a new filtering technique for attempting to enhance image resolution and to remove distractions from digital images [2]. However, because digital images had just started to be used in medical fields in the early 1970s, his novel idea of introducing an image-processing filter, called a “Metz Filter,” became widely applied in nuclear medicine, but not until more than a decade later [3, 4].

Soon after obtaining his Ph.D., Dr. Metz became an instructor in the Department of Radiology and at the Argonne Cancer Research Hospital at the University of Chicago. He ended his busy and very productive life having spent his whole career at the University of Chicago. Needless to say, he received rapid academic promotions: to assistant professor in 1971, associate professor in 1975, and professor in 1980. He served as director of the graduate program in medical physics from 1979 to 1986. He served on multiple institutional as well as national and international committees and advisory boards, including study sections for the US National Institutes of Health.

Dr. Metz was best known for extending ROC analysis to the medical imaging field. The procedure of ROC analysis had originally been introduced to medical imaging by Dr. Lee B. Lusted for the evaluation of diagnostic accuracy [5]. Dr. Metz followed this work together with Professor Kurt Rossmann, who was the director of the Center for Radiologic Image Research at the University of Chicago [6].

Figure 1 is a picture of Dr. Metz taken in 1978. In particular, he applied his talent to adopting theoretical principles from other fields for ROC analysis and wrote definitive papers on the application of ROC analysis to radiology. His most significant paper, entitled “Basic Principles of ROC Analysis,” has been cited more than 5,000 times since it was published in 1978 [7], according to Google Scholar.

Fig. 1

Charles E. Metz teaching a class when he was 36 years old

Dr. Metz established ROC methodology in the 1980s. Most researchers (particularly radiologists) in the 1970s had recognized the importance of ROC study for evaluating diagnostic accuracy using their own decision-making, but did not know how to perform ROC observer studies, except for workers at the University of Chicago. To help this situation, for making ROC analysis more widely accessible, Dr. Metz published two valuable papers related to ROC observer study, in 1986 and 1989 [8, 9].

When Lusted introduced ROC observer study, he used a five-category scale for indicating observers’ confidence levels regarding the detection of abnormal lesions [10]. A majority of ROC observer studies were performed by this five-category scale; however, data-categorizing (the use of five scales for indicating observers’ confidence levels for presence/absence of subtle lesions) was generally recognized to be a difficult task for observers. In the early 1990s, Dr. Metz proposed a new method for estimating a binormal ROC curve from continuously distributed test results [11, 12]. Using this new ROC paradigm, researchers were now able to use a continuous rating scale instead of the five-category scale.

Another important contribution by Professor Metz to ROC analysis was his development of various kinds of ROC software. He provided extensive packages of this software to more than 15,000 registered users worldwide. The first item of ROC software (called RSCORE II) was developed in 1968 by Dr. Donald D. Dorfman, who had been an investigator together with Dr. Metz; this was a binormal ROC model [13]. With some modifications of RSCORE II, Dr. Metz developed a new type of software, ROCFIT, in 1984, which was designed only for categorized scales [14]. Subsequently, Dr. Metz reported on a new method for estimating a binormal ROC curve from continuously distributed test results, for which he developed a new type of software in 1990 which he called LABROC4 [12]. LABROC4 could run on a DOS/V PC machine, a procedure which was not as expensive as the use of the workstation available at that time, and it could be distributed by the use of a 3.5-inch floppy disk. A number of researchers applied to obtain this novel ROC software. When Dr. Metz had developed and prepared to distribute LABROC4, a sub-group of the Japanese Society of Radiological Technology established “Metz’s ROC Software Users Group” and started to distribute LABROC4 in Japan with permission of Dr. Metz (Fig. 2).

Fig. 2

The 1st Metz’s ROC Software Users Group meeting held during the annual meeting of the JSRT at the Kobe Portopia Hotel in 1994. (Dr. Metz; left, Author: middle; Dr. Kodera, who was the chair of the imaging section of the JSRT: right)

In 1998, Dr. Metz proposed a new generalized method for ROC curve-fitting and statistical testing with a new algorithm called ROCKIT [15]. This new algorithm used maximum-likelihood estimation to fit a binormal ROC curve to either continuously distributed or ordinal-category data. ROCKIT also calculates the statistical significance of differences between ROC index estimates and parameters for unpaired, partially paired, or fully paired data.

As the use of the Internet was becoming the norm for researchers in the late 1990s, Dr. Metz started to distribute his ROC software to all researchers who were interested in ROC analysis using the internet from the web site of his laboratory (Fig. 3).

Fig. 3

Web sites of Metz ROC software at the University of Chicago, and Metz’s ROC Software Users Group, which was a sub-group of the Japanese Society of Radiological Technology (JSRT)

When image evaluation with ROC analysis became necessary for the performance of statistically significant tests to demonstrate evidence of diagnostic accuracy, Dr. Dorfman, Dr. Kevin S. Berbaum (who was a colleague of Dr. Dorfman), and Dr. Metz developed a new algorithm for comparison of multiple treatments (e.g., imaging modalities) by analyzing data obtained from multiple readers and multiple cases (MRMC) [16]. Then, Dr. Metz developed a new type of software, called LABMRMC, which employs the Dorfman-Berbaum-Metz (DBM) algorithm using LABROC4 curve-fitting, as well as DBM-MRMC, which added a “Proper” binormal ROC algorithm [17, 18] to LABMRMC.

In 2005, based on his outstanding contributions to medical imaging science, Dr. Metz was awarded the twelfth L. H. Gray Medal by the International Commission on Radiation Units and Measurements.

Professor Metz was respected and loved by everyone who worked with him because he was generous with his time, especially with younger colleagues and with students. He advised or served on dissertation and thesis advisory committees for 40 students. The recipient of a first and a second Kurt Rossmann Award for Excellence in Teaching in the Graduate Program in Medical Physics at the University of Chicago, he could not get a third award because the rule was changed so that it could be won only once (Fig. 4).

Fig. 4

Charles E. Metz sitting on his desk in his office at the University of Chicago. Dr. Metz was regarded as an authority on many aspects of the World War II German aircrafts

Professor Metz contributed significantly to numerous research projects [19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35] conducted at the Kurt Rossmann Laboratories for Radiologic Image Research (Director: Kunio Doi, Ph.D.) at the University of Chicago over the three decades from 1977 to 2011, particularly in the quantitative evaluation of the effect of image-processing and digital parameters on diagnostic accuracy as well as the usefulness of computer-aided diagnostic (CAD) schemes in many imaging modalities, including breast and chest imaging. Dr. Metz’s high level of expertise in the area of ROC analysis was expressed by his being a major contributor to many publications in the Rossmann Laboratories in the early phase of research on digital radiography, and in the development of many CAD schemes. All members of the Rossmann Laboratories benefitted greatly from his contributions to their work. The success in the development of CAD at the Rossmann Laboratories would not have been possible without Dr. Metz as a member. In return, Dr. Metz enjoyed and benefitted from his collaboration with many researchers on the use of ROC analysis, being able to apply his methodology to many clinical issues and also to basic imaging studies related to digital radiography.

Dr. Metz spent considerable time with clinicians, including Heber MacMahon, MD, in chest imaging and Robert A. Schmidt, MD, in breast imaging, discussing clinical issues that could be subjected to ROC analysis. Thus, he was made aware of complex and difficult clinical issues that were suitable for the application of ROC methodology.

The cited list of his publications [19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35] includes only some representative studies on many different projects. These papers had a variety of coauthors, indicating the broad range of research topics in which Dr. Metz was involved. He was a scientist with the highest, genuine integrity and the highest level of competence in theoretical work, but he also had a deep understanding of and knowledge about clinical issues.

Professor Metz was also loved by many Japanese researchers. He visited Japan six times and spent 4 months as a guest professor at the College of Biomedical Technology of Osaka University in 1989 (Fig. 5). He was able to understand ideas which were presented by Japanese researchers even in poor English, and he answered their questions very carefully and made valuable suggestions in slow and simple English. His advice was always positive, encouraging, and constructive.

Fig. 5

At the BBQ restaurant in Sapporo, Hokkaido, Japan. Dr. Metz was invited as a special lecturer at the autumn conference of the JSRT in 1998. (Dr. Ishida, who is the current editor in chief of the journal of the JSRT; left, Dr. Metz; middle, Author: right)

Professor Metz died of pancreatic cancer at the age of 69 years. At his last lecture on October 13, 2011, he presented his thoughts on careers, research, wants, needs, love, and a few other topics to a capacity audience of students and faculty at the University of Chicago. In this lecture, he remarked that the best way to choose a career is to find something that you love to do, and then to become so good at it that others will be eager to pay you well to do it. In addition, he said that, in order for you to succeed in research, it must be about something that you want to do, that you need to do, and that you love to do. Therefore, you should either cultivate the want, the need, and the love, or else find a career in which research is irrelevant.

In this last lecture, he announced “Metz’s law of scientific communication,” which was based on the principle that no idea is worth anything unless you can get it out of your head and put it more or less intact into other heads, if possible, in a form that will stimulate additional thoughts in the other heads. Additionally, he emphasized that you will have an advantage in your career if you can teach yourself to communicate well. In the closing part of this lecture, he referred to four quotations:

“When I am working on a problem, I never think about beauty; I think only of how to solve the problem. But when I’ve finished, if the solution is not beautiful, I know it is wrong.”

— Buckminster Fuller

“One of the principal objects of theoretical research in any department of knowledge is to find the point of view from which the subject appears in its greatest simplicity.”

— J. Willard Gibbs

“Everything should be made as simple as possible, but not simpler.”

— Albert Einstein

“God is in the details.”

— Ludwig Mies van der Rohe

ROC analysis and Metz’s ROC software have been used and extended in many fields of medical imaging, and in the evaluation of artificial intelligence and the products of machine learning. As we make use of his accomplishments, we will never forget the teachings and the philosophical insights we learned from Professor Metz.

Notes

Acknowledgements

The authors are grateful to Elisabeth Lanzl for improving the manuscript.

References

  1. 1.
    Metz CE: A mathematical investigation of radioisotope scan image processing. Ph.D. Thesis, University of Pennsylvania, 1969.Google Scholar
  2. 2.
    Metz CE, Beck RN. Quantitative effects of stationary linear image processing on noise and resolution of structure in radionuclide images. J Nucl Med. 1974;15:164–70.PubMedGoogle Scholar
  3. 3.
    King MA, Schwinger RB, Doherty PW, Penney BC. Two-dimensional filtering of SPECT images using the Metz and Wiener filters. J Nucl Med. 1984;25:1234–40.PubMedGoogle Scholar
  4. 4.
    King MA, Penney BC, Glick SJ. An image-dependent Metz filter for nuclear medicine images. J Nucl Med. 1988;29:1980–9.PubMedGoogle Scholar
  5. 5.
    Lusted LB. Signal detectability and medical decision-making. Science. 1971;171:1217–9.CrossRefGoogle Scholar
  6. 6.
    Metz CE, Goodenough DJ, Rossmann K. Evaluation of receiver operating characteristic curve data in terms of information theory, with applications in radiography. Radiology. 1973;109(2):297–303.CrossRefGoogle Scholar
  7. 7.
    Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8(4):283–98.CrossRefGoogle Scholar
  8. 8.
    Metz CE. ROC methodology in radiologic imaging. Invest Radiol. 1986;21(9):720–33.CrossRefGoogle Scholar
  9. 9.
    Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 1989;24(3):234–45.CrossRefGoogle Scholar
  10. 10.
    Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of receiver operating characteristic (ROC) curves. Radiology. 1974;110(1):89–95.CrossRefGoogle Scholar
  11. 11.
    Metz CE. New methods for estimating a binormal ROC curve from continuously-distributed test results. In: Presented at the 1990 Annual Meeting of the American Statistical Association, 1990.Google Scholar
  12. 12.
    Metz CE, Herman BA, Shen J-H. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat Med. 1998;17:1033–53.CrossRefGoogle Scholar
  13. 13.
    Dorfman DD. RSCORE II. In: Swets JA, Pickett RM, editors. Evaluation of diagnostic systems: Methods from signal detection theory. pp. 212–232. New York: Academic Press.Google Scholar
  14. 14.
    Metz CE, Shen J-H, Wang P-L, Kronman HB .ROCFIT: a program for maximum likelihood estimation of a binormal ROC curve and its associated parameters from a set of categorical rating-scale data, Department of Radiology and The Franklin McLean Memorial Research Institute, University of Chicago, 1989.Google Scholar
  15. 15.
    Metz CE, Herman BA, Roe CA. Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets. Med Decis Mak. 1998;18(1):110–21.CrossRefGoogle Scholar
  16. 16.
    Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: Generalization to the population of readers and patients with the jackknife method. Invest Radiol. 1992;27(9):723–31.CrossRefGoogle Scholar
  17. 17.
    Metz CE, Pan X. “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J Math Psychol. 1999;43:1–33.CrossRefGoogle Scholar
  18. 18.
    Pesce LL, Metz CE. Reliable and computationally efficient maximum-likelihood estimation of “Proper” binormal ROC curves. Acad Radiol. 2007;14:814–29.CrossRefGoogle Scholar
  19. 19.
    Ishida M, Doi K, Loo L-N, Metz CE, Lehr JL. Digital image processing: effect on the detectability of simulated low-contrast radiographic patterns. Radiology. 1984;150:569–75.CrossRefGoogle Scholar
  20. 20.
    Loo L-N, Doi K, Metz CE. A comparison of physical image quality indices and observer performance in the radiographic detection of nylon beads. Phys Med Biol. 1984;29:837–56.CrossRefGoogle Scholar
  21. 21.
    MacMahon H, Vyborny CJ, Metz CE, Doi K, Sabeti V, Solomon S. Digital radiography of subtle pulmonary abnormalities: an ROC study of the effect of pixel size on observer performance. Radiology. 1986;158:21–6.CrossRefGoogle Scholar
  22. 22.
    Chan HP, Vyborny CJ, MacMahon H, Metz CE, Doi K, Sickles EA. Digital mammography: ROC studies of the effects of pixel size and unsharp-mask filtering on the detection of subtle microcalcifications. Invest Radiol. 1987;22:581–9.CrossRefGoogle Scholar
  23. 23.
    MacMahon H, Metz CE, Doi K, Kim T, Giger ML, Chan HP. Digital chest radiography: effect on diagnostic accuracy of hardcopy, conventional video, and reversed grey scale video display format. Radiology. 1988;168:669–73.CrossRefGoogle Scholar
  24. 24.
    Ohara K, Doi K, Metz CE, Giger ML. Investigation of basic imaging properties in digital radiography. 13. Effect of structured noise on the detectability of simulated stenotic lesions. Med Phys. 1989;16:14–21.CrossRefGoogle Scholar
  25. 25.
    Chan HP, Doi K, Vyborny CJ, Schmidt RA, Metz CE, Lam KL, Ogura T, Wu Y, MacMahon H. Improvement in radiologists’ detection of clustered microcalcifications on mammograms: the potential of computer-aided diagnosis. Invest Radiol. 1990;25:1102–10.CrossRefGoogle Scholar
  26. 26.
    Giger ML, Ahn N, Doi K, MacMahon H, Metz CE. Computerized detection of pulmonary nodules in digital chest images: use of morphological filters in reducing false-positive detections. Med Phys. 1990;17:861–5.CrossRefGoogle Scholar
  27. 27.
    MacMahon H, Doi K, Sanada S, Montner SM, Giger ML, Metz CE, Nakamori N, Yin FF, Xu XW, Yonekawa H, Takeuchi H. Data compression: effect on diagnostic accuracy in digital chest radiography. Radiology. 1991;178:175–9.CrossRefGoogle Scholar
  28. 28.
    Yin FF, Giger ML, Doi K, Metz CE, Vyborny CJ, Schmidt RA. Computerized detection of masses in digital mammograms: analysis of bilateral subtraction images. Med Phys. 1991;18:955–63.CrossRefGoogle Scholar
  29. 29.
    Hoffmann KR, MacMahon H, Doi K, Metz CE, Yao L, Abe K. Evaluation of an enhanced digital film-duplication system by receiver operating characteristic analysis. Invest Radiol. 1993;28:1134–8.CrossRefGoogle Scholar
  30. 30.
    Nishikawa RM, Giger ML, Doi K, Metz CE, Yin F-F, Vyborny CJ, Schmidt RA. Effect of case selection of the performance of computer-aided detection schemes. Med Phys. 1994;21:265–9.CrossRefGoogle Scholar
  31. 31.
    Wu Y, Doi K, Giger ML, Metz CE, Zhang W. Reduction of false-positives in computerized detection of lung nodules in chest radiographs using artificial neural networks, discriminant analysis, and a rule-based scheme. J Digit Imaging. 1994;7:196–207.CrossRefGoogle Scholar
  32. 32.
    Jiang Y, Nishikawa RM, Wolverton DE, Metz CE, Giger ML, Schmidt RA, Vyborny CJ, Doi K. Malignant and benign clustered microcalcifications: automated feature analysis and classification. Radiology. 1996;198:671–8.CrossRefGoogle Scholar
  33. 33.
    Kobayashi T, Xu X-W, MacMahon H, Metz CE, Doi K. Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on radiographs. Radiology. 1996;199:843–8.CrossRefGoogle Scholar
  34. 34.
    MacMahon H, Engelmann R, Behlen F, Hoffmann KR, Ishida T, Roe C, Metz CE, Doi K. Computer-aided diagnosis of pulmonary nodules: results of a large scale observer test. Radiology. 1999;213:723–6.CrossRefGoogle Scholar
  35. 35.
    Shiraishi J, Pesce L, Metz CE, Doi K. Experimental design and data analysis in receiver operating characteristic studies: lessons learned from reports in Radiology from 1997 to 2006. Radiology. 2009;253:822–30.CrossRefGoogle Scholar

Copyright information

© Japanese Society of Radiological Technology and Japan Society of Medical Physics 2018

Authors and Affiliations

  1. 1.Faculty of Life SciencesKumamoto UniversityKumamotoJapan
  2. 2.Department of RadiologyThe University of ChicagoChicagoUSA

Personalised recommendations