Advertisement

Advances in Data Analysis and Classification

, Volume 12, Issue 3, pp 683–704 | Cite as

Rethinking an ROC partial area index for evaluating the classification performance at a high specificity range

  • Juana-María Vivo
  • Manuel Franco
  • Donatella Vicari
Regular Article
  • 79 Downloads

Abstract

The area under a receiver operating characteristic (ROC) curve is valuable for evaluating the classification performance described by the entire ROC curve in many fields including decision making and medical diagnosis. However, this can be misleading when clinical tasks demand a restricted specificity range. The partial area under a portion of the ROC curve (\({ pAUC}\)) has more practical relevance in such situations, but it is usually transformed to overcome some drawbacks and improve its interpretation. The standardized \({ pAUC}\) (\({ SpAUC}\)) index is considered as a meaningful relative measure of predictive accuracy. Nevertheless, this \({ SpAUC}\) index might still show some limitations due to ROC curves crossing the diagonal line, and to the problem when comparing two tests with crossing ROC curves in the same restricted specificity range. This paper provides an alternative \({ pAUC}\) index which overcomes these limitations. Tighter bounds for the \({ pAUC}\) of an ROC curve are derived, and then a modified \({ pAUC}\) index for any restricted specificity range is established. In addition, the proposed tighter partial area index (\({ TpAUC}\)) is also shown for classifier when high specificity must be clinically maintained. The variance of the \({ TpAUC}\) is also studied analytically and by simulation studies in a theoretical framework based on the most typical assumption of a binormal model, and estimated by using nonparametric bootstrap resampling in the empirical examples. Simulated and real datasets illustrate the practical utility of the \({ TpAUC}\).

Keywords

ROC curve Partial area under ROC curve Classification performance Binormal model Bootstrap Predictive accuracy 

Mathematics Subject Classification

62H30 62P10 

Notes

Acknowledgements

The authors would like to thank two reviewers, Associate Editor and the Coordinating Editor for their constructive suggestions, which have improved the presentation of this paper. The authors would also like to acknowledge Prof. A.I. Bandos by his help for the simulation process, and the research teams of Methods for Analysis of Diagnostic Performance at the University of Pittsburgh, and Diagnostic and Biomarkers Statistical Center, for providing the datasets. This work was partially supported by Spanish Ministry of Economy and Competitiveness/FEDER under Grant TIN2014-53749-C2-2R.

Supplementary material

11634_2017_295_MOESM1_ESM.docx (43 kb)
Supplementary material 1 (docx 43 KB)
11634_2017_295_MOESM2_ESM.docx (52 kb)
Supplementary material 2 (docx 51 KB)

References

  1. Baker SG, Pinsky PF (2001) A proposed design and analysis for comparing digital and analog mammography: special receiver operating characteristic methods for cancer screening. J Am Stat Assoc 96:421–428MathSciNetCrossRefGoogle Scholar
  2. Baker SG (2003) The central role of receiver operating characteristic (ROC) curves in evaluating tests for the early detection of cancer. J Natl Cancer Inst 95:511–515CrossRefGoogle Scholar
  3. Canty A, Ripley B (2016) boot: Bootstrap R (S-Plus) functions. R package version 1.3-18Google Scholar
  4. Dokoumetzidis A, Macheras P (2000) On the use of partial AUC as an early exposure metric. Eur J Pharm Sci 10:91–95CrossRefGoogle Scholar
  5. Dorfman DD, Alf E (1968) Maximum likelihood estimation of parameters of signal detection theory—a direct solution. Psychometrika 33:117–124CrossRefGoogle Scholar
  6. Dwyer AJ (1997) In pursuit of a piece of the ROC. Radiology 202:621–625Google Scholar
  7. Eng J (2005) Receiver operating characteristic analysis: a primer review article. Acad Radiol 12:909–916CrossRefGoogle Scholar
  8. Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, Mesirov JP, Coller H, Loh ML, Downing JR, Caligiuri MA, Bloomfield CD, Lander ES (1999) Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286:531–537CrossRefGoogle Scholar
  9. Hanley JA (1988) The robustness of the “binormal” assumption used in fitting ROC curves. Med Decis Mak 8:197–203CrossRefGoogle Scholar
  10. Hayashi K (2016) Asymptotic comparison of semi-supervised and supervised linear discriminant functions for heteroscedastic normal populations. Adv Data Anal Classif.  https://doi.org/10.1007/s11634-016-0266-6 CrossRefGoogle Scholar
  11. He Y, Escobar M (2008) Nonparametric statistical inference method for partial areas under receiver operating characteristic curves, with application to genomic studies. Stat Med 27:5291–5308MathSciNetCrossRefGoogle Scholar
  12. Herron JM, Bender TM, Campbell WL, Sumkin JH, Rockette HE, Gur D (2000) Effects of luminance and resolution on observer performance with chest radiographs. Radiology 215:169–174CrossRefGoogle Scholar
  13. Hillis SL, Metz CE (2012) An analytic expression for the binormal partial area under the ROC curve. Acad Radiol 19:1491–1498CrossRefGoogle Scholar
  14. Ishwaran H, Rao JS, Kogalur UB (2013) spikeslab: prediction and variable selection using spike and slab regression. R package version 1.1.5Google Scholar
  15. Komori O, Eguchi S (2010) A boosting method for maximizing the partial area under the ROC curve. BMC Bioinform 11:314CrossRefGoogle Scholar
  16. Lee LHN, Choi C, Gershkovich P, Barr AM, Horner WG, Procyshym RM (2016) Proposing the use of partial AUC as an adjunctive measure in establishing bioequivalence between deltoid and gluteal administration of long-acting injectable antipsychotics. Eur J Drug Metab Pharmacokinet 41:659–664CrossRefGoogle Scholar
  17. Li CR, Liao CT, Liu JP (2008) A non-inferiority for diagnostic accuracy based on the paired partial areas under ROC curves. Stat Med 27:1762–1776MathSciNetCrossRefGoogle Scholar
  18. Ma H, Bandos AI, Rockette HE, Gur D (2013) On use of partial area under the ROC curve for evaluation of diagnostic performance. Stat Med 32:3449–3458MathSciNetCrossRefGoogle Scholar
  19. McClish DK (1989) Analyzing a portion of the ROC curve. Med Decis Mak 9:190–195CrossRefGoogle Scholar
  20. McNeil BJ, Hanley JA (1984) Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Decis Mak 4:137–150CrossRefGoogle Scholar
  21. Metz CE (1986) ROC methodology in radiologic imaging. Invest Radiol 143:29–36Google Scholar
  22. Obuchowski NA (2005) Fundamentals of clinical research for radiologists. ROC analysis. Am J Roentgenol 184:364–372CrossRefGoogle Scholar
  23. Park SH, Goo JM, Jo CH (2004) Receiver operating characteristic (ROC) curve: practical review for radiologists. Korean J Radiol 5:11–18CrossRefGoogle Scholar
  24. Pepe MS (2003) The statistical evaluation of medical tests for classification and prediction. Oxford University Press, New YorkzbMATHGoogle Scholar
  25. Pepe MS, Longton G, Anderson GL, Schummer M (2003) Selecting differentially expressed genes from microarray experiments. Biometrics 59:133–142MathSciNetCrossRefGoogle Scholar
  26. Swets JA, Pickett RM (1982) Evaluation of diagnostic systems: methods from signal detection theory. Academic Press, New YorkGoogle Scholar
  27. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez J-C, Müller M (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinform 12:77CrossRefGoogle Scholar
  28. Thompson ML, Zuchini W (1989) On the statistical analysis of ROC curves. Stat Med 8:1277–1290CrossRefGoogle Scholar
  29. Tian L (2010) Confidence interval estimation of partial area under curve based on combined biomarkers. Comput Stat Data Anal 54:466–472MathSciNetCrossRefGoogle Scholar
  30. Walter SD (2005) The partial area under the summary ROC curve. Stat Med 24:2025–2040MathSciNetCrossRefGoogle Scholar
  31. Wang Z, Chang YCI (2011) Marker selection via maximizing the partial area under the ROC curve of linear risk scores. Biostatistics 12:369–395CrossRefGoogle Scholar
  32. Zhou XH, Obuchowski NA, McClish DK (2002) Statistical methods in diagnostic medicine. Wiley, New YorkCrossRefGoogle Scholar
  33. Zou KH, Liu A, Bandos AI, Ohno-Machado L, Rockette HE (2011) Statistical evaluation of diagnostic performance: topics in ROC analysis. Chapman & Hall/CRC Press, Boca RatonGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  1. 1.Department of Statistics and Operations Research, International Excellence Campus for Higher Education and Research “Campus Mare Nostrum”University of MurciaMurciaSpain
  2. 2.Department of Statistical SciencesSapienza University of RomeRomeItaly

Personalised recommendations