Lifetime Data Analysis

, Volume 19, Issue 2, pp 242–256 | Cite as

Assessing discrimination of risk prediction rules in a clustered data setting

  • Bernard Rosner
  • Weiliang Qiu
  • Mei-Ling T. Lee


The AUC (area under ROC curve) is a commonly used metric to assess discrimination of risk prediction rules; however, standard errors of AUC are usually based on the Mann–Whitney U test that assumes independence of sampling units. For ophthalmologic applications, it is desirable to assess risk prediction rules based on eye-specific outcome variables which are generally highly, but not perfectly correlated in fellow eyes [e.g. progression of individual eyes to age-related macular degeneration (AMD)]. In this article, we use the extended Mann–Whitney U test (Rosner and Glynn, Biometrics 65:188–197, 2009) for the case where subunits within a cluster may have different progression status and assess discrimination of different prediction rules in this setting. Both data analyses based on progression of AMD and simulation studies show reasonable accuracy of this extended Mann–Whitney U test to assess discrimination of eye-specific risk prediction rules.


Risk prediction \(\cdot \) ROC curves \(\cdot \) Clustered data \(\cdot \) GEE 



This work was supported by the National Institutes of Health Grant EY12269 from the National Eye Institute.

Supplementary material

10985_2012_9240_MOESM1_ESM.pdf (158 kb)
Supplementary material 1 (pdf 158 KB)


  1. Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Diagn Radiol 143:29–36Google Scholar
  2. Hodges JL Jr, Lehmann EL (1956) The efficiency of some nonparametric competitors of the t test. Ann Math Stat 27:324–335MathSciNetMATHCrossRefGoogle Scholar
  3. Kendall MG, Stuart A (1969) The advanced theory of statistics. Hafner, New YorkMATHGoogle Scholar
  4. Li G, Zhou K (2008) A unified approach to nonparametric comparison of receiver operating characteristic curves for longitudinal and clustered data. J Am Stat Assoc 103(482):705–713MATHCrossRefGoogle Scholar
  5. National Eye Institute (2011) Age-related macular degeneration. Retrieved March 2011
  6. Obuchowski NA (2003) Receiver operating characteristic curves and their use in radiology. Radiology 229:3–8CrossRefGoogle Scholar
  7. Obuchowski NA, McClish DK (1997) Sample size determination for diagnostic accuracy studies involving binormal roc curve indices. Stat Med 16(13):1529–1542CrossRefGoogle Scholar
  8. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: from area under the roc curve to reclassification and beyond. Stat Med 27:157–172MathSciNetCrossRefGoogle Scholar
  9. Rosner B, Glynn RJ (2009) Power and sample size estimation for the wilcoxon rank sum test with application to comparisons of c statistics from alternative prediction models. Biometrics 65:188–97MathSciNetMATHCrossRefGoogle Scholar
  10. Rosner B, Glynn RJ, Lee MT (2006) Extension of the rank sum test for clustered data: two-group comparisons with group membership defined at the subunit level. Biometrics 62:1251–1259MathSciNetMATHCrossRefGoogle Scholar
  11. Rubin DB (1987) Multiple imputation for non-response in surveys. Wiley, New YorkCrossRefGoogle Scholar
  12. Seddon JM, Cote J, Rosner B (2003) Progression of age-related macular degeneration. Arch Ophthalmol 121:1728–1737CrossRefGoogle Scholar
  13. Toledano AY, Gatsonis CA (1996) GEEs for ordinal categorical data: arbitrary patterns of missing responses and missingness in a key covariate. Biometrics 55:488–496MathSciNetCrossRefGoogle Scholar
  14. Zou KH, O’Malley AJ, Mauri L (2007) Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation 115:654–657CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  • Bernard Rosner
    • 1
  • Weiliang Qiu
    • 1
  • Mei-Ling T. Lee
    • 2
  1. 1.Channing Division of Network MedicineBrigham and Women’s Hospital/Harvard Medical SchoolBostonUSA
  2. 2.Department of Epidemiology and BiostatisticsUniversity of MarylandCollege ParkUSA

Personalised recommendations