Cancer Causes & Control

, Volume 30, Issue 10, pp 1145–1155 | Cite as

Improving the diagnostic accuracy of a stratified screening strategy by identifying the optimal risk cutoff

  • John T. BrintonEmail author
  • R. Edward Hendrick
  • Brandy M. Ringham
  • Mieke Kriege
  • Deborah H. Glueck
Original Paper



The American Cancer Society (ACS) suggests using a stratified strategy for breast cancer screening. The strategy includes assessing risk of breast cancer, screening women at high risk with both MRI and mammography, and screening women at low risk with mammography alone. The ACS chose their cutoff for high risk using expert consensus.


We propose instead an analytic approach that maximizes the diagnostic accuracy (AUC/ROC) of a risk-based stratified screening strategy in a population. The inputs are the joint distribution of screening test scores, and the odds of disease, for the given risk score. Using the approach for breast cancer screening, we estimated the optimal risk cutoff for two different risk models: the Breast Cancer Screening Consortium (BCSC) model and a hypothetical model with much better discriminatory accuracy. Data on mammography and MRI test score distributions were drawn from the Magnetic Resonance Imaging Screening Study Group.


A risk model with an excellent discriminatory accuracy (c-statistic \(= 0.947\)) yielded a reasonable cutoff where only about 20% of women had dual screening. However, the BCSC risk model (c-statistic \(= 0.631\)) lacked the discriminatory accuracy to differentiate between women who needed dual screening, and women who needed only mammography.


Our research provides a general approach to optimize the diagnostic accuracy of a stratified screening strategy in a population, and to assess whether risk models are sufficiently accurate to guide stratified screening. For breast cancer, most risk models lack enough discriminatory accuracy to make stratified screening a reasonable recommendation.


Cancer screening Stratified screening Risk assessment ROC analysis 



This manuscript was submitted to the Department of Biostatistics and Informatics in the Colorado School of Public Health, University of Colorado Denver, in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Biostatistics for JTB. Partial funding for DHG was provided by a generous grant from the Lundbeck Foundation, who provided a visiting professorship to the University of Copenhagen. The authors thank the BCSC investigators, participating mammography facilities, and radiologists who provided the relevant data for this study. A list of the BCSC investigators and procedures for requesting BCSC data for research purposes are provided at:


Funding was provided by Lundbeckfonden, National Cancer Institute (Grant Nos. 5K07CA088811, 1R03CA136048-01A1), and National Institute of Dental and Craniofacial Research (Grant No. RC2DE020779).


  1. 1.
    Aberle DR, Adams AM, Berg CD, Black WC, Clapp JD, Fagerstrom RM, Gareen IF, Gatsonis C, Marcus PM, Sicks JD (2011) Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med 365(5):395–409CrossRefGoogle Scholar
  2. 2.
    Amir E, Evans DG, Shenton A, Lalloo F, Moran A, Boggis C, Wilson M, Howell A (2003) Evaluation of breast cancer risk assessment packages in the family history evaluation and screening programme. J Med Genet 40(11):807–814CrossRefGoogle Scholar
  3. 3.
    Antoniou AC, Pharoah PPD, Smith P, Easton DF (2004) The BOADICEA model of genetic susceptibility to breast and ovarian cancer. Br J Cancer 91(8):1580–1590. CrossRefGoogle Scholar
  4. 4.
    Armstrong AC, Evans GD (2014) Management of women at high risk of breast cancer. BMJ 348(apr28 26):g2756–g2756. CrossRefGoogle Scholar
  5. 5.
    Atkin WS, Edwards R, Kralj-Hans I, Wooldrage K, Hart AR, Northover JMA, Parkin DM, Wardle J, Duffy SW, Cuzick J, UK Flexible Sigmoidoscopy Trial Investigators (2010) Once-only flexible sigmoidoscopy screening in prevention of colorectal cancer: a multicentre randomised controlled trial. Lancet (London, England) 375(9726):1624–1633. CrossRefGoogle Scholar
  6. 6.
    Baker SG (2000) Identifying combinations of cancer markers for further study as triggers of early intervention. Biometrics 56(4):1082–1087CrossRefGoogle Scholar
  7. 7.
    Barlow WE, White E, Ballard-Barbash R, Vacek PM, Titus-Ernstoff L, Carney PA, Tice JA, Buist DSM, Geller BM, Rosenberg R, Yankaskas BC, Kerlikowske K (2006) Prospective breast cancer risk prediction model for women undergoing screening mammography. J Natl Cancer Inst 98(17):1204–1214. CrossRefGoogle Scholar
  8. 8.
    Berry DA, Iversen ES Jr, Gudbjartsson DF, Hiller EH, Garber JE, Peshkin BN, Lerman C, Watson P, Lynch HT, Hilsenbeck SG, Rubinstein WS, Hughes KS, Parmigiani G (2002) BRCAPRO validation, sensitivity of genetic testing of BRCA1/BRCA2, and prevalence of other breast cancer susceptibility genes. J Clin Oncol 20(11):2701–2712CrossRefGoogle Scholar
  9. 9.
    Brawley O, Byers T, Chen A, Pignone M, Ransohoff D, Schenk M, Smith R, Sox H, Thorson AG, Wender R (2011) New American Cancer Society process for creating trustworthy cancer screening guidelines. J Am Med Assoc 306(22):2495–2499CrossRefGoogle Scholar
  10. 10.
    Claus E (2000) Risk models in genetic epidemiology. Stat Methods Med Res 9(6):589–601CrossRefGoogle Scholar
  11. 11.
    D’Souza G, Pawlita M, Westra WH (2007) Case–control study of human papillomavirus and oropharyngeal cancer. N Engl J Med 356(19):1944–1956CrossRefGoogle Scholar
  12. 12.
    Elmore JG, Barton MB, Moceri VM, Polk S, Arena PJ, Fletcher SW (1998) Ten-year risk of false positive screening mammograms and clinical breast examinations. N Engl J Med 338(16):1089–1096CrossRefGoogle Scholar
  13. 13.
    Gail MH, Pfeiffer RM (2005) On criteria for evaluating models of absolute risk. Biostatistics 6(2):227–239. CrossRefGoogle Scholar
  14. 14.
    Hagen AI, Kvistad KA, Maehle L, Holmen MM, Aase H, Styr B, Vabø A, Apold J, Skaane P, Møller P (2007) Sensitivity of MRI versus conventional screening in the diagnosis of BRCA-associated breast cancer in a national prospective series. Breast (Edinburgh, Scotland) 16(4):367–374. CrossRefGoogle Scholar
  15. 15.
    Hartman AR, Daniel BL, Kurian AW, Mills MA, Nowels KW, Dirbas FM, Kingham KE, Chun NM, Herfkens RJ, Ford JM, Plevritis SK (2004) Breast magnetic resonance image screening and ductal lavage in women at high genetic risk for breast carcinoma. Cancer 100(3):479–489. CrossRefGoogle Scholar
  16. 16.
    Hendrick RE, Smith RA, Rutledge JH, Smart CR (1997) Benefit of screening mammography in women aged 40–49: a new meta-analysis of randomized controlled trials. J Natl Cancer Inst 22:87–92CrossRefGoogle Scholar
  17. 17.
    Hosmer DW, Lemeshow S (2000) Applied logistic regression, 2nd edn. Wiley series in probability and statistics. Wiley, New YorkCrossRefGoogle Scholar
  18. 18.
    Kriege M, Brekelmans CTM, Boetes C, Besnard PE, Zonderland HM, Obdeijn IM, Manoliu RA, Kok T, Peterse H, Tilanus-Linthorst MMA, Muller SH, Meijer S, Oosterwijk JC, Beex LVAM, Tollenaar RAEM, de Koning HJ, Rutgers EJT, Klijn JGM (2004) Efficacy of MRI and mammography for breast-cancer screening in women with a familial or genetic predisposition. N Engl J Med 351(5):427–437CrossRefGoogle Scholar
  19. 19.
    Kuhl CK, Schrading S, Leutner CC, Morakkabati-Spitz N, Wardelmann E, Fimmers R, Kuhn W, Schild HH (2005) Mammography, breast ultrasound, and magnetic resonance imaging for surveillance of women at high familial risk for breast cancer. J Clin Oncol 23(33):8469–8476CrossRefGoogle Scholar
  20. 20.
    Leach MO, Boggis CRM, Dixon AK, Easton DF, Eeles RA, Evans DGR, Gilbert FJ, Griebsch I, Hoff RJC, Kessar P, Lakhani SR, Moss SM, Nerurkar A, Padhani AR, Pointon LJ, Thompson D, Warren RML (2005) Screening with magnetic resonance imaging and mammography of a UK population at high familial risk of breast cancer. Lancet 365(9473):1769–1778CrossRefGoogle Scholar
  21. 21.
    Lehman CD (2012) Diffusion weighted imaging (DWI) of the breast: ready for clinical practice. Eur J Radiol 81(Suppl 1):S80–81. CrossRefGoogle Scholar
  22. 22.
    Lehman CD, Blume JD, Weatherall P, Thickman D, Hylton N, Warner E, Pisano E, Schnitt SJ, Gatsonis C, Schnall M, DeAngelis GA, Stomper P, Rosen EL, O’Loughlin M, Harms S, Bluemke DA (2005) Screening women at high risk for breast cancer with mammography and magnetic resonance imaging. Cancer 103(9):1898–1905CrossRefGoogle Scholar
  23. 23.
    Ma H, Bandos AI, Gur D (2015) On the use of partial area under the ROC curve for comparison of two diagnostic tests. Biom J 57(2):304–320. CrossRefGoogle Scholar
  24. 24.
    Ma H, Bandos AI, Rockette HE, Gur D (2013) On use of partial area under the ROC curve for evaluation of diagnostic performance. Stat Med 32(20):3449–3458. CrossRefGoogle Scholar
  25. 25.
    McFarland EG, Levin B, Lieberman DA, Pickhardt PJ, Johnson CD, Glick SN, Brooks D, Smith RA (2008) Revised colorectal screening guidelines: joint effort of the American Cancer Society, U.S. Multisociety Task Force on Colorectal Cancer, and American College of Radiology. Radiology 248(3):717–720CrossRefGoogle Scholar
  26. 26.
    Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P (2004) Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 159(9):882–890. CrossRefGoogle Scholar
  27. 27.
    Pisano ED, Gatsonis C, Hendrick E, Yaffe M, Baum JK, Acharyya S, Conant EF, Fajardo LL, Bassett L, D’Orsi C, Jong R, Rebner M (2005) Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med 353(17):1773–1783CrossRefGoogle Scholar
  28. 28.
    Quante AS, Whittemore AS, Shriver T, Hopper JL, Strauch K, Terry MB (2015) Practical problems with clinical guidelines for breast cancer prevention based on remaining lifetime risk. J Natl Cancer Inst 107(7):djv124. CrossRefGoogle Scholar
  29. 29.
    Ross S (1984) First course in probability, 2nd edn. Macmillan Publishing Company, New YorkGoogle Scholar
  30. 30.
    Sankaranarayanan R, Ramadas K, Thomas G, Muwonge R, Thara S, Mathew B, Rajan B (2005) Effect of screening on oral cancer mortality in Kerala, India: a cluster-randomised controlled trial. Lancet 365(9475):1927–1933CrossRefGoogle Scholar
  31. 31.
    Sardanelli F, Aase HS, Álvarez M, Azavedo E, Baarslag HJ, Balleyguier C, Baltzer PA, Beslagic V, Bick U, Bogdanovic-Stojanovic D, Briediene R, Brkljacic B, Camps Herrero J, Colin C, Cornford E, Danes J, de Geer G, Esen G, Evans A, Fuchsjaeger MH, Gilbert FJ, Graf O, Hargaden G, Helbich TH, Heywang-Köbrunner SH, Ivanov V, Jónsson Á, Kuhl CK, Lisencu EC, Luczynska E, Mann RM, Marques JC, Martincich L, Mortier M, Müller-Schimpfle M, Ormandi K, Panizza P, Pediconi F, Pijnappel RM, Pinker K, Rissanen T, Rotaru N, Saguatti G, Sella T, Slobodníková J, Talk M, Taourel P, Trimboli RM, Vejborg I, Vourtsis A, Forrai G (2017) Position paper on screening for breast cancer by the European Society of Breast Imaging (EUSOBI) and 30 national breast radiology bodies from Austria, Belgium, Bosnia and Herzegovina, Bulgaria, Croatia, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Israel, Lithuania, Moldova, The Netherlands, Norway, Poland, Portugal, Romania, Serbia, Slovakia, Spain, Sweden, Switzerland and Turkey. Eur Radiol 27(7):2737–2743. CrossRefGoogle Scholar
  32. 32.
    Sardanelli F, Podo F, Santoro F, Manoukian S, Bergonzi S, Trecate G, Vergnaghi D, Federico M, Cortesi L, Corcione S, Morassut S, Di Maggio C, Cilotti A, Martincich L, Calabrese M, Zuiani C, Preda L, Bonanni B, Carbonaro LA, Contegiacomo A, Panizza P, Di Cesare E, Savarese A, Crecco M, Turchetti D, Tonutti M, Belli P, Maschio AD (2011) High breast cancer risk Italian 1 (HIBCRIT-1) Study: multicenter surveillance of women at high genetic breast cancer risk using mammography, ultrasonography, and contrast-enhanced magnetic resonance imaging (the high breast cancer risk italian 1 study): final results. Investig Radiol 46(2):94–105. CrossRefGoogle Scholar
  33. 33.
    Saslow D, Boetes C, Burke W, Harms S, Leach MO, Lehman CD, Morris E, Pisano E, Schnall M, Sener S, Smith RA, Warner E, Yaffe M, Andrews KS, Russell CA (2007) American Cancer Society guidelines for breast screening with MRI as an adjunct to mammography. CA 57(2):75–89Google Scholar
  34. 34.
    Sickles EA, D’Orsi CJ, Bassett LW (2013) ACR BI-RADS—mammographyGoogle Scholar
  35. 35.
    Smith RA, Andrews K, Brooks D, DeSantis CE, Fedewa SA, Lortet-Tieulent J, Manassaram-Baptiste D, Brawley OW, Wender RC (2016) Cancer screening in the United States, 2016: a review of current American Cancer Society guidelines and current issues in cancer screening. CA 66(2):96–114. Google Scholar
  36. 36.
    Smith RA, Cokkinides V, Brawley OW (2012) Cancer screening in the United States, 2012: a review of current American Cancer Society guidelines and current issues in cancer screening. CA 62(2):129–142Google Scholar
  37. 37.
    Thisted RA (1988) Elements of statistical computing: numerical computation, 1st edn. Chapman and Hall/CRC, Boca RatonGoogle Scholar
  38. 38.
    Tosteson ANA, Stout NK, Fryback DG, Acharyya S, Herman BA, Hannah LG, Pisano ED, DMIST Investigators (2008) Cost-effectiveness of digital mammography breast cancer screening. Ann Intern Med 148(1):1–10CrossRefGoogle Scholar
  39. 39.
    Tyrer J, Duffy SW, Cuzick J (2004) A breast cancer prediction model incorporating familial and personal risk factors. Stat Med 23(7):1111–1130CrossRefGoogle Scholar
  40. 40.
    Wald NJ, Hackshaw AK, Frost CD (1999) When can a risk factor be used as a worthwhile screening test? BMJ 319(7224):1562–1565. CrossRefGoogle Scholar
  41. 41.
    Wang Z, Luo X, Chang YcI (2015) Assessing the predictive power of newly added biomarkers. Biom J 57(5):797–807. CrossRefGoogle Scholar
  42. 42.
    Warner E (2008) The role of magnetic resonance imaging in screening women at high risk of breast cancer. Top Magn Reson Imaging 19(3):163–169. CrossRefGoogle Scholar
  43. 43.
    Warner E, Plewes DB, Hill KA, Causer PA, Zubovits JT, Jong RA, Cutrara MR, DeBoer G, Yaffe MJ, Messner SJ, Meschino WS, Piron CA, Narod SA (2004) Surveillance of BRCA1 and BRCA2 mutation carriers with magnetic resonance imaging, ultrasound, mammography, and clinical breast examination. JAMA 292(11):1317–1325. CrossRefGoogle Scholar
  44. 44.
    Yabuuchi H, Matsuo Y, Sunami S, Kamitani T, Kawanami S, Setoguchi T, Sakai S, Hatakenaka M, Kubo M, Tokunaga E, Yamamoto H, Honda H (2011) Detection of non-palpable breast cancer in asymptomatic women by using unenhanced diffusion-weighted and T2-weighted MR imaging: comparison with mammography and dynamic contrast-enhanced MR imaging. Eur Radiol 21(1):11–17. CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Biostatistics and InformaticsColorado School of Public HealthAuroraUSA
  2. 2.Department of Radiology, School of MedicineUniversity of Colorado DenverAuroraUSA
  3. 3.Lifecourse Epidemiology of Adiposity and Diabetes (LEAD) CenterUniversity of Colorado DenverAuroraUSA
  4. 4.Department of Medical OncologyErasmus MC Cancer InstituteRotterdamThe Netherlands
  5. 5.Department of PediatricsUniversity of Colorado School of MedicineAuroraUSA

Personalised recommendations