Post-imaging pulmonary nodule mathematical prediction models: are they clinically relevant?

  • Johanna Uthoff
  • Nicholas Koehn
  • Jared Larson
  • Samantha K. N. Dilger
  • Emily Hammond
  • Ann Schwartz
  • Brian Mullan
  • Rolando Sanchez
  • Richard M. Hoffman
  • Jessica C. SierenEmail author
  • for the COPDGene Investigators



Post-imaging mathematical prediction models (MPMs) provide guidance for the management of solid pulmonary nodules by providing a lung cancer risk score from demographic and radiologists-indicated imaging characteristics. We hypothesized calibrating the MPM risk score threshold to a local study cohort would result in improved performance over the original recommended MPM thresholds. We compared the pre- and post-calibration performance of four MPM models and determined if improvement in MPM prediction occurs as nodules are imaged longitudinally.

Materials and methods

A common cohort of 317 individuals with computed tomography-detected, solid nodules (80 malignant, 237 benign) were used to evaluate the MPM performance. We created a web-based application for this study that allows others to easily calibrate thresholds and analyze the performance of MPMs on their local cohort. Thirty patients with repeated imaging were tested for improved performance longitudinally.


Using calibrated thresholds, Mayo Clinic and Brock University (BU) MPMs performed the best (AUC = 0.63, 0.61) compared to the Veteran’s Affairs (0.51) and Peking University (0.55). Only BU had consensus with the original MPM threshold; the other calibrated thresholds improved MPM accuracy. No significant improvements in accuracy were found longitudinally between time points.


Calibration to a common cohort can select the best-performing MPM for your institution. Without calibration, BU has the most stable performance in solid nodules ≥ 8 mm but has only moderate potential to refine subjects into appropriate workup. Application of MPM is recommended only at initial evaluation as no increase in accuracy was achieved over time.

Key Points

• Post-imaging lung cancer risk mathematical predication models (MPMs) perform poorly on local populations without calibration.

• An application is provided to facilitate calibration to new study cohorts: the Mayo Clinic model, the U.S. Department of Veteran’s Affairs model, the Brock University model, and the Peking University model.

• No significant improvement in risk prediction occurred in nodules with repeated imaging sessions, indicating the potential value of risk prediction application is limited to the initial evaluation.


Risk assessment Lung neoplasms Tomography, x-ray computed Logistic models Area under the curve 



Area under the curve


British Thoracic Society


Brock University model


Chronic obstructive pulmonary disorder


Computed tomography


Median absolute deviation


Mayo Clinic model


Mathematical prediction models


Precision recall


Peking University model


Receiver-operator characteristic


Initial imaging encounter on which the pulmonary nodule was identified


Final imaging encounter before pulmonary nodule diagnosis


U.S. Department of Veterans Affairs model



We thank Kimberly Sprenger, Debra O’Connel-Moore, Mark Escher, Patrick Thalken, and Kimberly Schroeder for technical assistance.


This work was supported in part by Grant IRG-77-004-34 from the American Cancer Society, administered through the Holden Comprehensive Cancer Center at the University of Iowa. The COPDGene Study was supported by NHLBI U01 HL089897 and U01 HL089856. The COPDGene study (NCT00608764) is also supported by the COPD Foundation through contributions made to an Industry Advisory Committee comprised of AstraZeneca, Boehringer-Ingelheim, GlaxoSmithKline, Novartis, Pfizer, Siemens, and Sunovion. The INHALE study was supported by Award Number R01CA141769 and P30CA022453 from the National Cancer Institute, Health and Human Services Award HHSN26120130011I, and the Herrick Foundation.

Compliance with ethical standards


The scientific guarantor of this publication is Dr. Jessica C. Sieren.

Conflict of interest

The authors declare that they have no conflict of interest.

Statistics and biometry

The first and last authors, as biomedical engineers, have experience with biostatistics methods. No complex statistical methods were necessary for this paper.

Informed consent

The University of Iowa Institutional Review Board has approved this study (IRB 201603824). Informed consent was obtained from the research cohort participants through the parent studies, COPDgene and INHALE (including the approval of collected data for expanded research questions beyond the parent study purpose). For the retrospective clinical cohort, written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.

Study subjects or cohorts overlap

Forty subjects from study subjects or cohorts have been previously reported by our lab in a machine learning approach development [19, 20, 31].


• Retrospective

• observational

• performed at one institution

Supplementary material

330_2019_6168_MOESM1_ESM.docx (225 kb)
ESM 1 (DOCX 225 kb)


  1. 1.
    Siegel RL, Miller KD, Jemal A (2017) Cancer statistics, 2017. CA Cancer J Clin 67:7–30CrossRefGoogle Scholar
  2. 2.
    MacMahon H, Naidich DP, Goo JM et al (2017) Guidelines for management of incidental pulmonary nodules detected on CT images: from the Fleischner Society 2017. Radiology 284:228–243CrossRefGoogle Scholar
  3. 3.
    American College of Radiology (2014). Lung CT screening reporting and data system (Lung-RADS). Available via: Accessed 12/12/2018
  4. 4.
    Gould MK, Donington J, Lynch WR et al (2013) Evaluation of individuals with pulmonary nodules: when is it lung cancer? Diagnosis and management of lung cancer, 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest 143:e93S–e120SCrossRefGoogle Scholar
  5. 5.
    Mehta HJ, Mohammed TL, Jantz MA (2017) The American College of Radiology Lung Imaging Reporting and Data System: potential drawbacks and need for revision. Chest 151:539–543CrossRefGoogle Scholar
  6. 6.
    Pinsky PF, Gierada DS, Black W et al (2015) Performance of Lung-RADS in the National Lung Screening Trial: a retrospective assessment. Ann Intern Med 162:485–491CrossRefGoogle Scholar
  7. 7.
    van Riel SJ, Ciompi F, Jacobs C et al (2017) Malignancy risk estimation of screen-detected nodules at baseline CT: comparison of the PanCan model, Lung-RADS and NCCN guidelines. Eur Radiol 27:4019–4029CrossRefGoogle Scholar
  8. 8.
    Gray EP, Teare MD, Stevens J, Archer R (2016) Risk prediction models for lung cancer: a systematic review. Clin Lung Cancer 17:95–106CrossRefGoogle Scholar
  9. 9.
    Swensen SJ, Silverstein MD, Ilstrup DM, Schleck CD, Edell ES (1997) The probability of malignancy in solitary pulmonary nodules. Application to small radiologically indeterminate nodules. Arch Intern Med 157:849–855CrossRefGoogle Scholar
  10. 10.
    Gould MK, Ananth L, Barnett PG, Veterans Affairs SNAP Cooperative Study Group (2007) A clinical model to estimate the pretest probability of lung cancer in patients with solitary pulmonary nodules. Chest 131:383–388Google Scholar
  11. 11.
    McWilliams A, Tammemagi MC, Mayo JR et al (2013) Probability of cancer in pulmonary nodules detected on first screening CT. N Engl J Med 369:910–919CrossRefGoogle Scholar
  12. 12.
    Li Y, Wang J (2012) A mathematical model for predicting malignancy of solitary pulmonary nodules. World J Surg 36:830–835CrossRefGoogle Scholar
  13. 13.
    Herder GJ, van Tinteren H, Golding RP et al (2005) Clinical prediction model to characterize pulmonary nodules: validation and added value of 18F-fluorodeoxyglucose positron emission tomography. Chest 128:2490–2496CrossRefGoogle Scholar
  14. 14.
    Baldwin DR, Callister ME (2015) The British Thoracic Society guidelines on the investigation and management of pulmonary nodules. Thorax 70:794–798CrossRefGoogle Scholar
  15. 15.
    Hammer MM, Nachiappan AC, Barbosa EJM Jr (2018) Limited utility of pulmonary nodule risk calculators for managing large nodules. Curr Probl Diagn Radiol 47:23–27Google Scholar
  16. 16.
    Al-Ameri A, Malhotra P, Thygesen H et al (2015) Risk of malignancy in pulmonary nodules: a validation study of four prediction models. Lung Cancer 89:27–30CrossRefGoogle Scholar
  17. 17.
    Mehta HJ, Ravenel JG, Shaftman SR et al (2014) The utility of nodule volume in the context of malignancy prediction for small pulmonary nodules. Chest 145:464–472CrossRefGoogle Scholar
  18. 18.
    Perandini S, Soardi GA, Motton M, Montemezzi S (2015) Critique of Al-Ameri et al. (2015) - risk of malignancy in pulmonary nodules: a validation study of four prediction models. Lung Cancer 90:118–119CrossRefGoogle Scholar
  19. 19.
    Regan EA, Hokanson JE, Murphy JR et al (2010) Genetic epidemiology of COPD (COPDGene) study design. COPD 7:32–43CrossRefGoogle Scholar
  20. 20.
    Schwartz AG, Lusk CM, Wenzlaff AS et al (2016) Risk of lung cancer associated with COPD phenotype based on quantitative image analysis. Cancer Epidemiol Biomarkers Prevv 25:1341–1347Google Scholar
  21. 21.
    Swensen SJ, Silverstein MD, Edell ES et al (1999) Solitary pulmonary nodules: clinical prediction model versus physicians. Mayo Clin Proc 74:319–329CrossRefGoogle Scholar
  22. 22.
    Cummings SR, Lillington GA, Richard RJ (1986) Managing solitary pulmonary nodules. The choice of strategy is a “close call”. Am Rev Respir Dis 134:453–460Google Scholar
  23. 23.
    Chung K, Mets OM, Gerke PK et al (2018) Brock malignancy risk calculator for pulmonary nodules: validation outside a lung cancer screening population. Thorax 73:857–863CrossRefGoogle Scholar
  24. 24.
    Youden WJ (1950) Index for rating diagnostic tests. Cancer 3:32–35CrossRefGoogle Scholar
  25. 25.
    Jesse D, Goadrich M (2006) The relationship between precision-recall and ROC curves. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, 25–29 June 2006, pp 233–240Google Scholar
  26. 26.
    McNemar Q (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12:153–157CrossRefGoogle Scholar
  27. 27.
    Callister ME, Baldwin DR, Akram AR et al (2015) British Thoracic Society guidelines for the investigation and management of pulmonary nodules. Thorax 70(Suppl 2):ii1–ii54CrossRefGoogle Scholar
  28. 28.
    McNitt-Gray MF, Kim GH, Zhao B et al (2015) Determining the variability of lesion size measurements from CT patient data sets acquired under “no change” conditions. Transl Oncol 8:55–64CrossRefGoogle Scholar
  29. 29.
    Lin H, Huang C, Wang W, Luo J, Yang X, Liu Y (2017) Measuring interobserver disagreement in rating diagnostic characteristics of pulmonary nodule using the lung imaging database consortium and image database resource initiative. Acad Radiol 24:401–410CrossRefGoogle Scholar
  30. 30.
    Maiga AW, Deppen SA, Massion PP et al (2018) Communication about the probability of cancer in indeterminate pulmonary nodules. JAMA Surg 153:353–357CrossRefGoogle Scholar
  31. 31.
    Dilger SK, Uthoff J, Judisch A et al (2015) Improved pulmonary nodule classification utilizing quantitative lung parenchyma features. J Med Imaging (Bellingham) 2:041004CrossRefGoogle Scholar
  32. 32.
    Dhara AK, Mukhopadhyay S, Dutta A, Garg M, Khandelwal N (2016) A combination of shape and texture features for classification of pulmonary nodules in lung CT images. J Digit Imaging 29:466–475CrossRefGoogle Scholar
  33. 33.
    Ferreira JR Jr, Oliveira MC, de Azevedo-Marques PM (2017) Characterization of pulmonary nodules based on features of margin sharpness and texture. J Digit Imaging 31:451–463Google Scholar
  34. 34.
    Nibali A, He Z, Wollersheim D (2017) Pulmonary nodule classification with deep residual networks. Int J Comput Assist Radiol Surg 12:1799–1808CrossRefGoogle Scholar
  35. 35.
    Sun W, Zheng B, Qian W (2017) Automatic feature learning using multichannel ROI based on deep structured algorithms for computerized lung cancer diagnosis. Comput Biol Med 89:530–539CrossRefGoogle Scholar
  36. 36.
    Way TW, Sahiner B, Chan HP et al (2009) Computer-aided diagnosis of pulmonary nodules on CT scans: improvement of classification performance with nodule surface features. Med Phys 36:3086–3098CrossRefGoogle Scholar
  37. 37.
    Zhu Y, Tan Y, Hua Y, Wang M, Zhang G, Zhang J (2010) Feature selection and performance evaluation of support vector machine (SVM)-based classifier for differentiating benign and malignant pulmonary nodules by computed tomography. J Digit Imaging 23:51–65CrossRefGoogle Scholar

Copyright information

© European Society of Radiology 2019

Authors and Affiliations

  • Johanna Uthoff
    • 1
    • 2
  • Nicholas Koehn
    • 1
  • Jared Larson
    • 1
  • Samantha K. N. Dilger
    • 1
    • 2
  • Emily Hammond
    • 1
    • 2
  • Ann Schwartz
    • 3
  • Brian Mullan
    • 1
  • Rolando Sanchez
    • 4
  • Richard M. Hoffman
    • 4
  • Jessica C. Sieren
    • 1
    • 2
    Email author
  • for the COPDGene Investigators
  1. 1.Department of RadiologyUniversity of IowaIowa CityUSA
  2. 2.Department of Biomedical EngineeringUniversity of IowaIowa CityUSA
  3. 3.Karmanos Cancer InstituteWayne State UniversityDetroitUSA
  4. 4.Department of Internal MedicineUniversity of IowaIowa CityUSA

Personalised recommendations