Post-imaging pulmonary nodule mathematical prediction models: are they clinically relevant?
- 52 Downloads
Post-imaging mathematical prediction models (MPMs) provide guidance for the management of solid pulmonary nodules by providing a lung cancer risk score from demographic and radiologists-indicated imaging characteristics. We hypothesized calibrating the MPM risk score threshold to a local study cohort would result in improved performance over the original recommended MPM thresholds. We compared the pre- and post-calibration performance of four MPM models and determined if improvement in MPM prediction occurs as nodules are imaged longitudinally.
Materials and methods
A common cohort of 317 individuals with computed tomography-detected, solid nodules (80 malignant, 237 benign) were used to evaluate the MPM performance. We created a web-based application for this study that allows others to easily calibrate thresholds and analyze the performance of MPMs on their local cohort. Thirty patients with repeated imaging were tested for improved performance longitudinally.
Using calibrated thresholds, Mayo Clinic and Brock University (BU) MPMs performed the best (AUC = 0.63, 0.61) compared to the Veteran’s Affairs (0.51) and Peking University (0.55). Only BU had consensus with the original MPM threshold; the other calibrated thresholds improved MPM accuracy. No significant improvements in accuracy were found longitudinally between time points.
Calibration to a common cohort can select the best-performing MPM for your institution. Without calibration, BU has the most stable performance in solid nodules ≥ 8 mm but has only moderate potential to refine subjects into appropriate workup. Application of MPM is recommended only at initial evaluation as no increase in accuracy was achieved over time.
• Post-imaging lung cancer risk mathematical predication models (MPMs) perform poorly on local populations without calibration.
• An application is provided to facilitate calibration to new study cohorts: the Mayo Clinic model, the U.S. Department of Veteran’s Affairs model, the Brock University model, and the Peking University model.
• No significant improvement in risk prediction occurred in nodules with repeated imaging sessions, indicating the potential value of risk prediction application is limited to the initial evaluation.
KeywordsRisk assessment Lung neoplasms Tomography, x-ray computed Logistic models Area under the curve
Area under the curve
British Thoracic Society
Brock University model
Chronic obstructive pulmonary disorder
Median absolute deviation
Mayo Clinic model
Mathematical prediction models
Peking University model
Initial imaging encounter on which the pulmonary nodule was identified
Final imaging encounter before pulmonary nodule diagnosis
U.S. Department of Veterans Affairs model
We thank Kimberly Sprenger, Debra O’Connel-Moore, Mark Escher, Patrick Thalken, and Kimberly Schroeder for technical assistance.
This work was supported in part by Grant IRG-77-004-34 from the American Cancer Society, administered through the Holden Comprehensive Cancer Center at the University of Iowa. The COPDGene Study was supported by NHLBI U01 HL089897 and U01 HL089856. The COPDGene study (NCT00608764) is also supported by the COPD Foundation through contributions made to an Industry Advisory Committee comprised of AstraZeneca, Boehringer-Ingelheim, GlaxoSmithKline, Novartis, Pfizer, Siemens, and Sunovion. The INHALE study was supported by Award Number R01CA141769 and P30CA022453 from the National Cancer Institute, Health and Human Services Award HHSN26120130011I, and the Herrick Foundation.
Compliance with ethical standards
The scientific guarantor of this publication is Dr. Jessica C. Sieren.
Conflict of interest
The authors declare that they have no conflict of interest.
Statistics and biometry
The first and last authors, as biomedical engineers, have experience with biostatistics methods. No complex statistical methods were necessary for this paper.
The University of Iowa Institutional Review Board has approved this study (IRB 201603824). Informed consent was obtained from the research cohort participants through the parent studies, COPDgene and INHALE (including the approval of collected data for expanded research questions beyond the parent study purpose). For the retrospective clinical cohort, written informed consent was waived by the Institutional Review Board.
Institutional Review Board approval was obtained.
Study subjects or cohorts overlap
• performed at one institution
- 3.American College of Radiology (2014). Lung CT screening reporting and data system (Lung-RADS). Available via: https://www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Lung-Rads. Accessed 12/12/2018
- 10.Gould MK, Ananth L, Barnett PG, Veterans Affairs SNAP Cooperative Study Group (2007) A clinical model to estimate the pretest probability of lung cancer in patients with solitary pulmonary nodules. Chest 131:383–388Google Scholar
- 15.Hammer MM, Nachiappan AC, Barbosa EJM Jr (2018) Limited utility of pulmonary nodule risk calculators for managing large nodules. Curr Probl Diagn Radiol 47:23–27Google Scholar
- 20.Schwartz AG, Lusk CM, Wenzlaff AS et al (2016) Risk of lung cancer associated with COPD phenotype based on quantitative image analysis. Cancer Epidemiol Biomarkers Prevv 25:1341–1347Google Scholar
- 22.Cummings SR, Lillington GA, Richard RJ (1986) Managing solitary pulmonary nodules. The choice of strategy is a “close call”. Am Rev Respir Dis 134:453–460Google Scholar
- 25.Jesse D, Goadrich M (2006) The relationship between precision-recall and ROC curves. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, 25–29 June 2006, pp 233–240Google Scholar
- 33.Ferreira JR Jr, Oliveira MC, de Azevedo-Marques PM (2017) Characterization of pulmonary nodules based on features of margin sharpness and texture. J Digit Imaging 31:451–463Google Scholar