European Radiology

, Volume 29, Issue 3, pp 1518–1526 | Cite as

Development and validation of an ultrasound-based nomogram to improve the diagnostic accuracy for malignant thyroid nodules

  • Bao-liang Guo
  • Fu-sheng Ouyang
  • Li-zhu Ouyang
  • Zi-wei Liu
  • Shao-jia Lin
  • Wei Meng
  • Xi-yi Huang
  • Hai-xiong Chen
  • Shao-ming Yang
  • Qiu-gen HuEmail author



The aim of this study was to develop an ultrasound-based nomogram to improve the diagnostic accuracy of the identification of malignant thyroid nodules.


A total of 1675 histologically proven thyroid nodules (1169 benign, 506 malignant) were included in this study. The nodules were grouped into the training dataset (n = 700), internal validation dataset (n = 479), or external validation dataset (n = 496). The grayscale ultrasound features included the nodule size, shape, aspect ratio, echogenicity, margins, and calcification pattern. We applied least absolute shrinkage and selection operator (lasso) regression to select the strongest features for the nomogram. Nomogram discrimination (area under the receiver operating characteristic curve, AUC) and calibration were assessed. The nomogram was subjected to bootstrapping validation (1000 bootstrap resamples) to calculate a mean AUC and 95% confidence interval (CI).


The nomogram showed good discrimination in the training dataset, with an AUC of 0.936 (95% CI: 0.918–0.953) and good calibration. Application of the nomogram to the internal validation dataset also resulted in good discrimination (AUC: 0.935; 95% CI, 0.915–0.954) and good calibration. The model tested in an external validation dataset demonstrated a lower AUC of 0.782 (95% CI: 0.776–0.789).


This ultrasound-based nomogram can be used to quantify the probability of malignant thyroid nodules.

Key Points

• Ultrasound examination is helpful in the differential diagnosis of malignant and benign thyroid nodules.

• However, ultrasound accuracy relies heavily on examiner experience.

• A less subjective diagnostic model is desired, and the developed nomogram for thyroid nodules showed good discrimination and good calibration.


Thyroid nodule Ultrasonography Diagnosis Nomogram Area Under Curve 



Area under the curve


Confidence interval


Fine-needle aspiration


Least absolute shrinkage and selection operator


Standard deviation



This study has received funding from the science and technology project of Foshan (2017AB003623 and 2017AB003683).

Compliance with ethical standards


The scientific guarantor of this publication is Qiugen Hu

Conflict of interest

The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.


• retrospective

• diagnostic study

• performed at two institutions


  1. 1.
    Yoon JH, Lee HS, Kim EK, Moon HJ, Kwak JY (2016) Malignancy Risk Stratification of Thyroid Nodules: Comparison between the Thyroid Imaging Reporting and Data System and the 2014 American Thyroid Association Management Guidelines. Radiology 278(3):917–924CrossRefGoogle Scholar
  2. 2.
    Gharib H, Papini E, Paschke R et al (2010) American Association of Clinical Endocrinologists, Associazione Medici Endocrinologi, and European Thyroid Association Medical Guidelines for Clinical Practice for the Diagnosis and Management of Thyroid Nodules. Endocr Pract 16(3):468–475CrossRefGoogle Scholar
  3. 3.
    Xing M (2016) Clinical utility of RAS mutations in thyroid cancer: a blurred picture now emerging clearer. BMC Med 14:12CrossRefGoogle Scholar
  4. 4.
    Burman KD, Wartofsky L (2015) CLINICAL PRACTICE. Thyroid Nodules. N Engl J Med 373(24):2347–2356CrossRefGoogle Scholar
  5. 5.
    Haugen BR, Alexander EK, Bible KC et al (2016) 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid 26(1):1–133CrossRefGoogle Scholar
  6. 6.
    Kim D, Kim DW, Heo YJ et al (2017) Computed Tomography Features of Benign and Malignant Calcified Thyroid Nodules: A Single-Center Study. J Comput Assist Tomogr 41(6):937–940CrossRefGoogle Scholar
  7. 7.
    Shi RY, Yao QY, Zhou QY et al (2017) Preliminary study of diffusion kurtosis imaging in thyroid nodules and its histopathologic correlation. Eur Radiol 27(11):4710–4720CrossRefGoogle Scholar
  8. 8.
    Pathak KA, Goertzen AL, Nason RW, Klonisch T, Leslie WD (2016) A prospective cohort study to assess the role of FDG-PET in differentiating benign and malignant follicular neoplasms. Ann Med Surg (Lond) 12:27–31CrossRefGoogle Scholar
  9. 9.
    Raue F, Frank-Raue K (2016) Thyroid Cancer: Risk-Stratified Management and Individualized Therapy. Clin Cancer Res 22(10):5012–5021CrossRefGoogle Scholar
  10. 10.
    Kaur K, Sonkhya N, Bapna AS, Mital P (2002) A comparative study of fine needle aspiration cytology, ultrasonography and radionuclide scan in the management of solitary thyroid nodule: A prospective analysis of fifty cases. Indian J Otolaryngol Head Neck Surg 54(2):96–101Google Scholar
  11. 11.
    Tae HJ, Lim DJ, Baek KH et al (2007) Diagnostic value of ultrasonography to distinguish between benign and malignant lesions in the management of thyroid nodules. Thyroid 17(5):461–466CrossRefGoogle Scholar
  12. 12.
    Oberije C, Nalbantov G, Dekker A et al (2014) A prospective study comparing the predictions of doctors versus models for treatment outcome of lung cancer patients: a step toward individualized care and shared decision making. Radiother Oncol 112(1):37–43CrossRefGoogle Scholar
  13. 13.
    Iasonos A, Schrag D, Raj GV, Panageas KS (2008) How to build and interpret a nomogram for cancer prognosis. J Clin Oncol 26:1364–1370CrossRefGoogle Scholar
  14. 14.
    Abu-Rustum NR, Zhou Q, Gomez JD et al (2010) A nomogram for predicting overall survival of women with endometrial cancer following primary therapy: toward improving individualized cancer care. Gynecol Oncol 116:399–403CrossRefGoogle Scholar
  15. 15.
    Sternberg CN (2006) Are nomograms better than currently available stage groupings for bladder cancer? J Clin Oncol 24:3819–3820CrossRefGoogle Scholar
  16. 16.
    Kumamaru KK, Saboo SS, Aghayev A et al (2016) CT pulmonary angiography-based scoring system to predict the prognosis of acute pulmonary embolism. J Cardiovasc Comput Tomogr 10(6):473–479CrossRefGoogle Scholar
  17. 17.
    Vasquez MM, Hu C, Roe DJ, Chen Z, Halonen M, Guerra S (2016) Least absolute shrinkage and selection operator type methods for the identification of serum biomarkers of overweight and obesity: simulation and application. BMC Med Res Methodol 16(1):154CrossRefGoogle Scholar
  18. 18.
    Werga P, Wallin G, Skoog L, Hamberger B (2000) Expanding role of fine-needle aspiration cytology in thyroid diagnosis and management. World J Surg 24:907–912CrossRefGoogle Scholar
  19. 19.
    Theoharis CG, Schofield KM, Hammers L, Udelsman R, Chhieng DC (2009) The Bethesda thyroid fine-needle aspiration classification system: year 1 at an academic institution. Thyroid 19(11):1215–1223CrossRefGoogle Scholar
  20. 20.
    Mathur A, Weng J, Moses W et al (2010) A prospective study evaluating the accuracy of using combined clinical factors and candidate diagnostic markers to refine the accuracy of thyroid fine needle aspiration biopsy. Surgery 148(6):1170–1176CrossRefGoogle Scholar
  21. 21.
    Zhang F, Zhao X, Han R, Du M, Li P, Ji X (2017) Comparison of Acoustic Radiation Force Impulse Imaging and Strain Elastography in Differentiating Malignant From Benign Thyroid Nodules. J Ultrasound Med 36(12):2533–2543CrossRefGoogle Scholar
  22. 22.
    Chandramohan A, Therese M, Abhraham D, Paul TV, Mazhuvanchary PJ (2017) Can ARFI elastography be used to differentiate parathyroid from thyroid lesions? J Endocrinol Invest 41(1):111–119CrossRefGoogle Scholar
  23. 23.
    Zhan J, Diao XH, Chen L, Jin JM, Chen Y (2017) Role of Contrast-Enhanced Ultrasound in Diagnosis of Thyroid Nodules in Acoustic Radiation Force Impulse “Gray Zone”. Ultrasound Med Biol 43(6):1179–1186CrossRefGoogle Scholar
  24. 24.
    Colakoglu B, Yildirim D, Alis D et al (2016) Elastography in Distinguishing Benign from Malignant Thyroid Nodules. J Clin Imaging Sci 6:51CrossRefGoogle Scholar
  25. 25.
    Zahir ST, Vakili M, Ghaneei A, Sharahjin NS, Heidari F (2016) Ultrasound Assistance In Differentiating Malignant Thyroid Nodules From Benign Ones. J Ayub Med Coll Abbottabad 28(4):644–649Google Scholar
  26. 26.
    Iqbal M, Mehmood Z, Rasul S, Inamullah H, Shah SS, Bokhari I (2010) Carcinoma thyroid in multi and uninodular goiter. J Coll Physicians Surg Pak 20(5):310–312Google Scholar
  27. 27.
    Chammas MC, de Araujo Filho VJ, Moysés RA et al (2008) Predictive value for malignancy in the finding of microcalcifications on ultrasonography of thyroid nodules. Head Neck 30(9):1206–1210Google Scholar
  28. 28.
    Rahimi M, Farshchian N, Rezaee E, Shahebrahimi K, Madani H (2013) To differentiate benign from malignant thyroid nodule comparison of sonography with FNAC findings. Pak J Med Sci 29(1):77–80Google Scholar
  29. 29.
    Kim EK, Park CS, Chung WY et al (2002) New sonographic criteria for recommending fine-needle aspiration biopsy of nonpalpable solid nodules of the thyroid. AJR Am J Roentgenol 178:687–691CrossRefGoogle Scholar
  30. 30.
    Nikiforov YE, Seethala RR, Tallini G et al (2016) Nomenclature Revision for Encapsulated Follicular Variant of Papillary Thyroid Carcinoma: A Paradigm Shift to Reduce Overtreatment of Indolent Tumors. JAMA Oncol 2(8):1023–1029CrossRefGoogle Scholar

Copyright information

© European Society of Radiology 2018

Authors and Affiliations

  1. 1.Department of RadiologyShunde Hospital of Southern Medical University (The First People’s Hospital of Shunde)FoshanPeople’s Republic of China
  2. 2.Department of UltrasoundShunde Hospital of Southern Medical University (The First People’s Hospital of Shunde)FoshanPeople’s Republic of China
  3. 3.Department of LaboratoryLecong Hospital of ShundeFoshanPeople’s Republic of China

Personalised recommendations