Application of deep learning to the diagnosis of cervical lymph node metastasis from thyroid cancer with CT

  • Jeong Hoon Lee
  • Eun Ju HaEmail author
  • Ju Han Kim
Imaging Informatics and Artificial Intelligence



To develop a deep learning–based computer-aided diagnosis (CAD) system for use in the CT diagnosis of cervical lymph node metastasis (LNM) in patients with thyroid cancer.


A total of 995 axial CT images that included benign (n = 647) and malignant (n = 348) lymph nodes were collected from 202 patients with thyroid cancer who underwent CT for surgical planning between July 2017 and January 2018. The datasets were randomly split into training (79.0%), validation (10.5%), and test (10.5%) datasets. Eight deep convolutional neural network (CNN) models were used to classify the images into metastatic or benign lymph nodes. Pretrained networks were used on the ImageNet and the best-performing algorithm was selected. Class-specific discriminative regions were visualized with attention heatmap using a global average pooling method.


The area under the ROC curve (AUROC) for the tested algorithms ranged from 0.909 to 0.953. The sensitivity, specificity, and accuracy of the best-performing algorithm were all 90.4%, respectively. Attention heatmap highlighted important subregions for further clinical review.


A deep learning–based CAD system could accurately classify cervical LNM in patients with thyroid cancer on preoperative CT with an AUROC of 0.953. Whether this approach has clinical utility will require evaluation in a clinical setting.

Key Points

• A deep learning–based CAD system could accurately classify cervical lymph node metastasis. The AUROC for the eight tested algorithms ranged from 0.909 to 0.953.

• Of the eight models, the ResNet50 algorithm was the best-performing model for the validation dataset with 0.953 AUROC. The sensitivity, specificity, and accuracy of the ResNet50 model were all 90.4%, respectively, in the test dataset.

• Based on its high accuracy of 90.4%, we consider that this model may be useful in a clinical setting to detect LNM on preoperative CT in patients with thyroid cancer.


Artificial intelligence Lymphatic metastasis Thyroid cancer Multidetector computed tomography 



Area under the receiver operating characteristic curve


Computer-aided diagnosis


Class activation mapping


Convolutional neural network


Fine needle aspiration


Lymph node metastasis


Papillary thyroid carcinoma





The authors state that this work was supported by the National Research Foundation of Korea (no. 2017R1C1B5016217).

Compliance with ethical standards


The scientific guarantor of this publication is Eun Ju Ha.

Conflict of interest

The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was obtained from all patients before they underwent US.

Ethical approval

This study was approved by our institutional review board.


• retrospective

• case-control study

Supplementary material

330_2019_6098_MOESM1_ESM.doc (11.4 mb)
ESM 1 (DOC 11717 kb)


  1. 1.
    Davies L, Welch HG (2014) Current thyroid cancer trends in the United States. JAMA Otolaryngol Head Neck Surg 140:317–322CrossRefGoogle Scholar
  2. 2.
    Li N, Du XL, Reitzel LR, Xu L, Sturgis EM (2013) Impact of enhanced detection on the increase in thyroid cancer incidence in the United States: review of incidence trends by socioeconomic status within the surveillance, epidemiology, and end results registry, 1980-2008. Thyroid 23:103–110CrossRefGoogle Scholar
  3. 3.
    Haugen BR, Alexander EK, Bible KC et al (2016) 2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid Cancer: the American Thyroid Association guidelines task force on thyroid nodules and differentiated thyroid cancer. Thyroid 26:1–133CrossRefGoogle Scholar
  4. 4.
    Al-Saif O, Farrar WB, Bloomston M, Porter K, Ringel MD, Kloos RT (2010) Long-term efficacy of lymph node reoperation for persistent papillary thyroid cancer. J Clin Endocrinol Metab 95:2187–2194CrossRefGoogle Scholar
  5. 5.
    Stack BC Jr, Ferris RL, Goldenberg D et al (2012) American Thyroid Association consensus review and statement regarding the anatomy, terminology, and rationale for lateral neck dissection in differentiated thyroid cancer. Thyroid 22:501–508CrossRefGoogle Scholar
  6. 6.
    Durante C, Montesano T, Torlontano M et al (2013) Papillary thyroid cancer: time course of recurrences during postsurgery surveillance. J Clin Endocrinol Metab 98:636–642CrossRefGoogle Scholar
  7. 7.
    Tufano RP, Clayman G, Heller KS et al (2015) Management of recurrent/persistent nodal disease in patients with differentiated thyroid cancer: a critical review of the risks and benefits of surgical intervention versus active surveillance. Thyroid 25:15–27CrossRefGoogle Scholar
  8. 8.
    Yeh MW, Bauer AJ, Bernet VA et al (2015) American Thyroid Association statement on preoperative imaging for thyroid cancer surgery. Thyroid 25:3–14CrossRefGoogle Scholar
  9. 9.
    Wong SC, Gatt A, Stamatescu V, McDonnell MD (2016) Understanding data augmentation for classification: when to warp? Accessed 19 Nov 2018
  10. 10.
    Shin JH, Baek JH, Chung J et al (2016) Ultrasonography diagnosis and imaging-based management of thyroid nodules: revised Korean Society of Thyroid Radiology consensus statement and recommendations. Korean J Radiol 17:370–395CrossRefGoogle Scholar
  11. 11.
    Suh CH, Baek JH, Choi YJ, Lee JH (2017) Performance of CT in the preoperative diagnosis of cervical lymph node metastasis in patients with papillary thyroid cancer: a systematic review and meta-analysis. AJNR Am J Neuroradiol 38:154–161CrossRefGoogle Scholar
  12. 12.
    Lee Y, Kim JH, Baek JH et al (2018) Value of CT added to ultrasonography for the diagnosis of lymph node metastasis in patients with thyroid cancer. Head Neck.
  13. 13.
    Shen D, Wu G, Suk HI (2017) Deep learning in medical image analysis. Annu Rev Biomed Eng 19:221–248CrossRefGoogle Scholar
  14. 14.
    Miotto R, Wang F, Wang S, Jiang X, Dudley JT (2017) Deep learning for healthcare: review, opportunities and challenges. Brief BioinformGoogle Scholar
  15. 15.
    LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436CrossRefGoogle Scholar
  16. 16.
    Choi YJ, Baek JH, Park HS et al (2017) A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of thyroid nodules on ultrasound: initial clinical assessment. Thyroid 27:546–552CrossRefGoogle Scholar
  17. 17.
    Wu MH, Chen CN, Chen KY et al (2016) Quantitative analysis of echogenicity for patients with thyroid nodules. Sci Rep 6:35632CrossRefGoogle Scholar
  18. 18.
    Jeong EY, Kim HL, Ha EJ et al (2018) Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators. Eur RadiolGoogle Scholar
  19. 19.
    Lee JH, Baek JH, Kim JH et al (2018) Deep learning-based computer-aided diagnosis system for localization and diagnosis of metastatic lymph nodes on ultrasound: a pilot study. Thyroid 28:1332–1338CrossRefGoogle Scholar
  20. 20.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Accessed 19 Nov 2018
  21. 21.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognitionProceedings of the IEEE conference on computer vision and. Pattern Recogn:770–778Google Scholar
  22. 22.
    Chollet F (2017) Xception: deep learning with depthwise separable convolutions. Accessed 19 Nov 2018
  23. 23.
    Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. Accessed 19 Nov 2018
  24. 24.
    Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2016) Inception-v4, inception-resnet and the impact of residual connections on learning. Accessed 19 Nov 2018
  25. 25.
    Iandola F, MoskewiczM,Karayev S, Girshick R, Darrell T, Keutzer K (2014) Densenet: Implementing efficient convnet descriptor pyramids. Accessed 19 Nov 2018
  26. 26.
    Russakovsky O, Deng J, Su H et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115:211–252CrossRefGoogle Scholar
  27. 27.
    Abadi M, Barham P, Chen J et al (2016) Tensorflow: a system for large-scale machine learning. Accessed 19 Nov 2018
  28. 28.
    Mulla MG, Knoefel WT, Gilbert J, McGregor A, Schulte KM (2012) Lateral cervical lymph node metastases in papillary thyroid cancer: a systematic review of imaging-guided and prophylactic removal of the lateral compartment. Clin Endocrinol 77:126–131CrossRefGoogle Scholar
  29. 29.
    Moon HJ, Kim EK, Yoon JH, Kwak JY (2012) Differences in the diagnostic performances of staging US for thyroid malignancy according to experience. Ultrasound Med Biol 38:568–573CrossRefGoogle Scholar
  30. 30.
    Mulla M, Schulte KM (2012) Central cervical lymph node metastases in papillary thyroid cancer: a systematic review of imaging-guided and prophylactic removal of the central compartment. Clin Endocrinol 76:131–136CrossRefGoogle Scholar
  31. 31.
    Jeong HS, Baek CH, Son YI et al (2006) Integrated 18F-FDG PET/CT for the initial evaluation of cervical node level of patients with papillary thyroid carcinoma: comparison with ultrasound and contrast-enhanced CT. Clin Endocrinol 65:402–407CrossRefGoogle Scholar
  32. 32.
    Ahn JE, Lee JH, Yi JS et al (2008) Diagnostic accuracy of CT and ultrasonography for evaluating metastatic cervical lymph nodes in patients with thyroid cancer. World J Surg 32:1552–1558CrossRefGoogle Scholar
  33. 33.
    Lee DW, Ji YB, Sung ES et al (2013) Roles of ultrasonography and computed tomography in the surgical management of cervical lymph node metastases in papillary thyroid carcinoma. Eur J Surg Oncol 39:191–196CrossRefGoogle Scholar
  34. 34.
    Ha EJ, Baek JH, Na DG (2017) Risk stratification of thyroid nodules on ultrasonography: current status and perspectives. Thyroid 27:1463–1468CrossRefGoogle Scholar
  35. 35.
    Esteva A, Kuprel B, Novoa RA et al (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115CrossRefGoogle Scholar
  36. 36.
    Lee HJ, Yoon DY, Seo YL et al (2018) Intraobserver and interobserver variability in ultrasound measurements of thyroid nodules. J Ultrasound Med 37:173–178CrossRefGoogle Scholar

Copyright information

© European Society of Radiology 2019

Authors and Affiliations

  1. 1.Division of Biomedical InformaticsSeoul National University Biomedical Informatics (SNUBI), Seoul National University College of MedicineSeoulRepublic of Korea
  2. 2.Department of RadiologyAjou University School of MedicineSuwonSouth Korea

Personalised recommendations