Spotting malignancies from gastric endoscopic images using deep learning

  • Jang Hyung Lee
  • Young Jae Kim
  • Yoon Woo Kim
  • Sungjin Park
  • Youn-i Choi
  • Yoon Jae Kim
  • Dong Kyun Park
  • Kwang Gi KimEmail author
  • Jun-Won ChungEmail author



Gastric cancer is a common kind of malignancies, with yearly occurrences exceeding one million worldwide in 2017. Typically, ulcerous and cancerous tissues develop abnormal morphologies through courses of progression. Endoscopy is a routinely adopted means for examination of gastrointestinal tract for malignancy. Early and timely detection of malignancy closely correlate with good prognosis. Repeated presentation of similar frames from gastrointestinal tract endoscopy often weakens attention for practitioners to result in true patients missed out to incur higher medical cost and unnecessary morbidity. Highly needed is an automatic means for spotting visual abnormality and prompts for attention for medical staff for more thorough examination.


We conduct classification of benign ulcer and cancer for gastrointestinal endoscopic color images using deep neural network and transfer-learning approach. Using clinical data gathered from Gil Hospital, we built a dataset comprised of 200 normal, 367 cancer, and 220 ulcer cases, and applied the inception, ResNet, and VGGNet models pretrained on ImageNet. Three classes were defined—normal, benign ulcer, and cancer, and three separate binary classifiers were built—those for normal vs cancer, normal vs ulcer, and cancer vs ulcer for the corresponding classification tasks. For each task, considering inherent randomness entailed in the deep learning process, we performed data partitioning and model building experiments 100 times and averaged the performance values.


Areas under curves of respective receiver operating characteristics were 0.95, 0.97, and 0.85 for the three classifiers. The ResNet showed the highest level of performance. The cases involving normal, i.e., normal vs ulcer and normal vs cancer resulted in accuracies above 90%. The case of ulcer vs cancer classification resulted in a lower accuracy of 77.1%, possibly due to smaller difference in appearance than those cases involving normal.


The overall level of performance of the proposed method was very promising to encourage applications in clinical environments. Automatic classification using deep learning technique as proposed can be used to complement manual inspection efforts for practitioners to minimize dangers of missed out positives resulting from repetitive sequence of endoscopic frames and weakening attentions.


Gastrointestinal malignancy Endoscopy Ulcer Cancer Deep learning Neural network ResNet 



This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (2018-2-00861, Intelligent SW Technology Development for Medical Data Analysis) and the Gachon University Gil Medical Center (Grant No: 2018-5283). Authors Kim KG and Chung JW equally contributed to this work.


The authors state that this work has not received any funding.

Compliance with ethical standards


Authors Jang Hyung Lee, Young Jae Kim, Yoon Woo Kim, Sungjin Park, Yoon Yi Choi, Yoon Jae Kim, Dong Kyun Park, Kwang Gi Kim, and Jun-Won Chung have no financial arrangement or affiliation with any product or services used or discussed in this paper, nor any potential bias against another product or service. The authors declare that there are no conflicts of interest regarding the publication of this paper.

Supplementary material

464_2019_6677_MOESM1_ESM.docx (15 kb)
Supplementary material 1 (DOCX 14 KB)
464_2019_6677_MOESM2_ESM.docx (15 kb)
Supplementary material 2 (DOCX 14 KB)
464_2019_6677_MOESM3_ESM.docx (17 kb)
Supplementary material 3 (DOCX 16 KB)


  1. 1.
    World Cancer Report 2014 (2014) World Health OrganizationGoogle Scholar
  2. 2. Accessed 23 Jan 2019
  3. 3.
    Watanabe K, Nagata N, Shimbo T, Nakashima R, Furuhata E, Sakurai T, Akazawa N, Yokoi C, Kobayakawa M, Akiyama J, Mizokami M, Uemura N (2013) Accuracy of endoscopic diagnosis of Helicobacter pylori infection according to level of endoscopic experience and the effect of training. BMC Gastroenterol 13:128CrossRefGoogle Scholar
  4. 4.
    Menon S, Trudgill N (2014) How commonly is upper gastrointestinal cancer missed at endoscopy? A meta-analysis. Endosc Int Open 2(2):E46–E50CrossRefGoogle Scholar
  5. 5.
    Almadi MA, Sewitch M, Barkun AN (2015) Adenoma detection rates decline with increasing procedural hours in an endoscopist’s workload. Can J Gastroenterol Hepatol 29(6):304–308CrossRefGoogle Scholar
  6. 6.
    LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444CrossRefGoogle Scholar
  7. 7.
    Miyaki R, Yoshida S, Tanaka S, Kominami Y, Sanomura Y, Matsuo T, Oka S, Raytchev B, Tamaki T, Koide T, Kaneda K, Yoshihara M, Chayama K (2015) A computer system to be used with laser-based endoscopy for quantitative diagnosis of early gastric cancer. J Clin Gastroenterol 49(2):108–115CrossRefGoogle Scholar
  8. 8.
    Zhang X, Hu W, Chen F, Liu J, Yang Y, Wang L, Duan H, Si J (2017) Gastric precancerous diseases classification using CNN with a concise model. PLoS ONE 12(9):e0185508CrossRefGoogle Scholar
  9. 9.
    Zhu R, Zhang R, Xue D (2015) Gastric precancerous diseases classification using CNN with a concise model. In: 2015 8th International Congress on Image and Signal Processing (CISP) 14–16 OctGoogle Scholar
  10. 10.
    Billah M, Waheed S, Rahman MM (2017) An automatic gastrointestinal polyp detection system in video endoscopy using fusion of color wavelet and convolutional neural network features. Int J Biomed Imaging. Google Scholar
  11. 11.
    Byrne MF, Chapados N, Soudan F, Oertel C, Linares PM, Kelly R, Iqbal N, Chandelier F, Rex DK (2017) Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut 68:94–100CrossRefGoogle Scholar
  12. 12.
    Tajbakhsh N (2015) Automatic polyp detection in colonoscopy videos using an ensemble of convolutional neural networks. In: IEEE 12th International Symposium on Biomedical Imaging (ISBI)Google Scholar
  13. 13.
    Li B, Meng MQ (2009) Computer-aided detection of bleeding regions for capsule endoscopy images. IEEE Trans Biomed Eng 56(4):1032–1039CrossRefGoogle Scholar
  14. 14.
    Tamai N, Saito Y, Sakamoto T, Nakajima T, Matsuda T, Sumiyama K, Tajiri H, Koyama R, Kido S (2017) Effectiveness of computer-aided diagnosis of colorectal lesions using novel software for magnifying narrow-band imaging: a pilot study. Endosc Int Open 5(8):E690–E694CrossRefGoogle Scholar
  15. 15.
    Clements LM, Kockelman KM (2017) Economic effects of automated vehicles. J Transp Res Board 2606:2606–2614CrossRefGoogle Scholar
  16. 16.
    Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Driessche G, Schrittwieser J (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529:484–489CrossRefGoogle Scholar
  17. 17.
    Gerhardus D (2003) Robot-assisted surgery: the future is here. J Healthc Manag 48(4):242–251CrossRefGoogle Scholar
  18. 18.
    Maroulis DE, Savelonas MA, Karkanis SA, Iakovidis DK, Dimitropoulos N (2005) Computer-aided thyroid nodule detection in ultrasound images. Proceedings of the 18th IEEE symposium on computer-based medical systems (CBMS’05)Google Scholar
  19. 19.
    Pizer SM, Amburn EP, Austin JD, Cromartie R, Geselowitz A, Greer T, Romeny B, Zimmerman JB, Zuiderveld K (1987) Adaptive histogram equalization and its variations. Comput Vis Graph Image Process 39(3):355–368CrossRefGoogle Scholar
  20. 20.
    Zuiderveld K (1994) Contrast limited adaptive histogram equalization. Graphics gems IV. Academic Press Professional, Inc., San Diego, pp 474–485Google Scholar
  21. 21.
    Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) ImageNet: a large-scale hierarchical image database. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp. 248–255Google Scholar
  22. 22.
    Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. NIPS 1:1097–1105Google Scholar
  23. 23.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9Google Scholar
  24. 24.
    He K, Zhang C, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778Google Scholar
  25. 25.
    Canziani A, Paszke A, Culurciello E (2016) An analysis of deep neural network models for practical applications. ArXiv:1605.07678Google Scholar
  26. 26.
    Dauphin GMY, Glorot X, Rifai S, Bengio Y, Goodfellow I, Lavoie E, Muller X, Desjardins G, Warde-Farley D, Vincent P, Courville A, Bergstra J (2012) Unsupervised and transfer learning challenge: a deep learning approach. JMLR 27:97–110Google Scholar
  27. 27.
    Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639):115–118CrossRefGoogle Scholar
  28. 28.
    Pogorelov K, Randel KR, Griwodz C, Eskeland SL, de Lange T, Johansen D, Spampinato C, Nguyen DTD, Lux M, Schmidt PT, Riegler M, Halvorsen P (2017) Kvasir: a multi-class image dataset for computer aided gastrointestinal disease detection. In: Proceedings of the 8th ACM on Multimedia Systems Conference, Pages 164–169Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Biomedical Engineering, College of MedicineGachon UniversityIncheonSouth Korea
  2. 2.Department of Gastroenterology, Gachon-Gil Hospital, College of MedicineGachon UniversityIncheonSouth Korea
  3. 3.Department of Gastroenterology, Gil Medical Center, School of MedicineGachon UniversityIncheonSouth Korea

Personalised recommendations