Advertisement

Deep learning-based detection of motion artifacts in probe-based confocal laser endomicroscopy images

  • Marc Aubreville
  • Maike Stoeve
  • Nicolai Oetter
  • Miguel Goncalves
  • Christian Knipfer
  • Helmut Neumann
  • Christopher Bohr
  • Florian Stelzle
  • Andreas Maier
Original Article

Abstract

Purpose:

Probe-based confocal laser endomicroscopy (pCLE) is a subcellular in vivo imaging technique capable of producing images that enable diagnosis of malign structural modifications in epithelial tissue. Images acquired with pCLE are, however, often tainted by significant artifacts that impair diagnosis. This is especially detrimental for automated image analysis, which is why said images are often excluded from recognition pipelines.

Methods

We present an approach for the automatic detection of motion artifacts in pCLE images and apply this methodology to a data set of 15 thousand images of epithelial tissue acquired in the oral cavity and the vocal folds. The approach is based on transfer learning from intermediate endpoints within a pre-trained Inception v3 network with tailored preprocessing. For detection within the non-rectangular pCLE images, we perform pooling within the activation maps of the network and evaluate this at different network depths.

Results

We achieved area under the ROC curve values of 0.92 with the proposed method, compared to 0.80 for the best feature-based machine learning approach. Our overall accuracy with the presented approach is 94.8%.

Conclusion

Over traditional machine learning approaches with state-of-the-art features, we achieved significantly improved overall performance.

Keywords

Deep convolutional neural networks Confocal laser endomicroscopy Motion artifact detection 

Notes

Compliance with ethical standards

Conflict of interest

All authors declare that they do not have conflicts of interest regarding the work covered by this manuscript.

Human and animal participants

All procedures involving human participants were in accordance with the 1964 Declaration of Helsinki and its later amendments.

Informed consent

The local ethics committee approved the studies (ethics committee of the University of Erlangen-Nürnberg; reference numbers 243 12 B and 60 14 B), and all patients gave their written informed consent.

References

  1. 1.
    Abbaci M, Breuskin I, Casiraghi O, De Leeuw F (2014) Confocal laser endomicroscopy for non-invasive head and neck cancer imaging: a comprehensive review. Oral Oncol 50(8):711–6.  https://doi.org/10.1016/j.oraloncology.2014.05.002 CrossRefPubMedGoogle Scholar
  2. 2.
    Agaimy A, Weichert W (2016) Grading of head and neck neoplasms. Der Pathol 37(4):285–292.  https://doi.org/10.1007/s00292-016-0173-9 CrossRefGoogle Scholar
  3. 3.
    Araujo H, Dias JM (1996) An introduction to the log-polar mapping (image sampling). In: Proceedings 2nd workshop on cybernetic vision. IEEE, pp 139–144.  https://doi.org/10.1109/CYBVIS.1996.629454
  4. 4.
    Aubreville M, Knipfer C, Oetter N, Jaremenko Christian, Rodner E, Denzler J, Bohr C, Neumann H, Stelzle F, Maier A (2017) Automatic classification of cancerous tissue in laserendomicroscopy images of the oral cavity using deep learning. Sci Rep 7(1):11979.  https://doi.org/10.1038/s41598-017-12320-8 CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Bier B, Mualla F, Steidl S, Bohr C, Neumann H, Maier A, Hornegger J (2015) Band-pass filter design by segmentation in frequency domain for detection of epithelial cells in endomicroscope images. In: Bildverarbeitung für die Medizin. Springer, Berlin, pp 413–418.  https://doi.org/10.1007/978-3-662-46224-9_71
  6. 6.
    Chauhan SS, Dayyeh BKA, Bhat YM, Gottlieb KT, Hwang JH, Komanduri S, Konda V, Lo SK, Manfredi MA, Maple JT, Murad FM, Siddiqui UD, Banerjee S, Wallace MB (2014) Confocal laser endomicroscopy. Gastrointest Endosc 80(6):928–938.  https://doi.org/10.1016/j.gie.2014.06.021 CrossRefGoogle Scholar
  7. 7.
    Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. Medical Image Computing and Computer-Assisted Intervention—MICCAI 2013 16(Pt 2):411–418.  https://doi.org/10.1007/978-3-642-40763-5_51
  8. 8.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition, CVPR 2005. 1:886–893.  https://doi.org/10.1109/CVPR.2005.177
  9. 9.
    Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 248–255.  https://doi.org/10.1109/CVPR.2009.5206848
  10. 10.
    Deniz O, García GB, Salido J, De la Torre F (2011) Face recognition using histograms of oriented gradients. Pattern Recognit Lett 32(12):1598–1603.  https://doi.org/10.1016/j.patrec.2011.01.004 CrossRefGoogle Scholar
  11. 11.
    Dittberner A, Rodner E, Ortmann W, Stadler J, Schmidt C, Petersen I, Stallmach A, Denzler J, Guntinas-Lichius O (2016) Automated analysis of confocal laser endomicroscopy images to detect head and neck cancer. Head Neck 38(S1):E1419–E1426.  https://doi.org/10.1002/hed.24253 CrossRefPubMedGoogle Scholar
  12. 12.
    Gil D, Ramos-Terrades O, Minchole E, Sanchez C, de Frutos NC, Diez-Ferrer M, Ortiz RM, Rosell A (2017) Classification of confocal endomicroscopy patterns for diagnosis of lung cancer. In: Medical image computing and computer-assisted intervention—MICCAI 2017, pp. 151–159. Springer, Cham.  https://doi.org/10.1007/978-3-319-67543-5_15
  13. 13.
    Goncalves M, Iro H, Dittberner A, Agaimy A, Bohr C (2017) Value of confocal laser endomicroscopy in the diagnosis of vocal cord lesions. Eur Rev Med Pharmacol Sci 21:3990–3997PubMedGoogle Scholar
  14. 14.
    Hong J, Park By, Park H (2017) Convolutional neural network classifier for distinguishing Barrett’s esophagus and neoplasia endomicroscopy images. In: 39th Annual International conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 2892–2895.  https://doi.org/10.1109/EMBC.2017.8037461
  15. 15.
    Izadyyazdanabadi M, Belykh E, Cavallo C, Zhao X, Gandhi S, Moreira LB, Eschbacher J, Nakaji P, Preul MC, Yang Y (2018) Weakly-supervised learning-based feature localization in confocal laser endomicroscopy glioma images. arXiv preprint arXiv:1804.09428v1
  16. 16.
    Izadyyazdanabadi M, Belykh E, Martirosyan N, Eschbacher J, Nakaji P, Yang Y, Preul MC (2017)Improving utility of brain tumor confocal laser endomicroscopy - objective value assessment and diagnostic frame detection with convolutional neural networks. In: Medical Imaging 2017, vol 10134. SPIE.  https://doi.org/10.1117/12.2254902
  17. 17.
    Izadyyazdanabadi M, Belykh E, Mooney M, Eschbacher J, Nakaji P, Yang Y, Preul MC (2018) Prospects for theranostics in neurosurgical technology—empowering confocal laser endomicroscopy diagnostics via deep learning. arXiv preprint arXiv:1804.09873
  18. 18.
    Izadyyazdanabadi M, Belykh E, Mooney M, Martirosyan N, Eschbacher J, Nakaji P, Preul MC, Yang Y (2018) Convolutional neural networks: ensemble modeling, fine-tuning and unsupervised semantic localization for neurosurgical cle images. J Vis Commun Image Represent 54:10–20.  https://doi.org/10.1016/j.jvcir.2018.04.004 CrossRefGoogle Scholar
  19. 19.
    Jaremenko C, Maier A, Steidl S, Hornegger J, Oetter N, Knipfer C, Stelzle F, Neumann H (2015) Classification of confocal laser endomicroscopic images of the oral cavity to distinguish pathological from healthy tissue. In: Bildverarbeitung für die Medizin 2015. Springer, Berlin, pp 479–485.  https://doi.org/10.1007/978-3-662-46224-9_82
  20. 20.
    Kingma D, Ba J (2014) ADAM: a method for stochastic optimization. arXiv preprint arXiv:1412.6980
  21. 21.
    Laemmel E, Genet M, Le Goualher G, Perchant A, Le Gargasson JF, Vicaut E (2004) Fibered confocal fluorescence microscopy (Cell-viZio) facilitates extended imaging in the field of microcirculation. A comparison with intravital microscopy. J Vasc Res 41(5):400–411.  https://doi.org/10.1159/000081209 CrossRefPubMedGoogle Scholar
  22. 22.
    Macé V, Ahluwalia A, Coron E, Le Rhun M, Boureille A, Bossard C, Mosnier JF, Matysiak-Budnik T, Tarnawski AS (2015) Confocal laser endomicroscopy: a new gold standard for the assessment of mucosal healing in ulcerative colitis. J Gastroenterol Hepatol 30:85–92.  https://doi.org/10.1111/jgh.12748 CrossRefPubMedGoogle Scholar
  23. 23.
    Maier AK, Schebesch F, Syben C, Würfl T, Steidl S, Choi JH, Fahrig R (2017) Precision learning: towards use of known operators in neural networks. arXiv preprint arXiv:1712.00374
  24. 24.
    Maier H, Dietz A, Gewelke U, Heller W, Weidauer H (1992) Tobacco and alcohol and the risk of head and neck cancer. Clin Investig 70(3–4):320–327.  https://doi.org/10.1007/BF00184668 PubMedCrossRefGoogle Scholar
  25. 25.
    Muto M, Nakane M, Katada C, Sano Y, Ohtsu A, Esumi H, Ebihara S, Yoshida S (2004) Squamous cell carcinoma in situ at oropharyngeal and hypopharyngeal mucosal sites. Cancer 101(6):1375–1381.  https://doi.org/10.1002/cncr.20482 CrossRefPubMedGoogle Scholar
  26. 26.
    Nachalon Y, Alkan U, Shvero J, Yaniv D, Shkedy Y, Limon D, Popovtzer A (2017) Assessment of laryngeal cancer in patients younger than 40 years. The Laryngoscope.  https://doi.org/10.1002/lary.26951 CrossRefPubMedGoogle Scholar
  27. 27.
    Neumann H, Langner C, Neurath MF, Vieth M (2012) Confocal laser endomicroscopy for diagnosis of Barrett’s Esophagus. Front Oncol 2:42.  https://doi.org/10.3389/fonc.2012.00042 CrossRefPubMedPubMedCentralGoogle Scholar
  28. 28.
    Neumann H, Vieth M, Atreya R, Neurath MF, Mudter J (2011) Prospective evaluation of the learning curve of confocal laser endomicroscopy in patients with IBD. Histol Histopathol 26(7):867–872PubMedGoogle Scholar
  29. 29.
    Oetter N, Knipfer C, Rohde M, Wilmowsky C, Maier A, Brunner K, Adler W, Neukam FW, Neumann H, Stelzle F (2016) Development and validation of a classification and scoring system for the diagnosis of oral squamous cell carcinomas through confocal laser endomicroscopy. J Transl Med 14(1):1–11.  https://doi.org/10.1186/s12967-016-0919-4 CrossRefGoogle Scholar
  30. 30.
    Oquab M, Bottou L, Laptev I, Sivic J (2015) Is object localization for free? Weakly-supervised learning with convolutional neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition. IEEE, pp 685–694.  https://doi.org/10.1109/CVPR.2015.7298668
  31. 31.
    Parikh ND, Gibson J, Nagar A, Ahmed AA, Aslanian HR (2016) Confocal laser endomicroscopy features of sessile serrated adenomas/polyps. U Eur Gastroenterol J 4(4):599–603.  https://doi.org/10.1177/2050640615621819 CrossRefGoogle Scholar
  32. 32.
    Pavlov V, Meyronet D, Meyer-Bisch V, Armoiry X, Pikul B, Dumot C, Beuriat P-A, Signorelli F, Guyotat J (2016) Intraoperative probe-based confocal laser endomicroscopy in surgery and stereotactic biopsy of low-grade and high-grade gliomas. Neurosurgery 79(4):604–612.  https://doi.org/10.1227/NEU.0000000000001365 CrossRefPubMedGoogle Scholar
  33. 33.
    Robert Koch Institut. Zentrum für Krebsregisterdaten (2017) Krebs in Deutschland für 2013/2014, 11th edn. Robert Koch Institut, BerlinGoogle Scholar
  34. 34.
    Stoeve M, Aubreville M, Oetter N, Knipfer C, Neumann H, Stelzle F, Maier A (2018) Motion Artifact Detection in Confocal Laser Endomicroscopy Images. In: Bildverarbeitung für die Medizin. Springer, Berlin, pp 328–333.  https://doi.org/10.1007/978-3-662-56537-7_85
  35. 35.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: The IEEE conference on computer vision and pattern recognition (CVPR).  https://doi.org/10.1109/CVPR.2015.7298594
  36. 36.
    Vo K, Jaremenko Christian, Bohr C, Neumann H, Maier A (2017) Automatic classification and pathological staging of confocal laser endomicroscopic images of the vocal cords. In: Bildverarbeitung für die Medizin 2017. Springer, Berlin, pp 312–317.  https://doi.org/10.1007/978-3-662-54345-0_70
  37. 37.
    Westra WH (2015) The pathology of HPV-related head and neck cancer: implications for the diagnostic pathologist. Semin Diagn Pathol 32(1):42–53.  https://doi.org/10.1053/j.semdp.2015.02.023 CrossRefPubMedPubMedCentralGoogle Scholar
  38. 38.
    Wiesner C, Jäger W, Salzer A, Biesterfeld S, Kiesslich R, Hampel C, Thüroff JW, Goetz M (2010) Confocal laser endomicroscopy for the diagnosis of urothelial bladder neoplasia: a technology of the future? BJU Int 107(3):399–403.  https://doi.org/10.1111/j.1464-410X.2010.09540.x CrossRefPubMedGoogle Scholar
  39. 39.
    Xing F, Xie Y, Su H, Liu F, Yang L (2017) Deep learning in microscopy image analysis: a survey. IEEE Trans Neural Netw Learn Syst PP(99):1–19.  https://doi.org/10.1109/TNNLS.2017.2766168
  40. 40.
    Yang Y, Li T, Li W, Wu H, Fan W, Zhang W (2017) Lesion detection and grading of diabetic retinopathy via two-stages deep convolutional neural networks. In: Medical image computing and computer-assisted intervention—MICCAI 2017. Springer, Cham, pp 533–540.  https://doi.org/10.1007/978-3-319-66179-7_61

Copyright information

© CARS 2018

Authors and Affiliations

  1. 1.Pattern Recognition Lab, Computer ScienceFriedrich-Alexander-Universität Erlangen-NürnbergErlangenGermany
  2. 2.Department of Oral and Maxillofacial Surgery, University Hospital ErlangenFriedrich-Alexander-Universität Erlangen-NürnbergErlangenGermany
  3. 3.Department of Otorhinolaryngology, Head and Neck Surgery, University Hospital ErlangenFriedrich-Alexander-Universität Erlangen-NürnbergErlangenGermany
  4. 4.Department of Oral and Maxillofacial Surgery, University Medical Center Hamburg-EppendorfUniversität HamburgHamburgGermany
  5. 5.Department of Otorhinolaryngology, Head and Neck Surgery, University HospitalUniversität RegensburgRegensburgGermany
  6. 6.First Department of Internal Medicine, University Hospital MainzJohannes Gutenberg-Universität MainzMainzGermany

Personalised recommendations