Skip to main content
SpringerLink
Log in

Radiology Objects in COntext (ROCO): A Multimodal Image Dataset

  • Conference paper
  • First Online:
Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis (LABELS 2018, CVII 2018, STENT 2018)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11043))

Abstract

This work introduces a new multimodal image dataset, with the aim of detecting the interplay between visual elements and semantic relations present in radiology images. The objective is accomplished by retrieving all image-caption pairs from the open-access biomedical literature database PubMedCentral, as these captions describe the visual content in their semantic context. All compound, multi-pane, and non-radiology images were eliminated using an automatic binary classifier fine-tuned with a deep convolutional neural network system. Radiology Objects in COntext (ROCO) dataset contains over 81k radiology images with several medical imaging modalities including Computer Tomography, Ultrasound, X-Ray, Fluoroscopy, Positron Emission Tomography, Mammography, Magnetic Resonance Imaging, Angiography. All images in ROCO have corresponding caption, keywords, Unified Medical Language Systems Concept Unique Identifiers and Semantic Type. An out-of-class set with 6k images ranging from synthetic radiology figures to digital arts is provided, to improve prediction and classification performance. Adopting ROCO, systems for caption and keywords generation can be modeled, which allows multimodal representation for datasets lacking text representation. Systems with the goal of image structuring and semantic information tagging can be created using ROCO, which is beneficial and of assistance for image and information retrieval purposes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/razorx89/roco-dataset.

  2. 2.

    https://github.com/tensorflow/models/research/slim [last access: 09.04.2018].

References

  1. Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, Berkeley, CA, USA. USENIX Association (2016)

    Google Scholar 

  2. Bird, S., Klein, E., Loper, E.: Natural Language Processing with Python. O’Reilly (2009). http://www.oreilly.de/catalog/9780596516499/index.html

  3. Bodenreider, O.: The unified medical language system (UMLS): integrating biomedical terminology. Nucl. Acids Res. 32(Database–Issue), 267–270 (2004). https://doi.org/10.1093/nar/gkh061

    Article  Google Scholar 

  4. García Seco de Herrera, A., Kalpathy-Cramer, J., Demner Fushman, D., Antani, S., Müller, H.: Overview of the ImageCLEF 2013 medical tasks. In: Working Notes of CLEF 2013 - Conference and Labs of the Evaluation forum. CEUR-WS Proceedings Notes, vol. 1179, Valencia, Spain, 23–26 September 2013 (2013)

    Google Scholar 

  5. García Seco de Herrera, A., Müller, H., Bromuri, S.: Overview of the ImageCLEF 2015 medical classification task. In: Working Notes of CLEF 2015 - Conference and Labs of the Evaluation Forum. CEUR-WS Proceedings Notes, vol. 1391, Toulouse, France, 8–11 September 2015 (2015)

    Google Scholar 

  6. García Seco de Herrera, A., Schaer, R., Bromuri, S., Müller, H.: Overview of the ImageCLEF 2016 medical task. In: Working Notes of CLEF 2016 - Conference and Labs of the Evaluation forum, Évora. CEUR-WS Proceedings Notes, vol. 1609, Portugal, 5–8 September 2016 (2016)

    Google Scholar 

  7. Koitka, S., Friedrich, C.M.: Optimized convolutional neural network ensembles for medical subfigure classification. In: Jones, G.J.F., et al. (eds.) CLEF 2017. LNCS, vol. 10456, pp. 57–68. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1_5

    Chapter  Google Scholar 

  8. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, vol. 1, pp. 1097–1105. Curran Associates Inc., USA (2012)

    Google Scholar 

  9. LeCun, Y., Bengio, Y., Hinton, G.E.: Deep learning. Nature 521(7553), 436–444 (2015). https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  10. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part V. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  11. Müller, H., Kalpathy-Cramer, J., Demner-Fushman, D., Antani, S.: Creating a classification of image types in the medical literature for visual categorization. In: Proceedings of SPIE 8319, Medical Imaging 2012: Advanced PACS-Based Imaging Informatics and Therapeutic Applications, 83190P, 23 February 2012, vol. 8425, p. 194 (2012). https://doi.org/10.1117/12.911186

  12. Müller, H., et al.: Overview of the CLEF 2010 medical image retrieval track. In: CLEF 2010 LABs and Workshops, Notebook Papers. CEUR-WS Proceedings Notes, vol. 1176, Padua, Italy, 22–23 September 2010 (2010). http://ceur-ws.org/Vol-1176/CLEF2010wn-ImageCLEF-MullerEt2010.pdf

  13. Pelka, O., Friedrich, C.M.: FHDO biomedical computer science group at medical classification task of ImageCLEF 2015. In: Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum, Toulouse, France, 8–11 September 2015 (2015). http://ceur-ws.org/Vol-1391/14-CR.pdf

  14. Pelka, O., Friedrich, C.M.: Keyword generation for biomedical image retrieval with recurrent neural networks. In: Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum. CEUR-WS Proceedings Notes, vol. 1866, Dublin, Ireland, 11–14 September 2017 (2017)

    Google Scholar 

  15. Pelka, O., Nensa, F., Friedrich, C.M.: Adopting semantic information of grayscale radiographs for image classification and retrieval. In: Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018). BIOIMAGING, vol. 2, Funchal, Madeira, Portugal, 19–21 January 2018, pp. 179–187 (2018). https://doi.org/10.5220/0006732301790187

  16. Porter, M.F.: Snowball: a language for stemming algorithms (2001). http://www.webcitation.org/6yci04ExR

  17. Rajpurkar, P., et al.: Chexnet: radiologist-level pneumonia detection on chest x-rays with deep learning. CoRR abs/1711.05225 (2017). https://arxiv.org/abs/1711.05225

  18. Roberts, R.J.: PubMed central: the GenBank of the published literature. Proc. Natl. Acad. Sci. U.S.A. 98(2), 381–382 (2001). https://doi.org/10.1073/pnas.98.2.381

    Article  MathSciNet  Google Scholar 

  19. Russakovsky, O.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  20. Soldaini, L., Goharian, N.: QuickUMLS: a fast, unsupervised approach for medical concept extraction. In: MedIR Workshop, SIGIR (2016)

    Google Scholar 

  21. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-First AAAI Conference on Artificial Intelligence (2017). https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14806

  22. Tommasi, T., Caputo, B., Welter, P., Güld, M.O., Deserno, T.M.: Overviewof the CLEF 2009 medical image annotation track. In: Multilingual Information Access Evaluation II. Multimedia Experiments - 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, 30 September–2 October 2009, pp. 85–93 (2009). https://doi.org/10.1007/978-3-642-15751-6_9

  23. Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: lessons learned from the 2015 MSCOCO image captioning challenge. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 652–663 (2017). https://doi.org/10.1109/TPAMI.2016.2587640

    Article  Google Scholar 

  24. Xu, Y., Mo, T., Feng, Q., Zhong, P., Lai, M., Chang, E.I.: Deep learning of feature representation with multiple instance learning for medical image analysis. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2014, Florence, Italy, 4–9 May 2014, pp. 1626–1630 (2014). https://doi.org/10.1109/ICASSP.2014.6853873

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Applied Sciences and Arts Dortmund (FHDO), Dortmund, Germany

    Obioma Pelka, Sven Koitka, Johannes Rückert & Christoph M. Friedrich

  2. Department of Computer Science, TU Dortmund University, Dortmund, Germany

    Sven Koitka

  3. University of Duisburg-Essen, University Hospital Essen, Essen, Germany

    Obioma Pelka

  4. Department of Diagnostic and Interventional Radiology and Neuroradiology, Essen, Germany

    Sven Koitka & Felix Nensa

  5. Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), Essen, Germany

    Christoph M. Friedrich

Authors
  1. Obioma Pelka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sven Koitka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Johannes Rückert
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Felix Nensa
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christoph M. Friedrich
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Obioma Pelka .

Editor information

Editors and Affiliations

  1. University College London, London, UK

    Danail Stoyanov

  2. University of Leeds, Leeds, UK

    Zeike Taylor

  3. University of Barcelona, Barcelona, Spain

    Simone Balocco

  4. University of Bern, Bern, Switzerland

    Raphael Sznitman

  5. Sunnybrook Health Science Centre, Toronto, ON, Canada

    Anne Martel

  6. German Cancer Research Center, Heidelberg, Baden-Württemberg, Germany

    Lena Maier-Hein

  7. Ecole de Technologie Superieure, Montreal, QC, Canada

    Luc Duong

  8. Technical University of München, Garching, Bayern, Germany

    Guillaume Zahnd

  9. Technical University Munich, Munich, Germany

    Stefanie Demirci

  10. Technical University of München, Garching, Bayern, Germany

    Shadi Albarqouni

  11. Imperial College London, London, UK

    Su-Lin Lee

  12. University College London, London, UK

    Stefano Moriconi

  13. Eindhoven University of Technology, Eindhoven, The Netherlands

    Veronika Cheplygina

  14. Technical University of Munich, Garching, Germany

    Diana Mateus

  15. University of Dundee, Dundee, UK

    Emanuele Trucco

  16. École de Technologie Supérieure, Montréal, QC, Canada

    Eric Granger

  17. Universite de Rennes 1, Rennes, France

    Pierre Jannin

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pelka, O., Koitka, S., Rückert, J., Nensa, F., Friedrich, C.M. (2018). Radiology Objects in COntext (ROCO): A Multimodal Image Dataset. In: Stoyanov, D., et al. Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis. LABELS CVII STENT 2018 2018 2018. Lecture Notes in Computer Science(), vol 11043. Springer, Cham. https://doi.org/10.1007/978-3-030-01364-6_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-01364-6_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-01363-9

  • Online ISBN: 978-3-030-01364-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation