Advertisement

Can Masses of Non-Experts Train Highly Accurate Image Classifiers?

A Crowdsourcing Approach to Instrument Segmentation in Laparoscopic Images
  • Lena Maier-Hein
  • Sven Mersmann
  • Daniel Kondermann
  • Sebastian Bodenstedt
  • Alexandro Sanchez
  • Christian Stock
  • Hannes Gotz Kenngott
  • Mathias Eisenmann
  • Stefanie Speidel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8674)

Abstract

Machine learning algorithms are gaining increasing interest in the context of computer-assisted interventions. One of the bottlenecks so far, however, has been the availability of training data, typically generated by medical experts with very limited resources. Crowdsourcing is a new trend that is based on outsourcing cognitive tasks to many anonymous untrained individuals from an online community. In this work, we investigate the potential of crowdsourcing for segmenting medical instruments in endoscopic image data. Our study suggests that (1) segmentations computed from annotations of multiple anonymous non-experts are comparable to those made by medical experts and (2) training data generated by the crowd is of the same quality as that annotated by medical experts. Given the speed of annotation, scalability and low costs, this implies that the scientific community might no longer need to rely on experts to generate reference or training data for certain applications. To trigger further research in endoscopic image processing, the data used in this study will be made publicly available.

Keywords

Random Forest Training Image Majority Vote True Positive Rate Compute Tomography Colonography 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Maier-Hein, L., Mountney, P., Bartoli, A., Elhawary, H., Elson, D., Groch, A., Kolb, A., Rodrigues, M., Sorger, J., Speidel, S., Stoyanov, D.: Optical techniques for 3d surface reconstruction in computer-assisted laparoscopic surgery. Med. Image Anal. 17, 974–996 (2013)CrossRefGoogle Scholar
  2. 2.
    Ahmidi, N., Gao, Y., Béjar, B., Vedula, S.S., Khudanpur, S., Vidal, R., Hager, G.D.: String motif-based description of tool motion for detecting skill and gestures in robotic surgery. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013, Part I. LNCS, vol. 8149, pp. 26–33. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  3. 3.
    Katic, D., Wekerle, A.L., Görtler, J., Spengler, P., Bodenstedt, S., Röhl, S., Suwelack, S., Kenngott, H.G., Wagner, M., Müller-Stich, B.P., Dillmann, R., Speidel, S.: Context-aware augmented reality in laparoscopic surgery. Comp. Med. Imag. and Graph. 37(2), 174–182 (2013)CrossRefGoogle Scholar
  4. 4.
    Allan, M., Ourselin, S., Thompson, S., Hawkes, D., Kelly, J., Stoyanov, D.: Toward detection and localization of instruments in minimally invasive surgery. IEEE T. Bio-med. Eng. 60(4), 1050–1058 (2013)CrossRefGoogle Scholar
  5. 5.
    Ranard, B., Ha, Y., Meisel, Z., Asch, D., Hill, S., Becker, L., Seymour, A., Merchant, R.: Crowdsourcing - harnessing the masses to advance health and medicine, a systematic review. J. Gen. Intern. Med. 29(1), 187–203 (2014)CrossRefGoogle Scholar
  6. 6.
    Nguyen, T.B., Wang, S., Anugu, V., Rose, N., McKenna, M., Petrick, N., Burns, J.E., Summers, R.M.: Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography. Radiology 262(3), 824–833 (2012)CrossRefGoogle Scholar
  7. 7.
    Chen, C., White, L., Kowalewski, T., Aggarwal, R., Lintott, C., Comstock, B., Kuksenok, K., Aragon, C., Holst, D., Lendvay, T.: Crowd-sourced assessment of technical skills: a novel method to evaluate surgical performance. J. Surg. Res. 187, 65–71 (2014)CrossRefGoogle Scholar
  8. 8.
    Foncubierta Rodríguez, A., Müller, H.: Ground truth generation in medical imaging: A crowdsourcing-based iterative approach. In: Proceedings of the ACM Multimedia 2012 Workshop on Crowdsourcing for Multimedia, CrowdMM 2012, pp. 9–14. ACM, New York (2012)CrossRefGoogle Scholar
  9. 9.
    Chen, J.J., Menezes, N.J., Bradley, A.D., North, T.: Opportunities for crowdsourcing research on amazon mechanical turk. Interfaces 5(3) (2011)Google Scholar
  10. 10.
    Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001)CrossRefGoogle Scholar
  11. 11.
    Maier-Hein, L., et al.: Crowdsourcing for reference correspondence generation in endoscopic images. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8674, pp. 345–352. Springer, Heidelberg (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Lena Maier-Hein
    • 1
  • Sven Mersmann
    • 1
  • Daniel Kondermann
    • 2
  • Sebastian Bodenstedt
    • 3
  • Alexandro Sanchez
    • 2
  • Christian Stock
    • 4
  • Hannes Gotz Kenngott
    • 5
  • Mathias Eisenmann
    • 3
  • Stefanie Speidel
    • 3
  1. 1.Computer-assisted InterventionsGerman Cancer Research Center (DKFZ)Germany
  2. 2.Heidelberg Collaboratory for Image ProcessingUniversity of HeidelbergGermany
  3. 3.Institute for Anthropomatics and RoboticsKarlsruhe Institute of TechnologyGermany
  4. 4.Institute of Medical Biometry and InformaticsUniversity of HeidelbergGermany
  5. 5.Department of General, Visceral and Transplantation SurgeryUniversity of HeidelbergGermany

Personalised recommendations