Skip to main content
SpringerLink
Log in

Can a Machine Learn from Radiologists’ Visual Search Behaviour and Their Interpretation of Mammograms—a Deep-Learning Study

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

Visual search behaviour and the interpretation of mammograms have been studied for errors in breast cancer detection. We aim to ascertain whether machine-learning models can learn about radiologists’ attentional level and the interpretation of mammograms. We seek to determine whether these models are practical and feasible for use in training and teaching programmes. Eight radiologists of varying experience levels in reading mammograms reviewed 120 two-view digital mammography cases (59 cancers). Their search behaviour and decisions were captured using a head-mounted eye-tracking device and software allowing them to record their decisions. This information from radiologists was used to build an ensembled machine-learning model using top-down hierarchical deep convolution neural network. Separately, a model to determine type of missed cancer (search, perception or decision-making) was also built. Analysis and comparison of variants of these models using different convolution networks with and without transfer learning were also performed. Our ensembled deep-learning network architecture can be trained to learn about radiologists’ attentional level and decisions. High accuracy (95%, p value ≅ 0 [better than dumb/random model]) and high agreement between true and predicted values (kappa = 0.83) in such modelling can be achieved. Transfer learning techniques improve by < 10% with the performance of this model. We also show that spatial convolution neural networks are insufficient in determining the type of missed cancers. Ensembled hierarchical deep convolution machine-learning models are plausible in modelling radiologists’ attentional level and their interpretation of mammograms. However, deep convolution networks fail to characterise the type of false-negative decisions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Abbreviations

CC:

Craniocaudal view

MLO:

Mediolateral oblique view

ET:

Eye tracking

VSM:

Visual search map

FA:

Foveal area

PA:

Peripheral area

NFA:

Never fixated area

TP:

True positives

FP:

False positives

TN:

True negatives

FN:

False negatives

MLM:

Machine learning models

ConvNet:

Deep convolution neural network

ResNet:

Residual network

NASNet:

Neural architecture search network

VGGNet:

Visual geometry group network

iALD:

(Eye) Attentional level and decision

MC:

Missed cancer

References

  1. AIHW: Cancer in Australia 2017," in Cancer series no. 101. Cat. No. CAN 100. Canberra: AIHW, 2017

  2. (May, 2018). Australian Institute of Health and Welfare 2017. Australian Cancer Incidence and Mortality (ACIM) books: Breast Cancer. Available: https://www.aihw.gov.au/reports/cancer/acim-books

  3. S. I. Ferlay J, Ervik M, Dikshit R, Eser S, Mathers C, Rebelo M, Parkin DM, Forman D, Bray, F. (2014, 16/1/2015). GLOBOCAN 2012 v1.1, Cancer Incidence and Mortality Worldwide: IARC Cancer Base No. 11 [Internet]. Available: http://globocan.iarc.fr

  4. S. Mall, S. Lewis, P. Brennan, J. Noakes, and C. Mello-Thoms, "The role of digital breast tomosynthesis in the breast assessment clinic: a review, " Journal of Medical Radiation Sciences, pp. n/a-n/a, 2017.

  5. Nelson HD, Fu R, Cantor A, Pappas M, Daeges M, Humphrey L: Effectiveness of breast cancer screening: systematic review and meta-analysis to update the 2009 U.S. preventive services task force recommendation. Annals of Internal Medicine 164(4):244, 2016

    Article  PubMed  Google Scholar 

  6. Huynh PT, Jarolimek AM, Daye S: The false-negative mammogram. Radiographics 18(5):1137–1154, 1998

    Article  CAS  PubMed  Google Scholar 

  7. Alakhras M, Bourne R, Rickard M, Ng KH, Pietrzyk M, Brennan PC: Digital tomosynthesis: a new future for breast imaging? Clinical Radiology 68(5):e225–e236, 2013–May 2013

    Article  CAS  PubMed  Google Scholar 

  8. Kundel HL, Nodine CF, Toto L: Searching for lung nodules. The guidance of visual scanning," (in eng). Invest Radiol 26(9):777–781, Sep 1991

  9. Tuddenham WJ: Visual search, image organization, and reader error in roentgen diagnosis. Radiology 78(5):694–704, 1962

    Article  CAS  PubMed  Google Scholar 

  10. Kundel HL, Lafollet PS: Visual search patterns and experience with radiological images. Radiology 103(3):523, 1972

    Article  CAS  PubMed  Google Scholar 

  11. Mello-Thoms C et al.: Different search patterns and similar decision outcomes: how can experts agree in the decisions they make when reading digital mammograms? In: Krupinski EA Ed.. Lecture Notes in Computer ScienceDigital Mammography, Proceedings, Vol. 5116, 2008, pp. 212–219

    Chapter  Google Scholar 

  12. Krupinski EA: Visual scanning patterns of radiologists searching mammograms. Academic Radiology 3(2):137–144, Feb 1996

    Article  CAS  PubMed  Google Scholar 

  13. Kundel HL, Nodine CF, Conant EF, Weinstein SP: Holistic component of image perception in mammogram interpretation: gaze-tracking study. Radiology 242(2):396–402, Feb 2007

    Article  PubMed  Google Scholar 

  14. Kundel HL, Nodine CF, Carmody D: Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. (in eng), Invest Radiol 13(3):175–181, May-Jun 1978

    Article  CAS  Google Scholar 

  15. Nodine CF, Kundel HL: Using eye movements to study visual search and to improve tumor detection. Radiographics: a Review Publication of the Radiological Society of North America, Inc. 7(6):1241–1250, 1987–Nov 1987

    Article  CAS  Google Scholar 

  16. Kundel HL, Nodine CF, Krupinski EA: Searching for lung nodules—visual dwell indicates locations of false-positive and false-negative decisions. Investigative Radiology 24(6):472–478, Jun 1989

    Article  CAS  PubMed  Google Scholar 

  17. Mello-Thoms C, Dunn S, Nodine CF, Kundel HL: Image structure and perceptual errors in mammogram reading: a pilot study. In: Krupinski EA Ed.. (Proceedings of the Society of Photo-Optical Instrumentation Engineers (Spie), no. 26)Medical Imaging 2000: Image Perception and Performance, Vol. 1, 2000, pp. 170–173

  18. Nodine CF, Mello-Thoms C, Weinstein SP, Kundel HL, Toto LC: Do subtle breast cancers attract visual attention during initial impression? In: Krupinski EA Ed.. Ed. (Proceedings of the Society of Photo-Optical Instrumentation Engineers (Spie), no. 26)Medical Imaging 2000: Image Perception and Performance, Vol. 1, 2000, pp. 156–159

    Chapter  Google Scholar 

  19. Mall S, Brennan P, Mello-Thoms C: Fixated and not fixated regions of mammograms: a higher-order statistical analysis of visual search behavior. Academic Radiology 24(4):442–455, 2017

    Article  PubMed  Google Scholar 

  20. Mello-Thoms C, Dunn S, Nodine CF, Kundel HL, Weinstein SP: The perception of breast cancer: what differentiates missed from reported cancers in mammography? Academic Radiology 9(9):1004–1012, Sep 2002

    Article  PubMed  Google Scholar 

  21. Mello-Thoms C, Dunn SM, Nodine CF, Kundel HL: The perception of breast cancers—a spatial frequency analysis of what differentiates missed from reported cancers. Ieee Transactions on Medical Imaging 22(10):1297–1306, Oct 2003

    Article  PubMed  Google Scholar 

  22. Mello-Thoms C, Nodine CF, Kundel HL: Relating image based features to mammogram interpretation. In: Medical Imaging 2002 Conference, San Diego, CA, 2002, Vol. e4686, 2002, pp. 80–83

    Google Scholar 

  23. Berbaum KS et al.: The influence of clinical history on visual-search with single and multiple abnormalities. Investigative Radiology 28(3):191–201, Mar 1993

    Article  CAS  PubMed  Google Scholar 

  24. Samei E, Krupinski EA: The Handbook of Medical Image Perception and Techniques (no. Book, Whole). Cambridge: Cambridge University Press, 2010

    Google Scholar 

  25. Mall S, Brennan PC, Mello-Thoms C: A deep (learning) dive into visual search behaviour of breast radiologists. SPIE Medical Imaging 10577:11, 2018 SPIE

    Google Scholar 

  26. Hillstrom AP: Repetition effects in visual search," (in eng). Percept Psychophys 62(4):800–817, May 2000

  27. Kok EM, Jarodzka H, de Bruin ABH, BinAmir HAN, Robben SGF, van Merriënboer JJG: Systematic viewing in radiology: seeing more, missing less? Advances in Health Sciences Education 21:189–205, 07/16 2016

    Article  PubMed  Google Scholar 

  28. D. M Mount, S. Arya, S. E. Kemp, and G. Jefferis. (2015). Fast Nearest Neighbour Search (Wraps Arya and Mount's ANN: A Library for Approximate Nearest Neighbor Searching). Available: https://cran.r-project.org/web/packages/RANN/RANN.pdf and https://www.cs.umd.edu/~mount/ANN/

  29. X. Z. Kaiming He, Shaoqing Ren, Jian Sun, "Deep Residual Learning for Image Recognition," vol. arXiv:1512.03385, no. https://arxiv.org/abs/1512.03385, 2015.

    Google Scholar 

  30. C. Szegedy, Ioffe, S., Vanhoucke, V., "Inception-v4, Inception-resnet and the Impact of Residual Connections on Learning," vol. arXiv:1602.07261, no. https://arxiv.org/abs/1602.07261, 2016.

    Google Scholar 

  31. V. V. Barret Zoph, Jonathon Shlens, Quoc V. Le, "Learning Transferable Architectures for Scalable Image Recognition," vol. arXiv:1707.07012, no. https://arxiv.org/pdf/1707.07012.pdf.

  32. A. Z. Karen Simonyan, "Very Deep Convolutional Networks for Large-Scale Image Recognition," vol. arXiv:1409.1556, no. https://arxiv.org/abs/1409.1556, 2014.

    Google Scholar 

  33. Arel I, Rose DC, Karnowski TP: Research frontier: deep machine learning—a new frontier in artificial intelligence research. Comp. Intell. Mag. 5(4):13–18, 2010

    Article  Google Scholar 

  34. Greenspan H, Ginneken BV, Summers RM: Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Transactions on Medical Imaging 35(5):1153–1159, 2016

    Article  Google Scholar 

  35. Pan SJ, Yang Q: A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22(10):1345–1359, 2010

    Article  Google Scholar 

  36. C. N. Silla and A. A. Freitas, "A survey of hierarchical classification across different application domains," Data Mining and Knowledge Discovery, journal article vol. 22, no. 1, pp. 31–72, January 01 2011.

  37. A. M. Mateusz Buda, Maciej A. Mazurowski, "A Systematic Study of the Class Imbalance Problem in Convolutional Neural Networks," vol. https://arxiv.org/abs/1710.05381, no. arXiv:1710.05381, 2017.

  38. Tsehay YK et al.: Convolutional neural network based deep-learning architecture for prostate cancer detection on multiparametric magnetic resonance images. SPIE Medical Imaging 10134:11, 2017 SPIE

    Google Scholar 

  39. Sokolova M, Lapalme G: A systematic analysis of performance measures for classification tasks. Information Processing & Management 45(4):427–437, 2009/07/01/, 2009

    Article  Google Scholar 

  40. Manning DJ, Ethell SC, Donovan T: Detection or decision errors? Missed lung cancer from the posteroanterior chest radiograph. The British Journal of Radiology 77(915):231–235, 2004

    Article  CAS  PubMed  Google Scholar 

  41. Donovan T, Manning DJ: Successful reporting by non-medical practitioners such as radiographers, will always be task-specific and limited in scope. Radiography 12(1):7–12, 2006/02/01/, 2006

    Article  Google Scholar 

  42. Litchfield D, Ball LJ, Donovan T, Manning DJ, Crawford T: Viewing another person's eye movements improves identification of pulmonary nodules in chest X-ray inspection. Journal of Experimental Psychology: Applied 16(3):251–262, 2010

    PubMed  Google Scholar 

  43. Mello-Thoms C: Perception of breast cancer: eye-position analysis of mammogram interpretation. Academic Radiology 10(1):4–12, Jan 2003

    Article  PubMed  Google Scholar 

  44. Gandomkar Z, Tay K, Brennan PC, Mello-Thoms C: A Model Based on Temporal Dynamics of Fixations for Distinguishing Expert Radiologists' Scanpaths, Vol. 10136, 2017, pp. 1013606-1013606-9

    Google Scholar 

  45. A. R. Z. Ashesh Jain, Silvio Savarese, Ashutosh Saxena, "Structural-RNN: Deep Learning on Spatio-Temporal Graphs," https://arxiv.org/abs/1511.05298 , vol. arXiv:1511.05298, 2016.

    Google Scholar 

  46. H. Y. Bing Yu, Zhanxing Zhu, "Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting," https://arxiv.org/abs/1709.04875 , vol. arXiv:1709.04875, 2018.

    Google Scholar 

Download references

Acknowledgements

We would like to thank the radiologists that participated in our experiment.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.

Author information

Author notes

  1. Claudia Mello-Thoms

    Present address: Department of Radiology, University of Iowa, 200 Hawkins Drive, Iowa City, IA, 52242, USA

Authors and Affiliations

  1. Medical Image Optimisation and Perception Research Group (MIOPeG), Faculty of Medicine and Health, University of Sydney, 75 East Street, Lidcombe, NSW, 2141, Australia

    Suneeta Mall, Patrick C. Brennan & Claudia Mello-Thoms

Authors
  1. Suneeta Mall
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrick C. Brennan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Claudia Mello-Thoms
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Suneeta Mall.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mall, S., Brennan, P.C. & Mello-Thoms, C. Can a Machine Learn from Radiologists’ Visual Search Behaviour and Their Interpretation of Mammograms—a Deep-Learning Study. J Digit Imaging 32, 746–760 (2019). https://doi.org/10.1007/s10278-018-00174-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-018-00174-z

Keywords

Search

Navigation