Comprehensive Word-Level Classification of Screening Mammography Reports Using a Neural Network Sequence Labeling Approach
Radiology reports contain a large amount of potentially valuable unstructured data. Recently, neural networks have been employed to perform classification of radiology reports over a few classes at the document level. The success of neural networks in sequence-labeling problems such as named entity recognition and part of speech tagging suggests that they could be used to classify radiology report text with greater granularity. We employed a neural network architecture to comprehensively classify mammography report text at the word level using a sequence labeling approach. Two radiologists devised a comprehensive classification system for screening mammography reports. Each word in each report was manually categorized by a radiologist into one of 33 categories according to the classification system. Tagged words referencing the same finding were grouped into unique sets. We pre-labeled reports with a rule-based algorithm and then manually edited these annotations for 6705 screening mammography reports (25.1%, 66.8%, and 8.1% BI-RADS 0, 1, and 2, respectively). A combined convolutional and recurrent neural network model was used to label words in each sentence of the individual reports. A siamese recurrent neural network was then used to group findings into sets. Performance of the neural network-based method was compared to a rule-based algorithm and a conditional random field (CRF) model. Global accuracy (percentage of documents where all word tags were predicted correctly) and keyword accuracy (percentage of all words that were labeled correctly, excluding words tagged as unimportant) were calculated on an unseen 519 report test set. Two-tailed t tests were used to assess differences between algorithm performance, and p < 0.05 was used to determine statistical significance. The neural network-based approach showed significantly higher global accuracy compared to both the rule-based algorithm (88.3 vs 57.0%, p < 0.001) and the CRF model (88.3% vs. 75.8%, p < 0.001). The neural network also showed significantly higher keyword level accuracy compared to the rule-based algorithm (95.5% vs. 80.9% p < 0.001) and CRF model (95.5% vs. 76.9%, p < 0.001). We demonstrate the potential of neural networks to accurately perform word-level multilabel classification of free text radiology reports across 33 classes, thus showing the utility of a sequence labeling approach to NLP of radiology reports. We found that a neural network classifier outperforms a rule-based algorithm and a CRF classifier for comprehensive multilabel classification of free text screening mammography reports at the word level. By approaching radiology report classification as a sequence-labeling problem, we demonstrate the ability of neural networks to extract data from free text radiology reports at a level of granularity not previously reported.
KeywordsNatural language processing NLP Deep learning Radiology reporting
Compliance with Ethical Standards
Conflicts of Interest
Ryan G. Short is co-founder and CMO of Scanslated, Inc.
John Bralich is an employee of Scanslated, Inc.
Dave Bogaty is an employee of Scanslated, Inc.
Nicholas T. Befera is co-founder and CEO of Scanslated, Inc.
- 8.Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J: Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst:3111–3119, 2013Google Scholar
- 11.Chen MC, Ball RL, Yang L, Moradzadeh N, Chapman BE, Larson DB et al.: Deep Learning to Classify Radiology Free-Text Reports. Radiology 171115, 2017Google Scholar
- 13.Wang P, Qian Y, Soong FK, He L, Zhao H. Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Recurrent Neural Network. arXiv [csCL] 2015.Google Scholar
- 15.Cornegruta S, Bakewell R, Withey S, Montana G. Modelling Radiological Language with Bidirectional Long Short-Term Memory Networks. arXiv [csCL] 2016.Google Scholar
- 17.Gale W, Oakden-Rayner L, Carneiro G, Bradley AP, Palmer LJ. Detecting hip fractures with radiologist-level performance using deep neural networks. arXiv [csCV] 2017.Google Scholar
- 18.Bird S, Klein E, Loper E. Natural Language Processing with Python. “O’Reilly Media, Inc.”; 2009.Google Scholar
- 20.6.2. re — Regular expression operations — Python 3.6.4 documentation n.d. https://docs.python.org/3/library/re.html#module-re (accessed March 12, 2018).
- 21.American College of Radiology. ACR BI-RADS atlas: Breast Imaging Reporting and Data System ; Mammography, Ultrasound, Magnetic Resonance Imaging, Follow-up and Outcome Monitoring, Data Dictionary. 2013.Google Scholar
- 22.Lafferty JD, Mc Callum A, Pereira FCN. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Proceedings of the Eighteenth International Conference on Machine Learning, Morgan Kaufmann Publishers Inc.; 2001, p. 282–9.Google Scholar
- 23.Finkel JR, Grenager T, Manning C: Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2005, pp. 363–370Google Scholar
- 24.Silfverberg M, Ruokolainen T, Linden K, Kurimo M: Others. Part-of-speech tagging using conditional random fields: Exploiting sub-label dependencies for improved accuracy, 2014Google Scholar
- 26.Andrew G, Gao J. Scalable training ofL1-regularized log-linear models. Proceedings of the 24th international conference on Machine learning - ICML ‘07, 2007. https://doi.org/10.1145/1273496.1273501.
- 27.Huang Z, Xu W, Yu K. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv [csCL] 2015.Google Scholar
- 28.Řehůřek R, Sojka P: Software Framework for Topic Modelling with Large Corpora. Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. Valletta, ELRA, 2010, pp. 45–50Google Scholar
- 29.Chollet F. Keras 2015.Google Scholar
- 30.Cho K, van Merrienboer B, Bahdanau D, Bengio Y. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. arXiv [csCL] 2014.Google Scholar
- 31.Mueller J, Thyagarajan A. Siamese Recurrent Architectures for Learning Sentence Similarity. AAAI 2016.Google Scholar
- 32.Kingma DP, Ba J. Adam: A Method for Stochastic Optimization. arXiv [csLG] 2014.Google Scholar
- 33.Liu F, Baldwin T, Cohn T. Capturing Long-range Contextual Dependencies with Memory-enhanced Conditional Random Fields. arXiv [csCL] 2017.Google Scholar