Adverse Drug Events Detection in Clinical Notes by Jointly Modeling Entities and Relations Using Neural Networks
Background and Significance
Adverse drug events (ADEs) occur in approximately 2–5% of hospitalized patients, often resulting in poor outcomes or even death. Extraction of ADEs from clinical narratives can accelerate and automate pharmacovigilance. Using state-of-the-art deep-learning neural networks to jointly model concept and relation extraction, we achieved the highest integrated task score in the 2018 Medication and Adverse Drug Event (MADE) 1.0 challenge.
We used a combined bidirectional long short-term memory (BiLSTM) and conditional random fields (CRF) neural network to detect medical entities relevant to ADEs and a combined BiLSTM and attention network to determine relations, including the adverse drug reaction relation between medication and sign or symptom entities. Using these models, we conducted three experiments: (1) separate and sequential modeling of entities and relations; (2) joint modeling where relations between medications and sign or symptoms determined ADE and indication entities; (3) use of information from external resources such as the US FDA’s adverse event database as additional input to the second method.
Joint modeling improved the overall task accuracy from 0.62 to 0.65 F measure, and the additional use of external resources improved the accuracy to 0.66 F measure. Given the gold-standard medical entity labels, the joint model plus external resources method achieved F measures of 0.83 for ADE-relevant medical entity detection and 0.87 for relation detection.
It is important to use joint modeling techniques and external resources for effectively detecting ADEs from clinical narratives in electronic health record (EHR) systems. While the extraction of entities and relations individually achieved high accuracy, the integrated task still has room for further improvement.
Compliance with Ethical Standards
No sources of funding were used to conduct this study or prepare this manuscript.
Approval and consent
This study was conducted on de-identified clinical notes as part of a shared challenge, so no ethical approval or patient consent was required.
Conflict of interest
Bharath Dandala, Venkata Joopudi, and Murthy Devarakonda have no conflicts of interest that are directly relevant to the content of this article. Dr. Devarakonda is now on the faculty in Biomedical Informatics at Arizona State University, USA.
- 1.Huang Z, Xu W, Yu K. Bidirectional LSTM-CRF models for sequence tagging. 2015. arXiv:1508.01991.
- 2.Chalapathy R, Borzeshi EZ, Piccardi M. Bidirectional LSTM-CRF for clinical concept extraction. In: Procedings of the clinical natural language processing workshop. 2016. pp. 7–12.Google Scholar
- 5.Dandala B, Mahajan D, Devarakonda M. IBM research system at TAC 2017: adverse drug reactions extraction from drug labels. In: Text analysis conference (TAC) 2017 workshop at NIST. 2017.Google Scholar
- 6.Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. 2016. arXiv:1409.0473.
- 7.Hermann KM, et al. Teaching machines to read and comprehend. In: NIPS’15 proceedings of the 28th international conference on neural information processing systems, vol. 1. 2015. pp. 1693–1701.Google Scholar
- 8.Zhou P et al. Attention-based bidirectional long short-term memory networks for relation classification. In: Proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers); 2016. pp. 207–212.Google Scholar
- 9.UMass BioNLP. NLP challenges for detecting medication and adverse drug events from electronic health records (MADE 1.0). https://bio-nlp.org/index.php/projects/39-nlp-challenges. Accessed 5 Feb 2018.
- 10.US Food and Drug Administration, “FAERS”. https://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/AdverseDrugEffects/default.htm. Accessed 7 Feb 2018.
- 11.Ma X, Hovy E. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In: Proceedings of the 54th annual meeting of the association for computational linguistics. 2016. pp. 1064–1074.Google Scholar
- 12.Zhang D, Wang D. Relation classification via recurrent neural network. 2015. arXiv:1508.01006.
- 16.Graves A. Generating sequences with recurrent neural networks. 2014. arXiv:1308.0850.
- 17.Lample G, Ballesteros M, Subramanian S, Kawakami K, Dyer C. Neural architectures for named entity recognition. In: Proceedings of NAACL-HLT. 2016. pp. 260–270.Google Scholar
- 18.Collobert R, et al. Natural language processing (almost) from scratch. J Mach Learn Res. 2011;12:2493–537.Google Scholar
- 20.Zhang X, Zhao J, LeCun Y. Character-level convolutional networks for text classification. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R, editors. Advances in neural information processing systems, vol. 28. New York: Curran Associates Inc.; 2015. pp. 649–57.Google Scholar
- 21.Swampillai K, Stevenson M. Extracting relations within and across sentences. Proc Int Conf Recent Adv Nat Lang Process. 2011;2011:25–32.Google Scholar
- 22.Quirk C, Poon H. Distant supervision for relation extraction beyond the sentence boundary. In: Proceedings of the 15th conference of the European chapter of the association for computational linguistics. 2016. pp. 1171–1182.Google Scholar
- 24.Griffis D, Shivade C, Fosler-Lussier E, Lai AM. A quantitative and qualitative evaluation of sentence boundary detection for the clinical domain. AMIA Summits Transl Sci Proc. 2016;2016:88.Google Scholar
- 25.McCord MC, Bernth A. Using slot grammar. IBM TJ Watson Research Center, Yorktown Heights, NY, IBM Research Reports RC23978; 2010.Google Scholar
- 28.Dandala B, Devarakonda M, Bornea M, Nielson C. Scoring disease-medication associations using advanced NLP, machine learning, and multiple content sources. In: Proceedings of the fifth workshop on building and evaluating resources for biomedical text mining (BioTxtM 2016). 2016. pp. 125–133.Google Scholar
- 31.Rajani NF, Bornea M, Barker K. Stacking with auxiliary features for entity linking in the medical domain. BioNLP. 2017;2017:39–47.Google Scholar
- 32.Kingma D, Ba J. Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representations (ICLR). 2015. pp. 1–15.Google Scholar
- 33.Jagannatha AN, Yu H. Structured prediction models for RNN based sequence labeling in clinical text. In: Proceedings of the conference on empirical methods in natural language processing. conference on empirical methods in natural language processing, vol. 2016. 2016. p. 856.Google Scholar
- 35.Sahu SK, Anand A, Oruganty K, Gattu M. Relation extraction from clinical texts using domain invariant convolutional neural network. In: Proceedings of the 15th workshop on biomedical natural language processing. 2016. pp. 206–215.Google Scholar
- 37.Chapman AB, Peterson KS, Alba PR, DuVall SL, Patterson OV. Hybrid system for adverse drug event detection. In: Proceedings of machine learning research, vol. 90. 2018. pp. 16–24.Google Scholar
- 38.Xu D, Yadav V, Bethard S. UArizona at the MADE 1.0 NLP challenge. In: Proceedings of first international workshop on medication and adverse drug event detection. 2018. vol. 90, pp. 57–65.Google Scholar