Drug Safety

, Volume 42, Issue 1, pp 135–146 | Cite as

Adverse Drug Events Detection in Clinical Notes by Jointly Modeling Entities and Relations Using Neural Networks

  • Bharath Dandala
  • Venkata Joopudi
  • Murthy DevarakondaEmail author
Original Research Article
Part of the following topical collections:
  1. NLP Challenge for Detecting Medication and Adverse Drug Events from Electronic Health Records (MADE 1.0)


Background and Significance

Adverse drug events (ADEs) occur in approximately 2–5% of hospitalized patients, often resulting in poor outcomes or even death. Extraction of ADEs from clinical narratives can accelerate and automate pharmacovigilance. Using state-of-the-art deep-learning neural networks to jointly model concept and relation extraction, we achieved the highest integrated task score in the 2018 Medication and Adverse Drug Event (MADE) 1.0 challenge.


We used a combined bidirectional long short-term memory (BiLSTM) and conditional random fields (CRF) neural network to detect medical entities relevant to ADEs and a combined BiLSTM and attention network to determine relations, including the adverse drug reaction relation between medication and sign or symptom entities. Using these models, we conducted three experiments: (1) separate and sequential modeling of entities and relations; (2) joint modeling where relations between medications and sign or symptoms determined ADE and indication entities; (3) use of information from external resources such as the US FDA’s adverse event database as additional input to the second method.


Joint modeling improved the overall task accuracy from 0.62 to 0.65 F measure, and the additional use of external resources improved the accuracy to 0.66 F measure. Given the gold-standard medical entity labels, the joint model plus external resources method achieved F measures of 0.83 for ADE-relevant medical entity detection and 0.87 for relation detection.


It is important to use joint modeling techniques and external resources for effectively detecting ADEs from clinical narratives in electronic health record (EHR) systems. While the extraction of entities and relations individually achieved high accuracy, the integrated task still has room for further improvement.


Compliance with Ethical Standards


No sources of funding were used to conduct this study or prepare this manuscript.

Approval and consent

This study was conducted on de-identified clinical notes as part of a shared challenge, so no ethical approval or patient consent was required.

Conflict of interest

Bharath Dandala, Venkata Joopudi, and Murthy Devarakonda have no conflicts of interest that are directly relevant to the content of this article. Dr. Devarakonda is now on the faculty in Biomedical Informatics at Arizona State University, USA.


  1. 1.
    Huang Z, Xu W, Yu K. Bidirectional LSTM-CRF models for sequence tagging. 2015. arXiv:1508.01991.
  2. 2.
    Chalapathy R, Borzeshi EZ, Piccardi M. Bidirectional LSTM-CRF for clinical concept extraction. In: Procedings of the clinical natural language processing workshop. 2016. pp. 7–12.Google Scholar
  3. 3.
    Habibi M, Weber L, Neves M, Wiegandt DL, Leser U. Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics. 2017;33(14):i37–48.CrossRefGoogle Scholar
  4. 4.
    Li F, Zhang M, Tian B, Chen B, Fu G, Ji D. Recognizing irregular entities in biomedical text via deep neural networks. Pattern Recognit Lett. 2017;105:105–13.CrossRefGoogle Scholar
  5. 5.
    Dandala B, Mahajan D, Devarakonda M. IBM research system at TAC 2017: adverse drug reactions extraction from drug labels. In: Text analysis conference (TAC) 2017 workshop at NIST. 2017.Google Scholar
  6. 6.
    Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. 2016. arXiv:1409.0473.
  7. 7.
    Hermann KM, et al. Teaching machines to read and comprehend. In: NIPS’15 proceedings of the 28th international conference on neural information processing systems, vol. 1. 2015. pp. 1693–1701.Google Scholar
  8. 8.
    Zhou P et al. Attention-based bidirectional long short-term memory networks for relation classification. In: Proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers); 2016. pp. 207–212.Google Scholar
  9. 9.
    UMass BioNLP. NLP challenges for detecting medication and adverse drug events from electronic health records (MADE 1.0). Accessed 5 Feb 2018.
  10. 10.
  11. 11.
    Ma X, Hovy E. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In: Proceedings of the 54th annual meeting of the association for computational linguistics. 2016. pp. 1064–1074.Google Scholar
  12. 12.
    Zhang D, Wang D. Relation classification via recurrent neural network. 2015. arXiv:1508.01006.
  13. 13.
    Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735–80.CrossRefGoogle Scholar
  14. 14.
    Bengio Y, Simard P, Frasconi P. Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw. 1994;5(2):157–66.CrossRefGoogle Scholar
  15. 15.
    Graves A, Schmidhuber J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 2005;18(5):602–10.CrossRefGoogle Scholar
  16. 16.
    Graves A. Generating sequences with recurrent neural networks. 2014. arXiv:1308.0850.
  17. 17.
    Lample G, Ballesteros M, Subramanian S, Kawakami K, Dyer C. Neural architectures for named entity recognition. In: Proceedings of NAACL-HLT. 2016. pp. 260–270.Google Scholar
  18. 18.
    Collobert R, et al. Natural language processing (almost) from scratch. J Mach Learn Res. 2011;12:2493–537.Google Scholar
  19. 19.
    Sutton C, McCallum A, et al. An introduction to conditional random fields. Found Trends Mach Learn. 2012;4(4):267–373.CrossRefGoogle Scholar
  20. 20.
    Zhang X, Zhao J, LeCun Y. Character-level convolutional networks for text classification. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R, editors. Advances in neural information processing systems, vol. 28. New York: Curran Associates Inc.; 2015. pp. 649–57.Google Scholar
  21. 21.
    Swampillai K, Stevenson M. Extracting relations within and across sentences. Proc Int Conf Recent Adv Nat Lang Process. 2011;2011:25–32.Google Scholar
  22. 22.
    Quirk C, Poon H. Distant supervision for relation extraction beyond the sentence boundary. In: Proceedings of the 15th conference of the European chapter of the association for computational linguistics. 2016. pp. 1171–1182.Google Scholar
  23. 23.
    Peng N, Poon H, Quirk C, Toutanova K, Yih W. Cross-sentence n-ary relation extraction with graph lstms. Trans Assoc Comput Linguis. 2017;5:101–15.CrossRefGoogle Scholar
  24. 24.
    Griffis D, Shivade C, Fosler-Lussier E, Lai AM. A quantitative and qualitative evaluation of sentence boundary detection for the clinical domain. AMIA Summits Transl Sci Proc. 2016;2016:88.Google Scholar
  25. 25.
    McCord MC, Bernth A. Using slot grammar. IBM TJ Watson Research Center, Yorktown Heights, NY, IBM Research Reports RC23978; 2010.Google Scholar
  26. 26.
    Minsky M. Memoir on inventing the confocal scanning microscope. Scanning. 1988;10(4):128–38.CrossRefGoogle Scholar
  27. 27.
    Fillmore CJ. Frame semantics and the nature of language. Ann N Y Acad Sci. 1976;280(1):20–32.CrossRefGoogle Scholar
  28. 28.
    Dandala B, Devarakonda M, Bornea M, Nielson C. Scoring disease-medication associations using advanced NLP, machine learning, and multiple content sources. In: Proceedings of the fifth workshop on building and evaluating resources for biomedical text mining (BioTxtM 2016). 2016. pp. 125–133.Google Scholar
  29. 29.
    Banda JM, Evans L, Vanguri RS, Tatonetti NP, Ryan PB, Shah NH. A curated and standardized adverse drug event resource to accelerate drug safety research. Sci Data. 2016;3:1–11.CrossRefGoogle Scholar
  30. 30.
    Bodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32(suppl 1):D267–70.CrossRefGoogle Scholar
  31. 31.
    Rajani NF, Bornea M, Barker K. Stacking with auxiliary features for entity linking in the medical domain. BioNLP. 2017;2017:39–47.Google Scholar
  32. 32.
    Kingma D, Ba J. Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representations (ICLR). 2015. pp. 1–15.Google Scholar
  33. 33.
    Jagannatha AN, Yu H. Structured prediction models for RNN based sequence labeling in clinical text. In: Proceedings of the conference on empirical methods in natural language processing. conference on empirical methods in natural language processing, vol. 2016. 2016. p. 856.Google Scholar
  34. 34.
    Munkhdalai T, Liu F, Yu H. Clinical relation extraction toward drug safety surveillance using electronic health record narratives: classical learning versus deep learning. JMIR Public Health Surveill. 2018;4(2):e29.CrossRefGoogle Scholar
  35. 35.
    Sahu SK, Anand A, Oruganty K, Gattu M. Relation extraction from clinical texts using domain invariant convolutional neural network. In: Proceedings of the 15th workshop on biomedical natural language processing. 2016. pp. 206–215.Google Scholar
  36. 36.
    Li F, Zhang M, Fu G, Ji D. A neural joint model for entity and relation extraction from biomedical text. BMC Bioinform. 2017;18(1):1–11.CrossRefGoogle Scholar
  37. 37.
    Chapman AB, Peterson KS, Alba PR, DuVall SL, Patterson OV. Hybrid system for adverse drug event detection. In: Proceedings of machine learning research, vol. 90. 2018. pp. 16–24.Google Scholar
  38. 38.
    Xu D, Yadav V, Bethard S. UArizona at the MADE 1.0 NLP challenge. In: Proceedings of first international workshop on medication and adverse drug event detection. 2018. vol. 90, pp. 57–65.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.IBM ResearchYorktown HeightsUSA
  2. 2.Biomedical InformaticsArizona State UniversityTempeUSA

Personalised recommendations