Abstract
Advance care planning, which includes clarifying and documenting goals of care and preferences for future care, is essential for achieving end-of-life care that is consistent with the preferences of dying patients and their families. Physicians document their communication about these preferences as unstructured free text in clinical notes; as a result, routine assessment of this quality indicator is time consuming and costly. In this study, we trained and validated a deep neural network to detect documentation of advanced care planning conversations in clinical notes from electronic health records. We assessed its performance against rigorous manual chart review and rule-based regular expressions. For detecting documentation of patient care preferences at the note level, the algorithm had high performance; F1-score of 92.0 (95% CI, 89.1–95.1), sensitivity of 93.5% (95% CI, 90.0%–98.0%), positive predictive value of 90.5% (95% CI, 86.4%–95.1%) and specificity of 91.0% (95% CI, 86.4%–95.3%) and consistently outperformed regular expression. Deep learning methods offer an efficient and scalable way to improve the visibility of documented serious illness conversations within electronic health record data, helping to better quality of care.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cook, D., Rocker, G.: Dying with dignity in the intensive care unit. New Engl. J. Med. 370, 2506–2514 (2014)
Wright, A.A., Zhang, B., Ray, A., et al.: Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment. JAMA 300(14), 1665–1673 (2008)
Nicholas, L.H., Langa, K.M., Iwashyna, T.J., Weir, D.R.: Regional variation in the association between advance directives and end-of-life Medicare expenditures. JAMA 306(13), 1447–1453 (2011)
Teno, J.M., Gruneir, A., Schwartz, Z., Nanda, A., Wetle, T.: Association between advance directives and quality of end-of-life care: a national study. J. Am. Geriatr. Soc. 55(2), 189–194 (2007)
Detering, K.M., Hancock, A.D., Reade, M.C., Silvester, W.: The impact of advance care planning on end of life care in elderly patients: randomised controlled trial. BMJ 340, c1345 (2010)
Huynh, T.N., Kleerup, E.C., Raj, P.P., Wenger, N.S.: The opportunity cost of futile treatment in the intensive care unit. Crit. Care Med. 42(9), 1977–1982 (2014). https://doi.org/10.1097/CCM.0000000000000402
Huynh, T.N., et al.: The frequency and cost of treatment perceived to be futile in critical care. JAMA Intern. Med. 173, 1887–1994 (2013)
NQF #1626: Patients Admitted to ICU Who Have Care Preferences Documented. National Quality Forum
Khandelwal, N., Kross, E., Engelberg, R., Coe, N., Long, A., Curtis, J.: Estimating the effect of palliative care interventions and advance care planning on ICU utilization: a systematic review. Crit Care Med. 43, 1102–1111 (2015). https://doi.org/10.1097/CCM.0000000000000852
Rising, J., Corrigan, J., Valuck, T.: Building Additional Serious Illness Measures Into Medicare Programs. The Pew Charitable Trusts, Philadelphia (2017)
Walling, A.M., Tisnado, D., Asch, S.M., et al.: The quality of supportive cancer care in the veterans affairs health system and targets for improvement. JAMA Intern. Med. 173(22), 2071–2079 (2013)
Dy, S.M., Lorenz, K.A., O’Neill, S.M., et al.: Cancer quality-ASSIST supportive oncology quality indicator set: feasibility, reliability, and validity testing. Cancer 116(13), 3267–3275 (2010)
Aldridge, M.D., Meier, D.E.: It is possible: quality measurement during serious illness. JAMA Intern. Med. 173(22), 2080–2081 (2013)
Melton, G.B., Hripcsak, G.: Automated detection of adverse events using natural language processing of discharge summaries. JAMA 12(4), 448–457 (2005)
Honnibal, M., Johnson, M.: An improved non-monotonic transition system for dependency parsing. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, September 2015, Lisbon, Portugal, pp. 1373–1378. Association for Computational Linguistics (2015)
Carrell, D.S., et al.: Challenges in adapting existing clinical natural language processing systems to multiple, diverse healthcare settings. (JAMIA) 2, 986–991 (2017)
Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015). arXiv:1404.7828. https://doi.org/10.1016/j.neunet.2014.09.003. PMID 25462637
Johnson, A.E., et al.: MIMIC-III, a freely accessible critical care database. Sci. Data 3, 160035 (2016)
Walling, A.M., et al.: The quality of care provided to hospitalized patients at the end of life. Arch. Intern. Med. 170(12), 1057–1063 (2010)
Honnibal, M., Johnson, M.: An improved non-monotonic transition system for dependency parsing. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal (2015)
Dernoncourt, F., Lee, J.Y., Uzuner, O., Szolovits, P.: De-identification of patient notes with recurrent neural networks. J. Am. Med. Inform. Assoc. 24(3), 596–606 (2017)
Dernoncourt, F., Lee, J.Y., Szolovits, P.: NeuroNER: an easy-to-use program for named-entity recognition based on neural networks. In: Conference on Empirical Methods on Natural Language Processing (EMNLP) (2017)
Efron, B.: Better bootstrap confidence intervals. J. Am. Stat. Assoc. 82(397), 171–185 (1987)
Davison, A.C., Hinkley, D.V.: Bootstrap Methods and their Application. Cambridge University Press, Cambridge (1997)
D’Avolio, L.W., Nguyen, T.M., Goryachev, S., Fiore, L.D.: Automated concept-level information extraction to reduce the need for custom software and rules development. J. Am. Med. Inform. Assoc. 18(5), 607–613 (2011)
Xu, H., et al.: Facilitating pharmacogenetic studies using electronic health records and natural-language processing: a case study of warfarin. J. Am. Med. Inform. Assoc. 18(4), 387–391 (2011)
Bejnordi, B.E., Veta, M., van Diest, P.J., et al.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. J. Am. Med. Inform. Assoc. 318(22), 2199–2210 (2017)
Gulshan, V., Peng, L., Coram, M., et al.: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J. Am. Med. Inform. Assoc. 316(22), 2402–2410 (2016)
Ting, D.S.W., et al.: Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. J. Am. Med. Inform. Assoc. 318, 2211–2223 (2017)
Lindvall, C., et al.: Natural Language Processing to Assess End-of-Life Quality Indicators in Cancer Patients Receiving Palliative Surgery. J Palliatd Med., 17 October 2018. https://doi.org/10.1089/jpm.2018.0326
Acknowledgements
We are particularly grateful to Tristan Naumann, Franck Dernoncourt, Elena Sergeeva, Edward Moseley, and Alistair Johnson for helpful guidance and advice during the development of this research. Additionally, we would like to thank Peter Szolovits for providing computing resources, as well as Saad Salman, Sarah Kaminar Bourland, Haruki Matsumoto and Dickson Lui for annotating clinical notes. This research was facilitated by preliminary work done as part of course HST.953 in the Harvard-MIT Division of Health Sciences and Technology (HST) at Massachusetts Institute of Technology (MIT), Boston, MA.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
A Regular Expression Library
Domain | Keywords |
|---|---|
Patient care preferences | goc, goals of care, goals for care, goals of treatment, goals for treatment, treatment goals, family meeting, family discussion, family discussions, patient goals, dnr, dni, dnrdni, dnr/dni, DNI/R, do not resuscitate, do-not-resuscitate, do not intubate, do-not-intubate, chest compressions, no defibrillation, no endotracheal intubation, no mechanical intubation, shocks, cmo, comfort measures |
Goals of care conversations | goc, goals of care, goals for care, goals of treatment, goals for treatment, treatment goals, family meeting, family discussion, family discussions, patient goals |
Code status limitations | dnr, dni, dnrdni, dnrdni, DNIR, do not resuscitate, do-not-resuscitate, do not intubate, do-not-intubate, chest compressions, no defibrillation, no endotracheal intubation, no mechanical intubation, shocks, cmo, comfort measures |
Communication with family | Explicit conversations held during ICU stay period with patients or family members about the patient’s goals, values, or priorities for treatment and outcomes |
Full code status | full code |
B Token-Level Performance
See Table 4.
C Examples of Identified Text
Below are examples of correctly identified serious illness documentation by the neural network and regular expression methods in the validation dataset. Correctly identified tokens are bolded. Typographical errors are from the original text. Each cell includes an example of identified tokens in the same text and an example of documentation identified by the neural network that was missed by the regular expression method, if relevant.
Domain | Neural network | Regular expression |
|---|---|---|
Goals of care conversations | Hypercarbic resp failure: family meeting was held with son/HCP and in keeping with patients goals of care, there was no plan for intubation. Family was brought in and we explained the graveness of her ABG and her worsened mental status which had failed to improve with BiPAP. Family was comfortable with removing Bipap and providing comfort care including morphine prn family open to cmo but pt wants full code but also doesn’t want treatment or to be disturbed | Hypercarbic resp failure: family meeting was held with son/HCP and in keeping with patients goals of care, there was no plan for intubation.Family was brought in and we explained the graveness of her ABG and her worsened mental status which had failed to improve with BiPAP. Family was comfortable with removing Bipap and providing comfort care including morphine prn family open to cmo but pt wants full code but also doesn’t want treatment or to be disturbed |
Code status limitations | CODE: DNR/DNI, confirmed with healthcare manager who will be discussing with official HCP | CODE: DNR/DNI, confirmed with healthcare manager who will be discussing with official HCP |
Communication with family | Dr. [**First Name (STitle) **] from neurosurgery held family meeting and explained grave prognosis to the family lengthy discussion with the son who is health care proxy he wishes to pursue comfort measures due to severe and unrevascularizable cad daughter is not in agreement at this time but is not the proxy due to underlying psychiatric illness | Dr. [**First Name (STitle) **] from neurosurgery held family meeting and explained grave prognosis to the family lengthy discussion with the son who is health care proxy he wishes to pursue comfort measures due to severe and unrevascularizable cad daughter is not in agreement at this time but is not the proxy due to underlying psychiatric illness |
Full code status | Code: FULL; Discussed with daughter and HCP who says that patient is in a Hospice program with a “bridge" to DNR/DNI/CMO, but despite multiple conversations, the patient insists on being full code CODE: Presumed full | Code: FULL; Discussed with daughter and HCP who says that patient is in a Hospice program with a “bridge" to DNR/DNI/CMO, but despite multiple conversations, the patient insists on being full code CODE: Presumed full |
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Chien, I., Shi, A., Chan, A., Lindvall, C. (2019). Identification of Serious Illness Conversations in Unstructured Clinical Notes Using Deep Neural Networks. In: Koch, F., et al. Artificial Intelligence in Health. AIH 2018. Lecture Notes in Computer Science(), vol 11326. Springer, Cham. https://doi.org/10.1007/978-3-030-12738-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-12738-1_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-12737-4
Online ISBN: 978-3-030-12738-1
eBook Packages: Computer ScienceComputer Science (R0)