Abstract
Delirium is a serious medical complication associated with poor outcomes. Given the complexity of the syndrome, prevention and early detection are critical in mitigating its effects. We used Confusion Assessment Method (CAM) screening and Electronic Health Record (EHR) data for 64,038 inpatient visits to train and test a model predicting delirium arising in hospital. Incident delirium was defined as the first instance of a positive CAM occurring at least 48 h into a hospital stay. A Random Forest machine learning algorithm was used with demographic data, comorbidities, medications, procedures, and physiological measures. The data set was randomly partitioned 80% / 20% for training and validating the predictive model, respectively. Of the 51,240 patients in the training set, 2774 (5.4%) experienced delirium during their hospital stay; and of the 12,798 patients in the validation set, 701 (5.5%) experienced delirium. Under-sampling of the delirium negative population was used to address the class imbalance. The Random Forest predictive model yielded an area under the receiver operating characteristic curve (ROC AUC) of 0.909 (95% CI 0.898 to 0.921). Important variables in the model included previously identified predisposing and precipitating risk factors. This machine learning approach displayed a high degree of accuracy and has the potential to provide a clinically useful predictive model for earlier intervention in those patients at greatest risk of developing delirium.
Similar content being viewed by others
Notes
Confusion Assessment Method. © 1988, 2003, Hospital Elder Life Program. All rights reserved. Not to be reproduced without permission. Instructions for correct usage available at: http://www.hospitalelderlifeprogram.org
References
Inouye, S. K., Westendorp, R. G. J., and Saczynski, J. S., Delirium in elderly people. Lancet 383:911–922, 2014. https://doi.org/10.1016/S0140-6736(13)60688-1.
Inouye, S. K., Rushing, J. T., Foreman, M. D. et al., Does delirium contribute to poor hospital outcomes?: A three-site epidemiologic study. J. Gen. Intern. Med. 13:234–242, 1998. https://doi.org/10.1046/j.1525-1497.1998.00073.x.
Ely, E. W., Gautam, S., Margolin, R. et al., The impact of delirium in the intensive care unit on hospital length of stay. Intensive Care Med. 27:1892–1900, 2001. https://doi.org/10.1007/s00134-001-1132-2.
Witlox, J., Eurelings, L. S. M., De Jonghe, J. F. M. et al., Delirium in Elderly Patients and the Risk of Postdischarge Mortality. JAMA 304:443–451, 2010. https://doi.org/10.1001/jama.2010.1013.
Rudolph, J. L., and Marcantonio, E. R., Postoperative Delirium: Acute change with long-term implications. Anesth. Analg. 112:1202–1211, 2011. https://doi.org/10.1213/ANE.0b013e3182147f6d.
Girard, T. D., Jackson, J. C., Pandharipande, P. P. et al., Delirium as a predictor of long-term cognitive impairment in survivors of critical illness. Crit. Care Med. 38:1513–1520, 2010. https://doi.org/10.1097/CCM.0b013e3181e47be1.
Langan, C., Sarode, D. P., Russ, T. C. et al., Psychiatric symptomatology after delirium: a systematic review. Psychogeriatrics, 2017. https://doi.org/10.1111/psyg.12240.
Davis, D. H. J., Muniz-Terrera, G., Keage, H. A. D. et al., Association of Delirium With Cognitive Decline in Late Life. JAMA Psychiatry 74:244, 2017. https://doi.org/10.1001/jamapsychiatry.2016.3423.
Brown, E., and Douglas, V., Moving Beyond Metabolic Encephalopathy: An Update on Delirium Prevention, Workup, and Management. Semin. Neurol. 35:646–655, 2015. https://doi.org/10.1055/s-0035-1564685.
Hshieh, T. T., Yue, J., Oh, E. et al., Effectiveness of Multicomponent Nonpharmacological Delirium Interventions. JAMA Intern. Med. 175:512, 2015. https://doi.org/10.1001/jamainternmed.2014.7779.
Lawlor, P. G., and Bush, S. H., Delirium diagnosis, screening and management. Curr Opin Support Palliat Care 8:286–295, 2014. https://doi.org/10.1097/SPC.0000000000000062.
Inouye, S. K., Van Dyck, C. H., Alessi, C. A. et al., Clarifying confusion: The confusion assessment method: A new method for detection of delirium. Ann. Intern. Med. 113:941–948, 1990. https://doi.org/10.7326/0003-4819-113-12-941.
Waszynski, C., and Petrovic, K., Nurses’ evaluation of the Confusion Assessment Method: a pilot study. J. Gerontol. Nurs. 34:49–56, 2008. https://doi.org/10.3928/00989134-20080401-06.
Ely, E. W., Margolin, R., Francis, J. et al., Evaluation of delirium in critically ill patients: validation of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU). Crit. Care Med. 29:1370–1379, 2001.
Inouye, S. K., Kosar, C. M., Tommet, D. et al., The CAM-S: Development and validation of a new scoring system for delirium severity in 2 cohorts. Ann. Intern. Med. 160:526–533, 2014. https://doi.org/10.7326/M13-1927.
Van Meenen, L. C. C., van Meenen, D. M. P., de Rooij, S. E., and ter Riet, G., Risk prediction models for postoperative delirium: A systematic review and meta-analysis. J. Am. Geriatr. Soc. 62:2383–2390, 2014. https://doi.org/10.1111/jgs.13138.
van den Boogaard, M., Pickkers, P., Slooter, A. J. C. et al., Development and validation of PRE-DELIRIC (PREdiction of DELIRium in ICu patients) delirium prediction model for intensive care patients: observational multicentre study. BMJ 344:e420–e420, 2012. https://doi.org/10.1136/bmj.e420.
Kim MY, Park UJ, Kim HT, Cho WH (2016) DELirium Prediction Based on Hospital Information (Delphi) in General Surgery Patients. 95:1–7. doi: https://doi.org/10.1097/MD.0000000000003072
Kennedy, M., Enander, R. A., Tadiri, S. P. et al., Delirium risk prediction, healthcare use and mortality of elderly adults in the emergency department. J. Am. Geriatr. Soc. 62, 2014. https://doi.org/10.1111/jgs.12692.
Breiman, L., Random Forests. Mach. Learn. 45:5–32, 2001. https://doi.org/10.1023/A:1010933404324.
Quan, H., Li, B., Couris, C. M. et al., Updating and validating the charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am. J. Epidemiol. 173:676–682, 2011. https://doi.org/10.1093/aje/kwq433.
Wasey JO (2016) ICD: Tools for Working with ICD-9 and ICD-10 Codes, and Finding Comorbidities
Team TH a. (2017) h2o: R Interface for H2O
Chen C, Liaw A, Breiman L (2004) Using random forest to learn imbalanced data. Univ California, Berkeley 1–12. doi: ley.edu/sites/default/files/tech-reports/666.pdf
Khalilia, M., Chakraborty, S., and Popescu, M., Predicting disease risks from highly imbalanced data using random forest. BMC Med Inform Decis Mak 11:51, 2011. https://doi.org/10.1186/1472-6947-11-51.
Platt, J., Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv Large Margin Classif 10:61–74, 1999. 10.1.1.41.1639.
Kuhn M (2008) Building Predictive Models in R Using the caret Package. J Stat Software, Artic 28:1–26. https://doi.org/10.18637/jss.v028.i05.
Robin, X., Turck, N., Hainard, A. et al., pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77, 2011. https://doi.org/10.1186/1471-2105-12-77.
Grau, J., Grosse, I., and Keilwagen, J., PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics 31:2595–2597, 2015. https://doi.org/10.1093/bioinformatics/btv153.
Boulesteix, A.-L., Janitza, S., Kruppa, J., and König, I. R., Overview of random forest methodology and practical guidance with emphasis on computational biology and bioinformatics. Wiley Interdiscip Rev Data Min Knowl Discov 2:493–507, 2012. https://doi.org/10.1002/widm.1072.
Semancik, L., Waszynski, C., and Udeh, E., Delirium in hospitalized patients: recognition, prevention, and management. Conn. Med. 78:105–109, 2014.
Tieges, Z., McGrath, A., Hall, R. J., and MacLullich, A. M. J., Abnormal Level of Arousal as a Predictor of Delirium and Inattention: An Exploratory Study. Am. J. Geriatr. Psychiatry 21:1244–1253, 2013. https://doi.org/10.1016/j.jagp.2013.05.003.
Han, J. H., Vasilevskis, E. E., Schnelle, J. F. et al., The Diagnostic Performance of the Richmond Agitation Sedation Scale for Detecting Delirium in Older Emergency Department Patients. Acad. Emerg. Med. 22:878–882, 2015. https://doi.org/10.1111/acem.12706.
Morandi, A., Han, J. H., Meagher, D. et al., Detecting Delirium Superimposed on Dementia: Evaluation of the Diagnostic Performance of the Richmond Agitation and Sedation Scale. J. Am. Med. Dir. Assoc. 17:828–833, 2016. https://doi.org/10.1016/j.jamda.2016.05.010.
Corradi, J. P., Chhabra, J., Mather, J. F. et al., Analysis of multi-dimensional contemporaneous EHR data to refine delirium assessments. Comput. Biol. Med. 5799:1–24, 2016. https://doi.org/10.1016/j.compbiomed.2016.06.013.
Gulshan, V., Peng, L., Coram, M. et al., Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. Jama 304:649–656, 2016. https://doi.org/10.1001/jama.2016.17216.
Dawes, T. J. W., de Marvao, A., Shi, W. et al., Machine Learning of Three-dimensional Right Ventricular Motion Enables Outcome Prediction in Pulmonary Hypertension: A Cardiac MR Imaging Study. Radiology 161315, 2017. https://doi.org/10.1148/radiol.2016161315.
Ghose, S., Mitra, J., Khanna, S., and Dowling, J., An Improved Patient-Specific Mortality Risk Prediction in ICU in a Random Forest Classification Framework. Stud Health Technol Inform 214:56–61, 2015. https://doi.org/10.3233/978-1-61499-558-6-56.
Futoma, J., Morris, J., and Lucas, J., A comparison of models for predicting early hospital readmissions. J. Biomed. Inform. 56:229–238, 2015. https://doi.org/10.1016/j.jbi.2015.05.016.
Kessler, R. C., Warner, C. H., Ivany, C. et al., Predicting Suicides After Psychiatric Hospitalization in US Army Soldiers: The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry 72:1–9, 2014. https://doi.org/10.1001/jamapsychiatry.2014.1754.
Mortazavi, B. J., Downing, N. S., Bucholz, E. M. et al., Analysis of Machine Learning Techniques for Heart Failure Readmissions. Circ Cardiovasc Qual Outcomes 9:629–640, 2016. https://doi.org/10.1161/CIRCOUTCOMES.116.003039.
Taylor, R. A., Pare, J. R., Venkatesh, A. K. et al., Prediction of In-hospital Mortality in Emergency Department Patients with Sepsis: A Local Big Data-Driven, Machine Learning Approach. Acad. Emerg. Med. 23:269–278, 2016. https://doi.org/10.1111/acem.12876.
Kramer, D., Veeranki, S., Hayn, D., and Quehenberger, F., Development and Validation of a Multivariable Prediction Model for the Occurrence of Delirium in Hospitalized Gerontopsychiatry and Internal Medicine Patients. Stud Health Technol Inform:32–39, 2017. https://doi.org/10.3233/978-1-61499-759-7-32.
Wong, A., Young, A. T., Liang, A. S. et al., Development and Validation of an Electronic Health Record–Based Machine Learning Model to Estimate Delirium Risk in Newly Hospitalized Patients Without Known Cognitive Impairment. JAMA Netw. Open 1:e181018, 2018. https://doi.org/10.1001/jamanetworkopen.2018.1018.
Halladay, C. W., Sillner, A. Y., and Rudolph, J. L., Performance of Electronic Prediction Rules for Prevalent Delirium at Hospital Admission. JAMA Netw. Open 1:e181405, 2018. https://doi.org/10.1001/JAMANETWORKOPEN.2018.1405.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of Interest
The authors do not have any conflicts of interest to declare. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Ethical Approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed Consent
For this type of study formal consent is not required.
Additional information
This article is part of the Topical Collection on Systems-Level Quality Improvement
Rights and permissions
About this article
Cite this article
Corradi, J.P., Thompson, S., Mather, J.F. et al. Prediction of Incident Delirium Using a Random Forest classifier. J Med Syst 42, 261 (2018). https://doi.org/10.1007/s10916-018-1109-0
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10916-018-1109-0