Journal of General Internal Medicine

, Volume 35, Issue 1, pp 220–227 | Cite as

A Gradient Boosting Machine Learning Model for Predicting Early Mortality in the Emergency Department Triage: Devising a Nine-Point Triage Score

  • Maximiliano Klug
  • Yiftach Barash
  • Sigalit Bechler
  • Yehezkel S. Resheff
  • Talia Tron
  • Avi Ironi
  • Shelly Soffer
  • Eyal Zimlichman
  • Eyal KlangEmail author



Emergency departments (ED) are becoming increasingly overwhelmed, increasing poor outcomes. Triage scores aim to optimize the waiting time and prioritize the resource usage. Artificial intelligence (AI) algorithms offer advantages for creating predictive clinical applications.


Evaluate a state-of-the-art machine learning model for predicting mortality at the triage level and, by validating this automatic tool, improve the categorization of patients in the ED.


An institutional review board (IRB) approval was granted for this retrospective study. Information of consecutive adult patients (ages 18–100) admitted at the emergency department (ED) of one hospital were retrieved (January 1, 2012–December 31, 2018). Features included the following: demographics, admission date, arrival mode, referral code, chief complaint, previous ED visits, previous hospitalizations, comorbidities, home medications, vital signs, and Emergency Severity Index (ESI). The following outcomes were evaluated: early mortality (up to 2 days post ED registration) and short-term mortality (2–30 days post ED registration). A gradient boosting model was trained on data from years 2012–2017 and examined on data from the final year (2018). The area under the curve (AUC) for mortality prediction was used as an outcome metric. Single-variable analysis was conducted to develop a nine-point triage score for early mortality.

Key Results

Overall, 799,522 ED visits were available for analysis. The early and short-term mortality rates were 0.6% and 2.5%, respectively. Models trained on the full set of features yielded an AUC of 0.962 for early mortality and 0.923 for short-term mortality. A model that utilized the nine features with the highest single-variable AUC scores (age, arrival mode, chief complaint, five primary vital signs, and ESI) yielded an AUC of 0.962 for early mortality.


The gradient boosting model shows high predictive ability for screening patients at risk of early mortality utilizing data available at the time of triage in the ED.


machine learning gradient boosting triage emergency department early mortality 



This research was performed in collaboration with the Intuit data science team as part of the philanthropic framework, We Care and Give Back. It was also conducted with the help of ARC - The Innovation Center at Sheba Hospital.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.

Supplementary material

11606_2019_5512_MOESM1_ESM.docx (407 kb)
ESM 1 (DOCX 407 kb)


  1. 1.
    Carter EJ, Pouch SM, Larson EL. The relationship between emergency department crowding and patient outcomes: a systematic review. J Nurs Scholarsh. 2014;46(2):106–15.CrossRefGoogle Scholar
  2. 2.
    Johnson KD, Winkelman C. The effect of emergency department crowding on patient outcomes: a literature review. Adv Emerg Nurs J. 2011;33(1):39–54.CrossRefGoogle Scholar
  3. 3.
    Pines JM, Iyer S, Disbot M, Hollander JE, Shofer FS, Datner EM. The effect of emergency department crowding on patient satisfaction for admitted patients. Acad Emerg Med. 2008;15(9):825–31.CrossRefGoogle Scholar
  4. 4.
    Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang L-J, Han W, et al. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med.. 2013;61(6):605–11. e6.CrossRefGoogle Scholar
  5. 5.
    Chiu I-M, Lin Y-R, Syue Y-J, Kung C-T, Wu K-H, Li C-J. The influence of crowding on clinical practice in the emergency department. Am J Emerg Med. 2018;36(1):56–60.CrossRefGoogle Scholar
  6. 6.
    Farrohknia N, Castren M, Ehrenberg A, Lind L, Oredsson S, Jonsson H, et al. Emergency department triage scales and their components: a systematic review of the scientific evidence. Scand J Trauma Resuscitation Emerg Med.. 2011;19:42.CrossRefGoogle Scholar
  7. 7.
    Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Deutsches Arzteblatt Int. 2010;107(50):892–8.Google Scholar
  8. 8.
    McHugh M, Tanabe P, McClelland M, Khare RK. More Patients Are Triaged Using the Emergency Severity Index Than Any Other Triage Acuity System in the United States. Acad Emerg Med. 2012;19(1):106–9.CrossRefGoogle Scholar
  9. 9.
    Torabi M, Moeinaddini S, Mirafzal A, Rastegari A, Sadeghkhani N. Shock index, modified shock index, and age shock index for prediction of mortality in Emergency Severity Index level 3. Am J Emerg Med. 2016;34(11):2079–83.CrossRefGoogle Scholar
  10. 10.
    Levin S, Toerper M, Hamrock E, Hinson JS, Barnes S, Gardner H, et al. Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index. Ann Emerg Med. 2018;71(5):565–74 e2.CrossRefGoogle Scholar
  11. 11.
    Torabi M, Mirafzal A, Rastegari A, Sadeghkhani N. Association of triage time Shock Index, Modified Shock Index, and Age Shock Index with mortality in Emergency Severity Index level 2 patients. Am J Emerg Med. 2016;34(1):63–8.CrossRefGoogle Scholar
  12. 12.
    Arya R, Wei G, McCoy JV, Crane J, Ohman-Strickland P, Eisenstein RM. Decreasing Length of Stay in the Emergency Department With a Split Emergency Severity Index 3 Patient Flow Model. Acad Emerg Med. 2013;20(11):1171–9.CrossRefGoogle Scholar
  13. 13.
    Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, et al. Scalable and accurate deep learning with electronic health records. npj Digital Med.. 2018;1(1):18.CrossRefGoogle Scholar
  14. 14.
    Stewart J, Sprivulis P, Dwivedi G. Artificial intelligence and machine learning in emergency medicine. Emerg Med Australas. 0(0).Google Scholar
  15. 15.
    Coslovsky M, Takala J, Exadaktylos AK, Martinolli L, Merz TM. A clinical prediction model to identify patients at high risk of death in the emergency department. Intensive Care Med. 2015;41(6):1029–36.CrossRefGoogle Scholar
  16. 16.
    Pearl A, Bar-Or R, Bar-Or D. An artificial neural network derived trauma outcome prediction score as an aid to triage for non-clinicians. Stud Health Technol Inform. 2008;136:253.PubMedGoogle Scholar
  17. 17.
    Dugas AF, Kirsch TD, Toerper M, Korley F, Yenokyan G, France D, et al. An Electronic Emergency Triage System to Improve Patient Distribution by Critical Outcomes. J Emerg Med. 2016;50(6):910–8.CrossRefGoogle Scholar
  18. 18.
    Teubner DJ, Considine J, Hakendorf P, Kim S, Bersten AD. Model to predict inpatient mortality from information gathered at presentation to an emergency department: The Triage Information Mortality Model (TIMM). Emerg Med Australas. 2015;27(4):300–6.CrossRefGoogle Scholar
  19. 19.
    Schuetz P, Hausfater P, Amin D, Haubitz S, Fassler L, Grolimund E, et al. Optimizing triage and hospitalization in adult general medical emergency patients: the triage project. BMC Emerg Med. 2013;13:12.CrossRefGoogle Scholar
  20. 20.
    Barak-Corren Y, Israelit SH, Reis BY. Progressive prediction of hospitalisation in the emergency department: uncovering hidden patterns to improve patient flow. Emerg Med J. 2017;34(5):308–14.CrossRefGoogle Scholar
  21. 21.
    Sun Y, Heng BH, Tay SY, Seow E. Predicting hospital admissions at emergency department triage using routine administrative data. Acad Emerg Med. 2011;18(8):844–50.CrossRefGoogle Scholar
  22. 22.
    Barak-Corren Y, Fine AM, Reis BY. Early Prediction Model of Patient Hospitalization From the Pediatric Emergency Department. Pediatrics. 2017;139(5).CrossRefGoogle Scholar
  23. 23.
    Chen T, Guestrin C, editors. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining; 2016: ACM.Google Scholar
  24. 24.
    Breiman L. Random forests. Mach Learn. 2001;45(1):5–32.CrossRefGoogle Scholar
  25. 25.
    Biau G, Scornet E. A random forest guided tour. Test. 2016;25(2):197–227.CrossRefGoogle Scholar
  26. 26.
    Hong WS, Haimovich AD, Taylor RA. Predicting hospital admission at emergency department triage using machine learning. PLoS One. 2018;13(7):e0201016.CrossRefGoogle Scholar
  27. 27.
    Qiao Z, Sun N, Li X, Xia E, Zhao S, Qin Y. Using Machine Learning Approaches for Emergency Room Visit Prediction Based on Electronic Health Record Data. Stud Health Technol Inform. 2018;247:111–5.PubMedGoogle Scholar
  28. 28.
    Goto T, Camargo Jr CA, Faridi MK, Yun BJ, Hasegawa K. Machine learning approaches for predicting disposition of asthma and COPD exacerbations in the ED. Am J Emerg Med. 2018;36(9):1650–4.CrossRefGoogle Scholar
  29. 29.
    Bogle B, Balduino R, Wolk DM, Farag HA, Kethireddy S, Chatterjee A, et al. Predicting Mortality of Sepsis Patients in a Multi-Site Healthcare System using Supervised Machine Learning. Available at: Accessed July 1, 2019.
  30. 30.
    Ho EL, Tan I, Lee I, Wu P, Chong H. Predicting Readmission at Early Hospitalization Using Electronic Health Data: A Customized Model Development. Int J Integrated Care. 2017;17(5).Google Scholar
  31. 31.
    Taylor RA, Moore CL, Cheung K-H, Brandt C. Predicting urinary tract infections in the emergency department with machine learning. PLoS One. 2018;13(3):e0194085.CrossRefGoogle Scholar
  32. 32.
    Hill B, Brown RP, Gabel E, Lee C, Cannesson M, Loohuis LO, et al. Preoperative predictions of in-hospital mortality using electronic medical record data. bioRxiv. 2018:329813.Google Scholar
  33. 33.
    Maali Y, Perez-Concha O, Coiera E, Roffe D, Day RO, Gallego B. Predicting 7-day, 30-day and 60-day all-cause unplanned readmission: a case study of a Sydney hospital. BMC Med Inform Decis Mak. 2018;18(1):1.CrossRefGoogle Scholar
  34. 34.
    Awad A, Bader-El-Den M, McNicholas J, Briggs J. Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach. Int J Med Inform. 2017;108:185–95.CrossRefGoogle Scholar

Copyright information

© Society of General Internal Medicine 2019

Authors and Affiliations

  • Maximiliano Klug
    • 1
    • 2
  • Yiftach Barash
    • 1
    • 2
  • Sigalit Bechler
    • 3
  • Yehezkel S. Resheff
    • 3
  • Talia Tron
    • 3
  • Avi Ironi
    • 2
    • 4
  • Shelly Soffer
    • 1
    • 2
  • Eyal Zimlichman
    • 2
    • 5
  • Eyal Klang
    • 1
    • 2
    Email author
  1. 1.Department of Diagnostic Imaging The Chaim Sheba Medical CenterRamat GanIsrael
  2. 2.Sackler Faculty of MedicineTel Aviv UniversityTel AvivIsrael
  3. 3.Intuit Israel©Hod HasharonIsrael
  4. 4.Emergency RoomThe Chaim Sheba Medical CenterRamat GanIsrael
  5. 5.Hospital ManagementThe Chaim Sheba Medical CenterRamat GanIsrael

Personalised recommendations