Prioritizing Primary Care Patients for a Communication Intervention Using the “Surprise Question”: a Prospective Cohort Study

  • Joshua R. LakinEmail author
  • Margaret G. Robinson
  • Ziad Obermeyer
  • Brian W. Powers
  • Susan D. Block
  • Rebecca Cunningham
  • Joseph M. Tumblin
  • Christine Vogeli
  • Rachelle E. Bernacki
Original Research



Communication about priorities and goals improves the value of care for patients with serious illnesses. Resource constraints necessitate targeting interventions to patients who need them most.


To evaluate the effectiveness of a clinician screening tool to identify patients for a communication intervention.


Prospective cohort study.


Primary care clinics in Boston, MA.


Primary care physicians (PCPs) and nurse care coordinators (RNCCs) identified patients at high risk of dying by answering the Surprise Question (SQ): “Would you be surprised if this patient died in the next 2 years?”


Performance of the SQ for predicting mortality, measured by the area under receiver operating curve (AUC), sensitivity, specificity, and likelihood ratios.


Sensitivity of PCP response to the SQ at 2 years was 79.4% and specificity 68.6%; for RNCCs, sensitivity was 52.6% and specificity 80.6%. In univariate regression, the odds of 2-year mortality for patients identified as high risk by PCPs were 8.4 times higher than those predicted to be at low risk (95% CI 5.7–12.4, AUC 0.74) and 4.6 for RNCCs (3.4–6.2, AUC 0.67). In multivariate analysis, both PCP and RNCC prediction of high risk of death remained associated with the odds of 2-year mortality.


This study was conducted in the context of a high-risk care management program, including an initial screening process and training, both of which affect the generalizability of the results.


When used in combination with a high-risk algorithm, the 2-year version of the SQ captured the majority of patients who died, demonstrating better than expected performance as a screening tool for a serious illness communication intervention in a heterogeneous primary care population.


palliative care advance care planning patient identification end-of-life care 



We would like to thank Brandon Neal for his analytical support.


The implementation of the Serious Illness Care Program at Brigham and Women’s Hospital was funded by Partners HealthCare and the Charina and Branta Foundations.

Compliance with Ethical Standards

The institutional review board of Partners HealthCare approved this study.

Conflict of Interest

Dr. Lakin and Dr. Bernacki are supported by the Cambia Sojourns Scholar Leadership Program. Dr. Block serves as the Palliative Care editor for UpToDate. The remainder of the authors declare no conflicts of interest.


  1. 1.
    Tulsky JA. Improving quality of care for serious illness: findings and recommendations of the Institute of Medicine report on dying in America. JAMA internal medicine. 2015;175(5):840–1. Scholar
  2. 2.
    Teno JM, Clarridge BR, Casey V, Welch LC, Wetle T, Shield R, et al. Family perspectives on end-of-life care at the last place of care. JAMA : the journal of the American Medical Association. 2004;291(1):88–93. Scholar
  3. 3.
    Wright AA, Keating NL, Balboni TA, Matulonis UA, Block SD, Prigerson HG. Place of death: correlations with quality of life of patients with cancer and predictors of bereaved caregivers’ mental health. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. 2010;28(29):4457–64. CrossRefGoogle Scholar
  4. 4.
    Meier DE. Increased access to palliative care and hospice services: opportunities to improve value in health care. The Milbank quarterly. 2011;89(3):343–80. CrossRefPubMedCentralGoogle Scholar
  5. 5.
    Bell CL, Somogyi-Zalud E, Masaki KH. Factors associated with congruence between preferred and actual place of death. Journal of pain and symptom management. 2010;39(3):591–604. CrossRefPubMedCentralGoogle Scholar
  6. 6.
    Dahlin CM, Kelley JM, Jackson VA, Temel JS. Early palliative care for lung cancer: improving quality of life and increasing survival. International journal of palliative nursing. 2010;16(9):420–3CrossRefGoogle Scholar
  7. 7.
    Institute of Medicine. Dying in America: Improving Quality and Honoring Individual Preferences Near the End of Life. Washington, DC: National Academies Press; 2014.Google Scholar
  8. 8.
    Miller SC, Mor V, Wu N, Gozalo P, Lapane K. Does receipt of hospice care in nursing homes improve the management of pain at the end of life? Journal of the American Geriatrics Society. 2002;50(3):507–15CrossRefGoogle Scholar
  9. 9.
    Miller SC, Mor V, Teno J. Hospice enrollment and pain assessment and management in nursing homes. Journal of pain and symptom management. 2003;26(3):791–9CrossRefGoogle Scholar
  10. 10.
    Teno JM, Gozalo PL, Lee IC, Kuo S, Spence C, Connor SR, et al. Does hospice improve quality of care for persons dying from dementia?. Journal of the American Geriatrics Society. 2011;59(8):1531–6. CrossRefPubMedCentralGoogle Scholar
  11. 11.
    Teno JM, Shu JE, Casarett D, Spence C, Rhodes R, Connor S. Timing of referral to hospice and quality of care: length of stay and bereaved family members’ perceptions of the timing of hospice referral. Journal of pain and symptom management. 2007;34(2):120–5. Scholar
  12. 12.
    Pyenson B, Connor S, Fitch K, Kinzbrunner B. Medicare cost in matched hospice and non-hospice cohorts. Journal of pain and symptom management. 2004;28(3):200–10. Scholar
  13. 13.
    Temel JS, Greer JA, Muzikansky A, Gallagher ER, Admane S, Jackson VA, et al. Early palliative care for patients with metastatic non-small-cell lung cancer. The New England journal of medicine. 2010;363(8):733–42. Scholar
  14. 14.
    Obermeyer Z, Makar M, Abujaber S, Dominici F, Block S, Cutler D. Association between the Medicare hospice benefit and health care utilization and costs for patients with poor-prognosis cancer. JAMA : the journal of the American Medical Association. 2014;312(18):1888–96.CrossRefGoogle Scholar
  15. 15.
    Gazelle G. Understanding hospice--an underutilized option for life’s final chapter. The New England journal of medicine. 2007;357(4):321–4. Scholar
  16. 16.
    Teno JM, Gozalo PL, Bynum JP, Leland NE, Miller SC, Morden NE, et al. Change in end-of-life care for Medicare beneficiaries: site of death, place of care, and health care transitions in 2000, 2005, and 2009. JAMA : the journal of the American Medical Association. 2013;309(5):470–7. CrossRefGoogle Scholar
  17. 17.
    Quill TE, Abernethy AP. Generalist plus specialist palliative care--creating a more sustainable model. The New England journal of medicine. 2013;368(13):1173–5. Scholar
  18. 18.
    Tolle SW, Back AL, Meier DE. Clinical decisions. End-of-life advance directive. The New England journal of medicine. 2015;372(7):667–70. Scholar
  19. 19.
    Lakin JR, Block SD, Billings JA, Koritsanszky LA, Cunningham R, Wichmann L, et al. Improving Communication About Serious Illness in Primary Care: A Review. JAMA internal medicine. 2016.
  20. 20.
    Lupu D. Estimate of current hospice and palliative medicine physician workforce shortage. Journal of pain and symptom management. 2010;40(6):899–911. Scholar
  21. 21.
    Lum HD, Sudore RL, Bekelman DB. Advance care planning in the elderly. Med Clin North Am. 2015;99(2):391–403. Scholar
  22. 22.
    Billings JA, Bernacki R. Strategic targeting of advance care planning interventions: the Goldilocks phenomenon. JAMA internal medicine. 2014;174(4):620–4. Scholar
  23. 23.
    Block SD, Bernier GM, Crawley LM, Farber S, Kuhl D, Nelson W, et al. Incorporating palliative care into primary care education. National Consensus Conference on Medical Education for Care Near the End of Life. Journal of general internal medicine. 1998;13(11):768–73CrossRefPubMedCentralGoogle Scholar
  24. 24.
    Pattison M, Romer AL. Improving Care Through the End of Life: launching a primary care clinic-based program. Journal of palliative medicine. 2001;4(2):249–54CrossRefGoogle Scholar
  25. 25.
    Pang WF, Kwan BC, Chow KM, Leung CB, Li PK, Szeto CC. Predicting 12-month mortality for peritoneal dialysis patients using the “surprise” question. Peritoneal dialysis international : journal of the International Society for Peritoneal Dialysis. 2013;33(1):60–6. CrossRefGoogle Scholar
  26. 26.
    Cohen LM, Ruthazer R, Moss AH, Germain MJ. Predicting six-month mortality for patients who are on maintenance hemodialysis. Clinical journal of the American Society of Nephrology : CJASN. 2010;5(1):72–9. CrossRefGoogle Scholar
  27. 27.
    Moss AH, Lunney JR, Culp S, Auber M, Kurian S, Rogers J, et al. Prognostic significance of the “surprise” question in cancer patients. Journal of palliative medicine. 2010;13(7):837–40. Scholar
  28. 28.
    Moss AH, Ganjoo J, Sharma S, Gansor J, Senft S, Weaner B, et al. Utility of the “surprise” question to identify dialysis patients with high mortality. Clinical journal of the American Society of Nephrology : CJASN. 2008;3(5):1379–84. CrossRefGoogle Scholar
  29. 29.
    Moroni M, Zocchi D, Bolognesi D, Abernethy A, Rondelli R, Savorani G, et al. The ‘surprise’ question in advanced cancer patients: A prospective study among general practitioners. Palliative medicine. 2014;28(7):959–64. Scholar
  30. 30.
    Yourman LC, Lee SJ, Schonberg MA, Widera EW, Smith AK. Prognostic indices for older adults: a systematic review. JAMA : the journal of the American Medical Association. 2012;307(2):182–92. CrossRefGoogle Scholar
  31. 31.
    Lakin JR, Robinson MG, Bernacki RE, Powers BW, Block SD, Cunningham R, et al. Estimating 1-Year Mortality for High-Risk Primary Care Patients Using the “Surprise” Question. JAMA internal medicine. 2016.
  32. 32.
    White N, Kupeli N, Vickerstaff V, Stone P. How accurate is the ‘Surprise Question’ at identifying patients at the end of life? A systematic review and meta-analysis. BMC medicine. 2017;15(1):139. CrossRefPubMedCentralGoogle Scholar
  33. 33.
    Downar J, Goldman R, Pinto R, Englesakis M, Adhikari NK. The “surprise question” for predicting death in seriously ill patients: a systematic review and meta-analysis. CMAJ : Canadian Medical Association journal = journal de l’Association medicale canadienne. 2017;189(13):E484-e93. CrossRefGoogle Scholar
  34. 34.
    Bernacki R, Hutchings M, Vick J, Smith G, Paladino J, Lipsitz S, et al. Development of the Serious Illness Care Program: a randomised controlled trial of a palliative care communication intervention. BMJ open. 2015;5(10):e009032. CrossRefPubMedCentralGoogle Scholar
  35. 35.
    Lakin JR, Koritsanszky LA, Cunningham R, Maloney FL, Neal BJ, Paladino J, et al. A Systematic Intervention To Improve Serious Illness Communication In Primary Care. Health affairs (Project Hope). 2017;36(7):1258–64. Scholar
  36. 36.
    Haime V, Hong C, Mandel L, Mohta N, Iezzoni LI, Ferris TG, et al. Clinician considerations when selecting high-risk patients for care management. The American journal of managed care. 2015;21(10):e576–82Google Scholar
  37. 37.
    Hsu J, Price M, Vogeli C, Brand R, Chernew ME, Chaguturu SK, et al. Bending The Spending Curve By Altering Care Delivery Patterns: The Role Of Care Management Within A Pioneer ACO. Health affairs (Project Hope). 2017;36(5):876–84. CrossRefGoogle Scholar
  38. 38.
    Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of biomedical informatics. 2009;42(2):377–81. CrossRefGoogle Scholar
  39. 39.
    Gagne JJ, Glynn RJ, Avorn J, Levin R, Schneeweiss S. A combined comorbidity score predicted mortality in elderly patients better than existing scores. Journal of clinical epidemiology. 2011;64(7):749–59. CrossRefPubMedCentralGoogle Scholar
  40. 40.
    Sun JW, Rogers JR, Her Q, Welch EC, Panozzo CA, Toh S, et al. Adaptation and Validation of the Combined Comorbidity Score for ICD-10-CM. Medical care. 2017;55(12):1046–51. Scholar
  41. 41.
    Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. Scholar
  42. 42.
    Gomez-Batiste X, Martinez-Munoz M, Blay C, Amblas J, Vila L, Costa X, et al. Utility of the NECPAL CCOMS-ICO((c)) tool and the Surprise Question as screening tools for early palliative care and to predict mortality in patients with advanced chronic conditions: A cohort study. Palliative medicine. 2017;31(8):754–63. Scholar
  43. 43.
    Barnes S, Gott M, Payne S, Parker C, Seamark D, Gariballa S, et al. Predicting mortality among a general practice-based sample of older people with heart failure. Chronic Illn. 2008;4(1):5–12. Scholar
  44. 44.
    Einav L, Finkelstein A, Mullainathan S, Obermeyer Z. Predictive modeling of U.S. health care spending in late life. Science (New York, NY). 2018;360(6396):1462–5. CrossRefGoogle Scholar
  45. 45.
    Murphy DR, Meyer AN, Russo E, Sittig DF, Wei L, Singh H. The Burden of Inbox Notifications in Commercial Electronic Health Records. JAMA internal medicine. 2016;176(4):559–60. Scholar
  46. 46.
    Prochaska JO, Velicer WF. The transtheoretical model of health behavior change. Am J Health Promot. 1997;12(1):38–48CrossRefGoogle Scholar

Copyright information

© Society of General Internal Medicine 2019

Authors and Affiliations

  • Joshua R. Lakin
    • 1
    • 2
    • 3
    • 4
    Email author
  • Margaret G. Robinson
    • 5
  • Ziad Obermeyer
    • 3
    • 6
    • 7
  • Brian W. Powers
    • 4
  • Susan D. Block
    • 1
    • 2
    • 3
    • 4
    • 8
  • Rebecca Cunningham
    • 4
  • Joseph M. Tumblin
    • 9
  • Christine Vogeli
    • 10
    • 9
    • 11
  • Rachelle E. Bernacki
    • 1
    • 2
    • 3
    • 4
  1. 1.Department of Psychosocial Oncology and Palliative CareDana-Farber Cancer InstituteBostonUSA
  2. 2.Division of Palliative MedicineBrigham and Women’s HospitalBostonUSA
  3. 3.Ariadne LabsBrigham and Women’s Hospital & Harvard School of Public HealthBostonUSA
  4. 4.Department of MedicineBrigham and Women’s HospitalBostonUSA
  5. 5.Stanford University School of MedicineStanfordUSA
  6. 6.Department of Emergency MedicineBrigham and Women’s HospitalBostonUSA
  7. 7.Departments of Emergency Medicine and Health Care PolicyHarvard Medical SchoolBostonUSA
  8. 8.Department of PsychiatryBrigham and Women’s HospitalBostonUSA
  9. 9.Partners HealthCareBostonUSA
  10. 10.Harvard Medical SchoolBostonUSA
  11. 11.Department of MedicineMassachusetts General HospitalBostonUSA

Personalised recommendations