Debiasing versus knowledge retrieval checklists to reduce diagnostic error in ECG interpretation

  • Matt SibbaldEmail author
  • Jonathan Sherbino
  • Jonathan S. Ilgen
  • Laura Zwaan
  • Sarah Blissett
  • Sandra Monteiro
  • Geoffrey Norman


There is an ongoing debate regarding the cause of diagnostic errors. One view is that errors result from unconscious application of cognitive heuristics; the alternative is that errors are a consequence of knowledge deficits. The objective of this study was to compare the effectiveness of checklists that (a) identify and address cognitive biases or (b) promote knowledge retrieval, as a means to reduce errors in ECG interpretation. Novice postgraduate year (PGY) 1 emergency medicine and internal medicine residents (n = 40) and experienced cardiology fellows (PGY 4–6) (n = 21) were randomly allocated to three conditions: a debiasing checklist, a content (knowledge) checklist, or control (no checklist) to be used while interpreting 20 ECGs. Half of the ECGs were deliberately engineered to predispose to bias. Diagnostic performance under either checklist intervention was not significantly better than the control. As expected, more errors occurred when cases were designed to induce bias (F = 96.9, p < 0.0001). There was no significant interaction between the instructional condition and level of learner. Checklists attempting to help learners identify cognitive bias or mobilize domain-specific knowledge did not have an overall effect in reducing diagnostic errors in ECG interpretation, although they may help novices. Even when cognitive biases are deliberately inserted in cases, cognitive debiasing checklists did not improve participants’ performance.


Clinical reasoning Checklists Bias Diagnostic error 



The authors wish to thank Betty Howey for her technical support and facilitation of the study.


This study was supported by a Medical Education Grant from the Royal College of Physicians and Surgeons of Canada.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This study was approved by the Hamilton Integrated Ethics Board September 2017, #3856.


  1. Berner, E. S., Maisiak, R. S., Cobbs, C. G., & Taunton, O. D. (1999). Effects of a decision support system on physicians’ diagnostic performance. Journal of the American Medical Informatics Association, 6(5), 420–427. Scholar
  2. Blumenthal-Barby, J. S., & Krieger, H. (2015). Cognitive biases and heuristics in medical decision making. Medical Decision Making, 35(4), 539–557. Scholar
  3. Christensen, C., Heckerling, P. S., Mackesy, M. E., Bernstein, L. M., & Elstein, A. S. (1991). Framing bias among expert and novice physicians. Academic Medicine, 66(9), S76–S78. Scholar
  4. Croskerry, P. (2003a). Cognitive forcing strategies in clinical decisionmaking. Annals of Emergency Medicine, 41(1), 110–120. Scholar
  5. Croskerry, P. (2003b). The importance of cognitive errors in diagnosis and strategies to minimize them. Academic Medicine, 78(8). Retrieved from
  6. Croskerry, P. (2009). A universal model of diagnostic reasoning. Academic Medicine, 84(8), 1022–1028. Scholar
  7. Croskerry, P., Singhal, G., & Mamede, S. (2013a). Cognitive debiasing 1: Origins of bias and theory of debiasing. BMJ Quality & Safety, 22(Suppl 2), ii58 LP–ii64. Retrieved from
  8. Croskerry, P., Singhal, G., & Mamede, S. (2013b). Cognitive debiasing 2: Impediments to and strategies for change. BMJ Quality & Safety, 22(Suppl 2), ii65 LP–ii72. Retrieved from
  9. Dhaliwal, G. (2017). Premature closure? Not so fast. BMJ Quality & Safety, 26(2), 87–89. Scholar
  10. Donaldson, M. S., Corrigan, J. M., & Kohn, L. T. (2000). To err is human: Building a safer health system. Washington: National Academies Press.Google Scholar
  11. ECG Made Simple. (2017). Stepwise videos. Retrieved from: Accessed 16 Sept 2017.
  12. Elstein, A. S., Friedman, C. P., Wolf, F. M., Murphy, G., Miller, J., Fine, P., et al. (1996). Effects of a decision support system on the diagnostic accuracy of users: A preliminary report. Journal of the American Medical Informatics Association, 3(6), 422–428. Scholar
  13. Ely, J. W., Graber, M. L., & Croskerry, P. (2011). Checklists to reduce diagnostic errors. Academic Medicine, 86(3), 307–313. Scholar
  14. Eva, K. W., & Regehr, G. (2011). Exploring the divergence between self-assessment and self-monitoring. Advances in Health Sciences Education, 16(3), 311–329. Scholar
  15. Friedman, C. P., Elstein, A. S., Wolf, F. M., Murphy, G. C., Franz, T. M., Heckerling, P. S., et al. (1999). Enhancement of clinicians’ diagnostic reasoning by computer-based consultation. JAMA, 282(19), 1851. Scholar
  16. Graber, M. L., Kissam, S., Payne, V. L., Meyer, A. N. D., Sorensen, A., Lenfestey, N., et al. (2012). Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Quality & Safety, 21(7), 535 LP–557. Retrieved from
  17. Haenssle, H. A., et al. (2018). Man against machine: Diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Annals of Oncology, 29(8), 1836–1842.CrossRefGoogle Scholar
  18. Hatala, R., Norman, G. R., & Brooks, L. R. (1999). Impact of a clinical scenario on accuracy of electrocardiogram interpretation. Journal of General Internal Medicine, 14(2), 126–129. Scholar
  19. Haynes, A. B., Weiser, T. G., Berry, W. R., Lipsitz, S. R., Breizat, A.-H. S., Dellinger, E. P., et al. (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine, 360(5), 491–499. Scholar
  20. Kahneman, D., & Egan, P. (2011). Thinking, fast and slow. New York: Farrar, Straus, and Giroux.Google Scholar
  21. Mamede, S., & Schmidt, H. G. (2014). Reflection in diagnostic reasoning. Academic Medicine, 89(7), 959–960. Scholar
  22. Mamede, S., Schmidt, H. G., & Penaforte, J. C. (2008). Effects of reflective practice on the accuracy of medical diagnoses. Medical Education, 42(5), 468–475. Scholar
  23. Mamede, S., van Gog, T., van den Berge, K., Rikers, R. M. J. P., van Saase, J. L. C. M., van Guldener, C., et al. (2010). Effect of availability bias and reflective reasoning on diagnostic accuracy among internal medicine residents. JAMA, 304(11), 1198. Scholar
  24. Marsteller, J. A., Sexton, J. B., Hsu, Y.-J., Hsiao, C.-J., Holzmueller, C. G., Pronovost, P. J., et al. (2012). A multicenter, phased, cluster-randomized controlled trial to reduce central line-associated bloodstream infections in intensive care units*. Critical Care Medicine, 40(11), 2933–2939. Scholar
  25. Monteiro, S. D., Sherbino, J., Patel, A., Mazzetti, I., Norman, G. R., & Howey, E. (2015). Reflecting on diagnostic errors: Taking a second look is not enough. Journal of General Internal Medicine, 30(9), 1270–1274. Scholar
  26. Norman, G. R., Brooks, L. R., Colle, C. L., & Hatala, R. M. (1999). The benefit of diagnostic hypotheses in clinical reasoning: Experimental study of an instructional intervention for forward and backward reasoning. Cognition and Instruction, 17(4), 433–448. Scholar
  27. Norman, G. R., & Eva, K. W. (2009). Diagnostic error and clinical reasoning. Medical Education, 44(1), 94–100. Scholar
  28. Obermeyer, Z., & Emanuel, E. J. (2016). Predicting the future—Big data, machine learning, and clinical medicine. The New England Journal of Medicine, 375(13), 1216.CrossRefGoogle Scholar
  29. Reilly, J. B., Ogdie, A. R., Von Feldt, J. M., & Myers, J. S. (2013). Teaching about how doctors think: A longitudinal curriculum in cognitive bias and diagnostic error for residents. BMJ Quality & Safety, 22(12), 1044 LP–1050. Retrieved from
  30. Sherbino, J., Dore, K. L., Siu, E., & Norman, G. R. (2011). The effectiveness of cognitive forcing strategies to decrease diagnostic error: An exploratory study. Teaching and Learning in Medicine, 23(1), 78–84. Scholar
  31. Sherbino, J., Dore, K. L., Wood, T. J., Young, M. E., Gaissmaier, W., Kreuger, S., et al. (2012). The relationship between response time and diagnostic accuracy. Academic Medicine, 87(6), 785–791. Scholar
  32. Sherbino, J., Kulasegaram, K., Howey, E., & Norman, G. (2014). Ineffectiveness of cognitive forcing strategies to reduce biases in diagnostic reasoning: A controlled trial. CJEM, 16(01), 34–40. Scholar
  33. Shimizu, T., Matsumoto, K., & Tokuda, Y. (2013). Effects of the use of differential diagnosis checklist and general de-biasing checklist on diagnostic performance in comparison to intuitive diagnosis. Medical Teacher, 35(6), e1218–e1229. Scholar
  34. Sibbald, M., de Bruin, A. B. H., & van Merrienboer, J. J. G. (2013). Checklists improve experts’ diagnostic decisions. Medical Education, 47(3), 301–308. Scholar
  35. Sibbald, M., De Bruin, A. B. H., & van Merrienboer, J. J. G. (2014). Finding and fixing mistakes: Do checklists work for clinicians with different levels of experience? Advances in Health Sciences Education, 19(1), 43–51. Scholar
  36. Zwaan, L., Monteiro, S., Sherbino, J., Ilgen, J., Howey, B., & Norman, G. (2017). Is bias in the eye of the beholder? A vignette study to assess recognition of cognitive biases in clinical case workups. BMJ Quality & Safety, 26(2), 104 LP–110. Retrieved from

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Department of Medicine, Centre for Simulation Based LearningMcMaster UniversityHamiltonCanada
  2. 2.McMaster Faculty of Health Sciences Education Research, Innovation and Theory ProgramMcMaster UniversityHamiltonCanada
  3. 3.Department of Emergency Medicine and Center for Leadership and Innovation in Medical EducationUniversity of WashingtonSeattleUSA
  4. 4.Institute of Medical Education Research RotterdamErasmus MCRotterdamThe Netherlands
  5. 5.University of CaliforniaSan FranciscoUSA
  6. 6.Department of Health Research Methods, Evidence and ImpactMcMaster UniversityHamiltonCanada

Personalised recommendations