“I Think I Was Losing the Forest for the Trees”: Evaluation of an Internal Medicine Residency Quality Improvement Curriculum



Quality improvement (QI) training during residency may not be adequately preparing physicians for achieving Accreditation Council for Graduate Medical Education goals and the Institute for Healthcare Improvement (IHI) Triple Aim. The purpose of this evaluation was to identify residents’ perceptions and impact of their QI curriculum.


We conducted a mixed-methods evaluation of an active-learning QI curriculum for internal medicine residents at one academic medical center. Data from 2017 to 2018 included a focus group, pre-post survey, project data, and curricular materials. Results were categorized using Kirkpatrick’s model of evaluation.


All second-year internal medicine residents completed the curriculum (N = 14). Residents were satisfied with the structure and perceived accomplishment with the curriculum, however were dissatisfied by the impact of inconsistent attendance due to clinical conflicts. Their confidence in QI increased; however, they reported difficulty retaining knowledge and skills. Survey scores related to usefulness and anticipated application of QI were unchanged from baseline.


This applied QI curriculum appeared to improve short-term learning. However, the curriculum did not promote long-term understanding of QI. Finding ways to promote skills and retention beyond the curriculum requires further study.

This is a preview of subscription content, log in to check access.


  1. 1.

    Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Project Hope). 2008;27(3):759–69. https://doi.org/10.1377/hlthaff.27.3.759.

    Article  Google Scholar 

  2. 2.

    Institute of Medicine (U.S.). Committee on Quality of Health Care in America. Crossing the quality chasm : a new health system for the 21st century. Washington, D.C.: National Academy Press; 2001.

    Google Scholar 

  3. 3.

    ACGME. The Internal Medicine Milestone Project 2015. https://www.acgme.org/Portals/0/PDFs/Milestones/InternalMedicineMilestones.pdf. Accessed 13 Feb 2019.

  4. 4.

    Massagli TL, Zumsteg JM, Osorio MB. Quality improvement education in residency training: a review. Am J Phys Med Rehabil. 2018;97(9):673–8. https://doi.org/10.1097/phm.0000000000000947.

    Article  Google Scholar 

  5. 5.

    Patow CA, Karpovich K, Riesenberg LA, Jaeger J, Rosenfeld JC, Wittenbreer M, et al. Residents’ engagement in quality improvement: a systematic review of the literature. Acad Med. 2009;84(12):1757–64. https://doi.org/10.1097/ACM.0b013e3181bf53ab.

    Article  Google Scholar 

  6. 6.

    Wong BM, Levinson W, Shojania KG. Quality improvement in medical education: current state and future directions. Med Educ. 2012;46(1):107–19. https://doi.org/10.1111/j.1365-2923.2011.04154.x.

    Article  Google Scholar 

  7. 7.

    Singh MK, Ogrinc G, Cox KR, Dolansky M, Brandt J, Morrison LJ, et al. The Quality Improvement Knowledge Application Tool Revised (QIKAT-R). Acad Med. 2014;89(10):1386–91. https://doi.org/10.1097/acm.0000000000000456.

    Article  Google Scholar 

  8. 8.

    Butler JM, Anderson KA, Supiano MA, Weir CR. “It feels like a lot of extra work”: resident attitudes about quality improvement and implications for an effective learning health care system. Acad Med. 2017;92(7):984–90. https://doi.org/10.1097/acm.0000000000001474.

    Article  Google Scholar 

  9. 9.

    Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s four levels of training evaluation. Alexandria: ATD Press; 2016.

    Google Scholar 

  10. 10.

    Wong BM, Goldman J, Goguen JM, Base C, Rotteau L, Van Melle E, et al. Faculty-resident “co-learning”: a longitudinal exploration of an innovative model for faculty development in quality improvement. Acad Med. 2017;92(8):1151–9. https://doi.org/10.1097/acm.0000000000001505.

    Article  Google Scholar 

  11. 11.

    Langley GJ. The improvement guide: a practical approach to enhancing organizational performance. 2nd ed. San Francisco: Jossey-Bass; 2009.

    Google Scholar 

  12. 12.

    Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986–92. https://doi.org/10.1136/bmjqs-2015-004411.

    Article  Google Scholar 

  13. 13.

    Krueger RA, Casey MA. Focus groups : a practical guide for applied research. 5th edition. Ed. SAGE: Thousand Oaks, California; 2015.

    Google Scholar 

  14. 14.

    Crabtree BF, Miller WL. Doing qualitative research. 2nd ed. Thousand Oaks: Sage Publications; 1999.

    Google Scholar 

  15. 15.

    Moreau KA. Has the new Kirkpatrick generation built a better hammer for our evaluation toolbox? Med Teach. 2017;39(9):999–1001. https://doi.org/10.1080/0142159x.2017.1337874.

    Article  Google Scholar 

Download references


The authors wish to thank Kathryn N. Huggett, Ph.D., Director of The Teaching Academy at the Larner College of Medicine at the University of Vermont for her assistance in facilitating the focus group session with the residents.

Author information



Corresponding author

Correspondence to Amanda G. Kennedy.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Research Involving Human Participants

The evaluation was reviewed by the University of Vermont Committees on Human Research under a determination of not research (e.g. quality assurance, educational evaluation).

Informed Consent

Physician residents were informed that their participation in the focus group and surveys was voluntary and that their responses may be used in a manuscript. There was no formal informed consent document, as this evaluation was reviewed by the University of Vermont Committees on Human Research under a determination of not research (e.g., quality assurance, educational evaluation).

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kennedy, A.G., Burnett, M., Muthukrishnan, P. et al. “I Think I Was Losing the Forest for the Trees”: Evaluation of an Internal Medicine Residency Quality Improvement Curriculum. Med.Sci.Educ. 30, 197–202 (2020). https://doi.org/10.1007/s40670-019-00854-7

Download citation


  • Curriculum
  • Education
  • Medical
  • Graduate/standards
  • Internship and Residency
  • Quality Improvement
  • Quality of Health Care