Skip to main content
SpringerLink
Log in

Evaluation of Anesthesiology Residents

  • Chapter
  • First Online:
Comprehensive Guide to Education in Anesthesia

Abstract

During the early days of graduate medical education, an apprenticeship model was followed, and evaluation was directly based on daily performance. As time evolved global assessment and high-stakes written examinations came to be the norm. With increasing pressure to demonstrate the impact of graduate medical education on outcome, there has been a movement toward assessment and selection of assessment tools with appropriate structure, objectivity, and validity. While the role of high-stakes examinations has not been eliminated, it is also clear that their impact does not necessarily encourage learning and may, in fact, discourage long-term retention of knowledge. There is increasing knowledge about the optimum characteristics of assessment tools and about the performance of these tools individually. An ideal assessment program contains multiple tools, is administered with timely feedback to the learner, and is designed to fit the learning environment. Options include audits, direct observation, mentorship, objective structured clinical examinations, oral exams, peer review, portfolio assessment, self-assessment, simulation, standardized patients, and written examinations. With upcoming changes in the accreditation process by the Accreditation Council for Graduate Medical Education, time will be replaced by criteria for performance in advancement and promotion of residents, using “milestones.” Optimum assessment in anesthesiology residency is possible, but must overcome obstacles, including “I know it when I see it,” “studying to the test,” the halo effect, grade inflation, and the “hidden curriculum.” Evidence of success in the future may be obtained by programs based on the performance of their graduates.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Abbreviations

ABA:

American Board of Anesthesiology

ACGME:

Accreditation Council for Graduate Medical Education

AKT:

Anesthesia Knowledge Test

AOS:

Area of strength

CA-1:

Clinical anesthesia, year 1

CA-2:

Clinical anesthesia, year 2

CA-3:

Clinical anesthesia, year 3

GME:

Graduate medical education

ICU:

Intensive care unit

ITE:

In-training examination

MCQ:

Multiple choice question examination

OSCE:

Objective structured clinical examination

PACU:

Post-anesthesia care unit

PBLI:

Practice-based learning and improvement

RIME:

“Reporter,” “interpreter,” “manager,” or “educator”

RRC:

Residency Review Committee

SP:

Standard patient

TAFI:

Targeted area for improvement

References

  1. Loyd GE, Koenig HM. Assessment of learning outcomes. Summative evaluations. Int Anesthesiol Clin. 2008;46:97–111.

    Article  PubMed  Google Scholar 

  2. Glance LG, Neuman M, Martinez EA, Pauker KY, Dutton RP. Performance measures at a “tipping point”. Anesth Analg. 2011;112:958–66.

    Article  PubMed  Google Scholar 

  3. Accreditation Council for Graduate Medical Education. Common program requirements: general competencies. http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs2013.pdf. Accessed 23 Sep 2013.

  4. Pangaro L. A new vocabulary and other innovations for improving descriptive in-training evaluations. Acad Med. 1999;74:1203–7.

    Article  PubMed  CAS  Google Scholar 

  5. Tetzlaff JE. Assessment of competence in anesthesiology. Anesthesiology. 2007;106:812–25.

    Article  PubMed  Google Scholar 

  6. Turnbull J, Gray J, MacFadyen J. Improving in-training evaluation programs. J Gen Intern Med. 1998;13:317–23.

    Article  PubMed  CAS  Google Scholar 

  7. De Oliveira Filho GR, Vieira JE. The relationship of learning environment, quality of life, and study strategies measures to anesthesiology resident academic performance. Anesth Analg. 2007;104:1467–72.

    Article  PubMed  Google Scholar 

  8. Shumway JM, Harden RM. AMEE guide no. 25: The assessment of learning outcomes for the competent and reflective physician. Med Teach. 2003;25:569–84.

    Article  PubMed  CAS  Google Scholar 

  9. Friedman Ben-David M. AMEE guide no. 14: outcome-based education: part 3—assessment in outcome-based education. Med Teach. 1999;21:121–3.

    Google Scholar 

  10. Williams III RG. Use of NBME and USMLE examinations to evaluate medical education programs. Acad Med. 1993;68:748–52.

    Article  PubMed  CAS  Google Scholar 

  11. Norcini JJ, Swanson DB, Grosso LF, Shea JA, Webster GD. A comparison of knowledge, synthesis and clinical judgment: multiple choice questions in the assessment of physician competence. Eval Health Prof. 1984;7:485–500.

    Article  PubMed  CAS  Google Scholar 

  12. Slogoff S, Hughes FP. Validity of scoring “dangerous answers” on a written certification examination. J Med Educ. 1987;62:625–31.

    PubMed  CAS  Google Scholar 

  13. Friedman Ben-David M. The role of assessment in expanding professional horizons. Med Teach. 2000;22:9–16.

    Google Scholar 

  14. Hawkins RE, Sumption KF, Gaglione MM, Holmbor ES. The in training examination in internal medicine: resident perceptions and the lack of correlation between resident score and faculty prediction of resident performance. Am J Med. 1999;106:206–10.

    Article  PubMed  CAS  Google Scholar 

  15. Wise S, Stagg L, Szucs R, Gay S, Mauger D, Hartman D. Assessment of resident knowledge: subjective assessment versus performance on the ACR in-training examination. Acad Radiol. 1999;6:66–71.

    Article  PubMed  CAS  Google Scholar 

  16. Schwartz RW, Donnelly MB, Sloan DA, Johnson SB, Strodel WE. Assessing senior residents’ knowledge and performance: an integrated evaluation program. Surgery. 1994;116:634–40.

    PubMed  CAS  Google Scholar 

  17. Baker K. Clinical teaching improves with resident evaluation and feedback. Anesthesiology. 2010;113:693–703.

    PubMed  Google Scholar 

  18. Norman GR, van der Vleuten CPM, de Graffe E. Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability. Med Educ. 1991;25:119–26.

    Article  PubMed  CAS  Google Scholar 

  19. Pangaro LN. Investing in descriptive evaluation: a vision for the future of assessment. Med Teach. 2000;22:478–81.

    Article  PubMed  CAS  Google Scholar 

  20. Hemmer PA, Pangaro LN. The effect of formal evaluation sessions during clinical clerkships in better identifying students with marginal fund of knowledge. Acad Med. 1997;72:641–3.

    Article  PubMed  CAS  Google Scholar 

  21. Hemmer PA, Hawkins R, Jackson JL, Pangaro LN. Assessing how well three evaluation methods detect deficiencies in medical students’ professionalism in two settings of an internal medicine clerkship. Acad Med. 2000;75:167–73.

    Article  PubMed  CAS  Google Scholar 

  22. Lavin B, Pangaro LN. Internship ratings as a validity measure for an evaluation system to identify inadequate clerkship performance. Acad Med. 1998;75:998–1002.

    Article  Google Scholar 

  23. Rose SH, Burkle CM. Accreditation Council for Graduate Medical Education competencies and the American Board of Anesthesiology Clinical Competence Committee: a comparison. Anesth Analg. 2006;102:212–6.

    Article  PubMed  Google Scholar 

  24. Baker K. Determining resident performance. Getting beyond the noise. Anesthesiology. 2011;115:862–78.

    Article  PubMed  Google Scholar 

  25. Tetzlaff JE. Assessment of competence in anesthesiology. Curr Opin Anaesthesiol. 2009;22:809–13.

    Article  PubMed  Google Scholar 

  26. Wainwright JR, Sullivan FM, Morrison JM, MacNaughton RJ, McConnadrie A. Audit encourages an evidence-based approach to medical practice. Med Educ. 1999;33:907–14.

    Article  PubMed  CAS  Google Scholar 

  27. Epstein RM. Assessment in medical education. N Engl J Med. 2007;356:387–96.

    Article  PubMed  CAS  Google Scholar 

  28. Noel G, Herbers J, Caplow M, Cooper G, Pangaro L, Harvey J. How well do internal medicine faculty members evaluate the clinical skills of residents? Ann Intern Med. 1992;117(9):757–65.

    Article  PubMed  CAS  Google Scholar 

  29. Herberts J, Gordon N, Cooper G, Harvey J, Pangaro L, Weaver M. How accurate are faculty evaluations of clinical competence? J Gen Intern Med. 1989;4:202–8.

    Article  Google Scholar 

  30. Cydulka RK, Emerman CL, Jouriles NJ. Evaluation of resident performance and intensive bedside teaching during direct observation. Acad Emerg Med. 1996;3:345–51.

    Article  PubMed  CAS  Google Scholar 

  31. Connor MP, Bynoe AG, Redfern N, Pokora J, Clarke J. Developing senior doctors as mentors: a form of continuing professional development. Report of an initiative to develop a network of senior doctors as mentors: 1994–99. Med Educ. 2000;34:747–53.

    Article  PubMed  CAS  Google Scholar 

  32. MacRae H, Regehr G, Leadbetter W, Reznick R. A comprehensive examination for senior surgery residents. Am J Surg. 2000;179:190–3.

    Article  PubMed  CAS  Google Scholar 

  33. Bann S, Datta V, Khan M, Darzi A. The surgical error examination is a novel method for objective technical knowledge assessment. Am J Surg. 2003;185(6):507–11.

    Article  PubMed  Google Scholar 

  34. Boulet JR, Murray D, Kras J, Woodhouse J, McAllister J, Ziv A. Reliability and validity of a simulation-based acute care skills assessment for medical students and residents. Anesthesiology. 2003;99(6):1270–80.

    Article  PubMed  Google Scholar 

  35. Berkenstadt H, Ziv A, Gafni N, Sidi A. Incorporating simulation-based objective structured clinical examination into the Israeli national board examination in anesthesiology. Anesth Analg. 2006;102:853–8.

    Article  PubMed  Google Scholar 

  36. Schubert A, Tetzlaff JE, Licina M, Mascha E, Smith MP. Organization of a comprehensive anesthesiology oral practice examination program: planning, structure, startup, administration, growth and evaluation. J Clin Anesth. 1999;11:504–18.

    Article  PubMed  CAS  Google Scholar 

  37. Jacobsohn E, Klock PA, Avidan M. Poor inter-rater reliability on mock anesthesia oral examinations. Can J Anaesth. 2006;53:659–68.

    Article  PubMed  Google Scholar 

  38. Eagle CJ, Martineau R, Hamilton K. The oral examination in anesthetic resident evaluation. Can J Anaesth. 1993;40:947–53.

    Article  PubMed  CAS  Google Scholar 

  39. Dannefer EF, Henson LC, Bierer SB, Grady-Weliky TA, Meldrum S, Nofziger AC, Barclay C, Epstein RM. Peer assessment of professional competence. Med Educ. 2005;39:713–22.

    Article  PubMed  Google Scholar 

  40. Ramsey PG, Wenrich MD, Carline JD, et al. Use of peer ratings to evaluate physician performance. JAMA. 1993;269:1655–60.

    Article  PubMed  CAS  Google Scholar 

  41. Delphin E, Davidson M. Teaching and evaluating group competency in systems-based practice in anesthesiology. Anesth Analg. 2008;106:1837–43.

    Article  PubMed  Google Scholar 

  42. O’Sullivan PS, Cogbill KK, McClain T, Reckase MD, Clardy JA. Portfolios as a novel approach for residency evaluation. Acad Psychiatry. 2002;26:173–9.

    Article  PubMed  Google Scholar 

  43. Friedman Ben-David M, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ. AMEE medical education guide no. 24: portfolios as a method of student assessment. Med Teach. 2001;23:535–51.

    Article  PubMed  Google Scholar 

  44. O’Sullivan P, Greene C. Portfolios: possibilities for addressing emergency medicine resident competencies. Acad Emerg Med. 2002;9:1305–9.

    Article  PubMed  Google Scholar 

  45. Houben KW, van den Hombergh CLM, Stalmeijer RE, Scherpbier AJ, Marcus MAE. New training strategies for anaesthesia residents. Curr Opin Anaesthesiol. 2011;24:682–6.

    Article  PubMed  Google Scholar 

  46. Driessen EW, van Tartwijk J, Overeem K, Vermunt JD, van der Vleuten CPM. Conditions for successful reflective use of portfolios in undergraduate medical education. Med Educ. 2005;39:1230–5.

    Article  PubMed  Google Scholar 

  47. Parboosingh J. Learning portfolios: potential to assist health professionals with self-directed learning. J Contin Educ Health Prof. 1996;16:75–81.

    Article  Google Scholar 

  48. Davis DA, Mazmanian PE, Fordis M, Harrison RV, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296:1094–102.

    Article  PubMed  CAS  Google Scholar 

  49. Schartel SA, Metro DG. Evaluation: measuring performance, ensuring competence, achieving long-term excellence. Anesthesiology. 2010;112:519–20.

    Article  PubMed  Google Scholar 

  50. Boet S, Bould MD, Bruppacher HR, Desjardins F, Chandra DB, Naik VN. Looking in the mirror: self-debriefing versus instructor debriefing for simulated crises. Crit Care Med. 2011;39:1377–81.

    Article  PubMed  Google Scholar 

  51. Weller JM, Robinson BJ, Jolly B, Watterson LM, Joseph M, Bajenov S, Haughton AJ, Larsen PD. Psychometric characteristics of simulation-based assessment in anaesthesia and accuracy of self-assessed score. Anaesthesia. 2005;60:245–50.

    Article  PubMed  CAS  Google Scholar 

  52. LeBlanc VR. Review article: simulation in anesthesia: state of the science and looking forward. Can J Anaesth. 2012;59:193–202.

    Article  PubMed  Google Scholar 

  53. Mudumbai SC, Gaba DM, Boulet JR, Howard SK, Davies MF. External validation of simulation-based assessment with other performance measures of third-year anesthesiology residents. Simul Healthc. 2012;7:73–80.

    Article  PubMed  Google Scholar 

  54. Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology. 1998;89(1):8–18.

    Article  PubMed  CAS  Google Scholar 

  55. Murray DJ, Boulet JR, Kras JF, Woodhouse JA, Cox T, McAllister JD. Acute care skills in anesthesia practice: a simulation-based resident performance assessment. Anesthesiology. 2004;101(5):1084–95.

    Article  PubMed  Google Scholar 

  56. Schwid HA, Rooke GA, Carline J, Steadman RH, Murray WB, Olympio M, Tarver S, Steckner K, Wetstone S; Anesthesia Simulator Research Consortium. Evaluation of anesthesia residents using mannequin-based simulation: a multi-institutional study. Anesthesiology. 2002;97(6):1434–44.

    Google Scholar 

  57. Helmreich RL, Davies JM. Anaesthetic simulation and lessons to be learned from aviation. Can J Anaesth. 1997;44:907–12.

    Article  PubMed  CAS  Google Scholar 

  58. Murray DJ, Boulet JR, Avidan M, Kras JF, Henrichs D, Woodhouse J. Performance of residents and anesthesiologists in a simulation-based skill assessment. Anesthesiology. 2007;107:705–13.

    Article  PubMed  Google Scholar 

  59. Fehr JJ, Boulet JR, Waldrop WB, Snider R, Brockel M, Murray DJ. Simulation-based assessment of pediatric anesthesia skills. Anesthesiology. 2011;115:1308–15.

    PubMed  Google Scholar 

  60. Boulet JR, Murray D. Review article: assessment in anesthesiology education. Can J Anaesth. 2012;59:182–92.

    Article  PubMed  Google Scholar 

  61. Waldrop WB, Murray DJ, Boulet JR, Kras JF. Management of anesthesia equipment failure: a simulation-based resident skill assessment. Anesth Analg. 2009;109:426–33.

    Article  PubMed  Google Scholar 

  62. Pott LM, Santrock D. Teaching without a teacher: developing competencies with a Bullard laryngoscope using only a structured self-learning course and practicing on a mannequin. J Clin Anesth. 2007;19:583–6.

    Article  PubMed  Google Scholar 

  63. Savoldelli GL, Naik VN, Joo HS, Houston PL, Graham M, Yee B, Hamstra SJ. Evaluation of patient simulator performance as an adjunct to the oral examination for senior anesthesia residents. Anesthesiology. 2006;104:475–81.

    Article  PubMed  Google Scholar 

  64. Gallagher CJ, Tan JM. The current status of simulation in the maintenance of certification. Int Anesthesiol Clin. 2010;48:83–99.

    Article  PubMed  Google Scholar 

  65. Steadman RH, Huang YM. Simulation for quality assurance in training, credentialing and maintenance of certification. Best Pract Res Clin Anaesthesiol. 2012;26:3–15.

    Article  PubMed  Google Scholar 

  66. Kearney RA, Sullivan P, Skakun E. Performance on ABA-ASA in-training examination predicts success for RCPSC certification. Can J Anaesth. 2000;47:914–8.

    Article  PubMed  CAS  Google Scholar 

  67. Berner ES, Brooks CM, Erdmann IV JB. Use of the USMLE to select residents. Acad Med. 1993;68:753–9.

    Article  PubMed  CAS  Google Scholar 

  68. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366:1051–6.

    Article  PubMed  CAS  Google Scholar 

  69. Yeates P, O’Neill P, Mann K, Eva KW. Effect of exposure to good versus poor medical trainee performance on attending physician ratings of subsequent performances. JAMA. 2012;308:2226–32.

    Article  PubMed  CAS  Google Scholar 

  70. Asch DA, Epstein A, Nicholson S. Evaluating medical training programs by the quality of care delivered by their alumni. JAMA. 2007;298:1049–51.

    Article  PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Cleveland Clinic Lerner College of Medicine, Case Western Reserve University, Cleveland, OH, USA

    John E. Tetzlaff M.D.

  2. Department of General Anesthesia, Anesthesiology Institute, Cleveland Clinic, 9500 Euclid Avenue, Cleveland, OH, 44195, USA

    John E. Tetzlaff M.D.

Authors
  1. John E. Tetzlaff M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to John E. Tetzlaff M.D. .

Editor information

Editors and Affiliations

  1. Department of Anesthesiology, Icahn Medical Center at Mount Sinai, New York, New York, USA

    Elizabeth A.M. Frost

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer Science+Business Media New York

About this chapter

Cite this chapter

Tetzlaff, J.E. (2014). Evaluation of Anesthesiology Residents. In: Frost, E. (eds) Comprehensive Guide to Education in Anesthesia. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-8954-2_10

Download citation

  • DOI: https://doi.org/10.1007/978-1-4614-8954-2_10

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4614-8953-5

  • Online ISBN: 978-1-4614-8954-2

  • eBook Packages: MedicineMedicine (R0)

Publish with us

Policies and ethics

Search

Navigation