How biased are you? The effect of prior performance information on attending physician ratings and implications for learner handover

Abstract

Learner handover (LH), the process of sharing of information about learners between faculty supervisors, allows for longitudinal assessment fundamental in the competency-based education model. However, the potential to bias future assessments has been raised as a concern. The purpose of this study is to determine whether prior performance information such as LH influences the assessment of learners in the clinical context. Between December 2017 and June 2018, forty-two faculty members and final-year residents from the Department of Medicine at the University of Ottawa were assigned to one of three study groups through quasi-randomisation, taking into account gender, speciality and rater experience. In a counter-balanced design, each group received either positive, negative or no LH prior to watching six simulated learner–patient encounter videos. Participants rated each video using the mini-CEX and completed a questionnaire on the raters’ general impressions of LH. A significant difference in the mean mini-CEX competency scale scores between the negative (M = 5.29) and positive (M = 5.97) LH groups (P < .001, d = 0.81) was noted. Similar findings were found for the single overall clinical competence ratings. In the post-study questionnaire, 22/28 (78%) of participants had correctly deduced the purpose of the study and 14/28 (50%) felt LH did not influence their assessment. LH influenced mini-CEX scores despite raters’ awareness of the potential for bias. These results suggest that LH could influence a rater’s performance assessment and careful consideration of the potential implications of LH is required.

This is a preview of subscription content, log in to check access.

References

  1. Arts, J. A. R., Gijselaers, W. H., & Boshuizen, H. P. A. (2006). Understanding managerial problem-seolving, knowledge use and information processing: Investigating stages from school to the workplace. Contemporary Educational Psychology,31(4), 387–410.

    Google Scholar 

  2. Baltes, B. B., & Parker, C. P. (2000). Reducing the effects of performance expectations on behavioral ratings. Organizational Behavior and Human Decision Processes,82(2), 237–267.

    Google Scholar 

  3. Busing, N., Rosenfield, J., Rungta, K., Raegele, M., Warren, A., Wright, B., et al. (2018). Smoothing the transition points in Canadian medical education. Academic Medicine,93(5), 715–721.

    Google Scholar 

  4. Chong, L., Taylor, S., Haywood, M., Adelstein, B.-A., & Shulruf, B. (2018). Examiner seniority and experience are associated with bias when scoring communication, but not examination, skills in objective structured clinical examinations in Australia. Journal of Education Evaluation for Health Professionals,15, 17.

    Google Scholar 

  5. Cleary, L. (2008). “Forward feeding” about students’ progress: The case for longitudinal, progressive, and shared assessment of medical students. Academic Medicine,83(9), 800.

    Google Scholar 

  6. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York: Psychology Press.

    Google Scholar 

  7. Cohen, G. S., & Blumberg, P. (1991). Investigating whether teachers should be given assessments of students made by previous teachers. Academic Medicine,66(5), 288–289.

    Google Scholar 

  8. Cox, S. M. (2008). “Forward feeding” about students’ progress: Information on struggling medical students should not be shared among clerkship directors or with students’ current teachers. Academic Medicine,83(9), 801.

    Google Scholar 

  9. Dory, D., Young, M., Danoff, D., Plotnick, L., Pal, N., Gumuchian, S., et al., (2019). Learner handover: Does providing assessors with information about learners’ weaknesses influence assessment scores? Presented at the Annual Meeting of the Canadian Conference on Medical Education (CCME), Niagara Falls, Ontario.

  10. Dudek, N. L., Marks, M. B., & Regehr, G. (2005). Failure to fail: The perspectives of clinical supervisors. Academic Medicine,80(10), S84–S87.

    Google Scholar 

  11. Earl-Slater, A. (2002). The handbook of clinical trials and other research. Oxon: Radcliffe Medical Press Ltd.

    Google Scholar 

  12. e-Laws. (2019). Freedom of Information and Protection of Privacy Act, R.S.O. 1990, c.F.31. Retrieved from https://www.ontario.ca/laws/statute/90f31.

  13. Epstein, R. M. (2007). Assessment in medical education. New England Journal of Medicine,356(4), 387–396.

    Google Scholar 

  14. Foti, R. J., & Hauenstein, N. M. A. (1993). Processing demands and the effects of prior impressions on subsequent judgments: Clarifying the assimilation/contrast debate. Organizational Behavior and Human Decision Processes,56(2), 167–189.

    Google Scholar 

  15. Frellsen, S. L., Baker, E. A., Papp, K. K., & Durning, S. J. (2008). Medical school policies regarding strugging medical students during the internal medicine clerkships: Results of a national survey. Academic Medicine,83(9), 876–881.

    Google Scholar 

  16. Gauthier, G., St-Onge, C., & Tavares, W. (2016). Rater cognition: Review and integration of research findings. Medical Education,50, 511–522.

    Google Scholar 

  17. Gold, W. L., McArdle, P., & Federman, D. D. (2002). Should medical school faculty see assessments of students made by previous teachers? Academic Medicine,77(11), 1096–1100.

    Google Scholar 

  18. Govaerts, M. J. B., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education Theory and Practice,16(2), 151–165.

    Google Scholar 

  19. Hanges, P. J., Braverman, E., & Rentsch, J. R. (1991). Changes in Raters’ perception of subordinates: A catastrophe model. Journal of Applied Psychology,76(6), 878–888.

    Google Scholar 

  20. Hatala, R., Ainslie, M., Kassen, B. O., Mackie, I., & Roberts, J. M. (2006). Assessing the mini-clinical evaluation exercise in comparison to a national specialty examination. Medical Education,40, 950–956.

    Google Scholar 

  21. Holmboe, E., Ginsburg, S., & Bernabeo, E. (2011). The rotational approach to medical education: Time to confront our assumptions? Medical Education,45(1), 69–80.

    Google Scholar 

  22. Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., & Frank, J. R. (2010). The role of assessment in competency-based medical education. Medical Teacher,32(8), 676–682.

    Google Scholar 

  23. Huber, V. L. (1989). Comparison of the effects of specific and general performance standards on performance appraisal decisions. Decision Sciences,20(3), 545–557.

    Google Scholar 

  24. Humphrey-Murto, S., Leblanc, A., Touchie, C., Pugh, D., Wood, T. J., Cowley, L., et al. (2019). The influence of prior performance information on ratings of present performance: Implications for learner handover: A scoping review. Academic Medicine,94(7), 1050–1057.

    Google Scholar 

  25. Ioannidis, J. P. A. (2018). The proposal to lower P value thresholds to.005. Journal of the American Medical Association,319(14), 1429–1430.

    Google Scholar 

  26. Jones, K. T., Hunt, S. C., & Chen, C. C. (2008). Auditors’ performance evaluations: An experimental analysis of the effects of initial impressions and task-specific experience on information later recalled. Accounting Forum,32(3), 213–224.

    Google Scholar 

  27. Kassam, A., Ruetalo, M., Topps, M., et al. (2019). Key stakeholder opinions for a national learner education handover. BMC Medical Education,19, 150.

    Google Scholar 

  28. Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. Journal of the American Medical Association,302(12), 1316–1326.

    Google Scholar 

  29. Kravitz, D. A., & Balzer, W. K. (1992). Context effects in performance appraisal: A methodological critique and empirical study. Journal of Applied Psychology,77(1), 24–31.

    Google Scholar 

  30. Lockyer, J., Carraccio, C., Chang, M.-K., Hart, D., Smee, S., Touchie, C., et al. (2017). Core principles of assessment in competency-based medical education. Medical Teacher,39(6), 609–616.

    Google Scholar 

  31. Mims, L. D., DeCastro, A. O., & Kelly, A. G. (2017). Perspectives of family medicine clerkship directors regarding forward feeding: A CERA study. Family Medicine,49(9), 699–705.

    Google Scholar 

  32. Murphy, K. R., Balzer, W. K., Lockhart, M. C., & Eisenman, E. J. (1985). Effects of previous performance on evaluations of present performance. Journal of Applied Psychology,70(1), 72–84.

    Google Scholar 

  33. Murphy, K. R., Gannett, B. A., Herr, B. M., & Chen, J. A. (1986). Effects of subsequent performance on evaluations of previous performance. Journal of Applied Psychology,71(3), 427–431.

    Google Scholar 

  34. Nickell, L., Mountjoy, M., Bandiera, G., Topps, M., Maggi, J., Forgie, M., et al., (2018, April). Learner Education Handover: Easing the transition from undergraduate to postgraduate medicine. Symposium presented at the Canadian Conference on Medical Education, Halifax, Nova Scotia, Canada.

  35. Nieminen, L. R. G., Rudolph, C. W., Baltes, B. B., Casper, C. M., Wynne, K. T., & Kirby, L. C. (2013). The combined effect of Ratee’s bodyweight and past performance information on performance judgments. Journal of Applied Social Psychology,43, 527–543.

    Google Scholar 

  36. Norcini, J., Anderson, B., Bollela, V., Burch, V., Costa, M. J., Duvivier, R., et al. (2011). Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference. Medical Teacher,33(3), 206–214.

    Google Scholar 

  37. Reilly, S. P., Smither, J. W., Warech, M. A., & Reilly, R. R. (1998). The influence of indirect knowledge of previous performance on ratings of present performance: The effects of job familiarity and rater training. Journal of Business and Psychology,12(4), 421–435.

    Google Scholar 

  38. Royal College of Physicians and Surgeons of Canada. (2018). Guidelines for Educational Handover in Competence by Design (CBD). Retrieved from http://www.royalcollege.ca/rcsite/documents/cbd/guidelines-for-educational-handover-in-competence-by-design-e.pdf

  39. Salvemini, N. J., Reilly, R. R., & Smither, J. W. (1993). The influence of rater motivation on assimilation effects and accuracy in performance ratings. Organizational Behavior and Human Decision Processes,55(1), 41–60.

    Google Scholar 

  40. Smither, J. W., Reilly, R. R., & Buda, R. (1988). Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited. Journal of Applied Psychology,73(3), 487–496.

    Google Scholar 

  41. Sozener, C. B., Lypson, M. L., House, J. B., Hopson, L. R., Dooley-Hash, S. L., Hauff, S., et al. (2016). Reporting achievement of medical student milestones to residency program directors: An educational handover. Academic Medicine,91(5), 676–684.

    Google Scholar 

  42. Stroud, L., Herold, J., Tomlinson, G., & Cavalcanti, R. B. (2011). Who you know of what you know? Effect of examiner familiarity with residents on OSCE scores. Academic Medicine,86(10), S8–S11.

    Google Scholar 

  43. Sullivan, G. M., & Feinn, R. (2012). Using effect size—or why the P value is not enough. Journal of Graduate Medical Education,4(3), 279–282.

    Google Scholar 

  44. ten Cate, O., & Regehr, G. (2019). The power of subjectivity in the assessment of medical trainees (2018). Academic Medicine,94(3), 333–337.

    Google Scholar 

  45. U.S Department of Education. (2018). Family Educational Rights and Privacy Act (FERPA). Retrieved from https://www2.ed.gov/policy/gen/guid/fpco/ferpa/index.html.

  46. van der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005). Assessing professional competence: From methods to programmes. Medical Education,39(3), 309–317.

    Google Scholar 

  47. van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K. J., et al. (2012). A model for programmatic assessment fit for purpose. Medical Teacher,34(3), 205–214.

    Google Scholar 

  48. Wancata, L. M., Morgan, H., Sandhu, G., Santen, S., & Hughes, D. T. (2017). Using the ACMGE milestones as a handover tool from medical school to surgery residency. Journal of Surgical Education,74(3), 519–529.

    Google Scholar 

  49. Warm, E. J., Englander, R., Pereira, A., & Barach, P. (2017). Improving learner handovers in medical education. Academic Medicine,92(7), 927–931.

    Google Scholar 

  50. Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2013). ‘You’re certainly relatively competent’: Assessor bias due to recent experiences. Medical Education,47(9), 910–922.

    Google Scholar 

Download references

Acknowledgements

The authors thank Lesley Ananny, Katherine Scowcroft and the Research Support Unit at the Department of Innovation in Medical Education for their ongoing support. The authors also thank the Department of Medicine at the University of Ottawa for their participation in this project.

Funding

The authors thank the Department of Medicine at the University of Ottawa, the Canadian Association of Medical Education (CAME) and the Association of Medical Education in Europe for their financial support through the Wooster Family grant and research grant programme respectively.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Tammy Shaw.

Ethics declarations

Conflict of interest

None reported.

Ethical approval

The authors obtained ethical approval from the Ottawa Health Network Research Ethics Board (OHSN-REB).

Previous presentations

The authors have presented the results of this study at the University of Ottawa’s 2019 Meridith Marks Day local conference in Ottawa March 29, 2019, the 2019 Canadian Association of Medical Education (CAME) annual conference in Niagara Falls, Ontario April 14, 2019 and will present at the 2019 Association of Medical Education in Europe annual conference in Vienna, Austria August 27, 2019. The project is also Dr. Shaw’s M.Ed. thesis project at the University of Dundee (manuscript in progress).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1: Study design

See Table 5.

Table 5 Study design

Appendix 2: Mini-CEX form

figurea

Appendix 3: learner–patient clinical encounter videos

See Table 6.

Table 6 Description of medical issues, patients and learners used in learner–patient clinical encounter videos

Appendix 4: Learner handover form—example

The Program Director is now forwarding educational information about each learner based on previous evaluations (ITERs, written examinations, OSCEs) over the last 12 months. Here is the information about this learner for you to reflect upon.

figureb
figurec

Appendix 5: Post-video questionnaire—example of questions for groups A&B

  1. 1.

    What did you think was the purpose of this study?

  2. 2.

    Did you recognize any of the learners in the videos? If yes, which ones? If no, skip Questions 3&4.

  3. 3.

    On a scale of 1 (poor) to 10 (excellent), what was your impression of this (these) learner(s) prior to this session?

  4. 4.

    Did your former impression of this (these) learner(s) impact your assessment of the current performance?

You were provided with information about the learner. Please answer the following questions. Please elaborate on your answers.

  1. 5.

    How did you consider the learner handover/forward feeding information as you watched the video?

  2. 6.

    What information, if any, provided to you in the learner handover/forward feeding information was relevant to you in making assessment judgments? If so, in which instance? If varied with each video, please comment.

  3. 7.

    Did you feel the information provided was credible? Why or why not?

  4. 8.

    How do you think the learner handover/forward feeding information impacted your assessment?

  5. 9.

    What is your general opinion regarding learner handover?

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Shaw, T., Wood, T.J., Touchie, C. et al. How biased are you? The effect of prior performance information on attending physician ratings and implications for learner handover. Adv in Health Sci Educ (2020). https://doi.org/10.1007/s10459-020-09979-6

Download citation

Keywords

  • Assessment
  • Bias
  • Competency-based medical education
  • Learner handover
  • Transitions