Skip to main content
SpringerLink
Log in

From opening the ‘black box’ to looking behind the curtain: cognition and context in assessor-based judgements

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

The increasing use of direct observation tools to assess routine performance has resulted in the growing reliance on assessor-based judgements in the workplace. However, we have a limited understanding of how assessors make judgements and formulate ratings in real world contexts. The current research on assessor cognition has largely focused on the cognitive domain but the contextual factors are equally important, and both are closely interconnected. This study aimed to explore the perceived cognitive and contextual factors influencing Mini-CEX assessor judgements in the Emergency Department setting. We used a conceptual framework of assessor-based judgement to develop a sequential mixed methods study. We analysed and integrated survey and focus group results to illustrate self-reported cognitive and contextual factors influencing assessor judgements. We used situated cognition theory as a sensitizing lens to explore the interactions between people and their environment. The major factors highlighted through our mixed methods study were: clarity of the assessment, reliance on and variable approach to overall impression (gestalt), role tension especially when giving constructive feedback, prior knowledge of the trainee and case complexity. We identified prevailing tensions between participants (assessors and trainees), interactions (assessment and feedback) and setting. The two practical implications of our research are the need to broaden assessor training to incorporate both cognitive and contextual domains, and the need to develop a more holistic understanding of assessor-based judgements in real world contexts to better inform future research and development in workplace-based assessments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Bandiera, G., & Lendrum, D. (2008). Daily encounter cards facilitate competency-based feedback while leniency bias persists. Canadian Journal of Emergency Medicine, 10(1), 44–50.

    Google Scholar 

  • Berendonk, C., Stalmeijer, R. E., & Schuwirth, L. W. T. (2013). Expertise in performance assessment: Assessors’ perspectives. Advances in Health Sciences Education, 18(4), 559–571.

    Article  Google Scholar 

  • Bredo, E. (1994). Reconstructing educational psychology: Situated cognition and Deweyian pragmatism. Educational Psychology, 29(1), 23–35.

    Article  Google Scholar 

  • Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42.

    Article  Google Scholar 

  • Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G., & Pankratz, V. S. (2009). Effect of rater training on reliability and accuracy of Mini-CEX scores: A randomized, controlled trial. Journal of General Internal Medicine, 24(1), 74–79.

    Article  Google Scholar 

  • Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks: SAGE Publications Inc.

    Google Scholar 

  • De Vaus, D. (2014). Surveys in social research (6th ed.). Abingdon-on-Thames: Routledge.

    Google Scholar 

  • Driessen, E., & Scheele, F. (2013). What is wrong with assessment in postgraduate training? Lessons from clinical practice and educational research. Medical Teacher, 35(7), 569–574.

    Article  Google Scholar 

  • Durning, S. J., & Artino, A. R. (2011). Situativity theory: A perspective on how participants and the environment can interact: AMEE Guide no. 52. Medical Teacher, 33(3), 188–199.

    Article  Google Scholar 

  • Durning, S. J., Artino, A. R., Boulet, J. R., Dorrance, K., van der Vleuten, C., & Schuwirth, L. (2012). The impact of selected contextual factors on experts’ clinical reasoning performance (does context impact clinical reasoning performance in experts?). Advances in Health Sciences Education, 17(1), 65–79.

    Article  Google Scholar 

  • Durning, S. J., Artino, A. R., Jr., Pangaro, L. N., van der Vleuten, C., & Schuwirth, L. (2010). Perspective: Redefining context in the clinical encounter: Implications for research and training in medical education. Academic Medicine, 85(5), 894–901.

    Article  Google Scholar 

  • Durning, S., Artino, A. R., Pangaro, L., van der Vleuten, C., & Schuwirth, L. (2011). Context and clinical reasoning: Understanding the perspective of the expert’s voice. Medical Education, 45(9), 927–938.

    Article  Google Scholar 

  • Eva, K. W. (2018). Cognitive influences on complex performance assessment: Lessons from the interplay between medicine and psychology. Journal of Applied Research in Memory and Cognition, 7(2), 177–188.

    Article  Google Scholar 

  • Gauthier, G., St-Onge, C., & Tavares, W. (2016). Rater cognition: Review and integration of research findings. Medical Education, 50(5), 511–522.

    Article  Google Scholar 

  • Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014a). Seeing the “black box” differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068.

    Article  Google Scholar 

  • Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine, 86(10 Suppl), S1–S7.

    Article  Google Scholar 

  • Gingerich, A., van der Vleuten, C. P. M., Eva, K. W., & Regehr, G. (2014b). More consensus than idiosyncrasy: Categorizing social judgments to examine variability in Mini-CEX ratings. Academic Medicine, 89(11), 1–10.

    Article  Google Scholar 

  • Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780–786.

    Article  Google Scholar 

  • Govaerts, M. J. B., Schuwirth, L. W. T., van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16(2), 151–165.

    Article  Google Scholar 

  • Govaerts, M. J. B., Van de Wiel, M. W. J., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2013). Workplace-based assessment: Raters’ performance theories and constructs. Advances in Health Sciences Education, 18(3), 375–396.

    Article  Google Scholar 

  • Govaerts, M. J. B., van der Vleuten, C. P. M., Schuwirth, L. W. T., & Muijtjens, A. M. M. (2007). Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Advances in Health Sciences Education, 12(2), 239–260.

    Article  Google Scholar 

  • Greenwald, A. G., & Banaji, M. R. (2017). The implicit revolution: Reconceiving the relation between conscious and unconscious. American Psychologist, 72(9), 861–871.

    Article  Google Scholar 

  • Hall, K. H. (2002). Reviewing intuitive decision-making and uncertainty: The implications for medical education. Medical Education, 36(3), 216–224.

    Article  Google Scholar 

  • Hill, Faith, Kathleen, Kendall, Galbraith, Kevin, & Crossley, J. (2009). Implemanting the undergradate Mini-CEX: A tailored approach at Southmpatom University. Medical Education, 43(4), 326–334.

    Article  Google Scholar 

  • Hoffman, K. G., & Donaldson, J. F. (2004). Contextual tensions of the clinical environment and their influence on teaching and learning. Medical Education, 38(4), 448–454.

    Article  Google Scholar 

  • Holmboe, E. S., Hawkins, R. E., & Huot, S. J. (2004). Effects of training in direct observation of medical residents’ clinical competence: A randomized trial. Annals of Internal Medicine, 140(11), 874–881.

    Article  Google Scholar 

  • Huber, V. L., Neale, M. A., & Nofthcraft, G. B. (1987). Judgment by heuristics: Effects of ratee and rater characteristics and performance standards on performance-related judgments. Organizational Behavior and Human Decision Processes, 40(2), 149–169.

    Article  Google Scholar 

  • Kogan, J. R., Conforti, L. N., Bernabeo, E. C., Durning, S. J., Hauer, K. E., & Holmboe, E. S. (2012). Faculty staff perceptions of feedback to residents after direct observation of clinical skills. Medical Education, 46(2), 201–215.

    Article  Google Scholar 

  • Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048–1060.

    Article  Google Scholar 

  • Kogan, J. R., Conforti, L. N., Bernabeo, E., Iobst, W., & Holmboe, E. (2015). How faculty members experience workplace-based assessment rater training: A qualitative study. Medical Education, 49(7), 692–708.

    Article  Google Scholar 

  • Kogan, J. R., Conforti, L. N., Iobst, W. F., & Holmboe, E. S. (2014). Reconceptualizing variable rater assessments as both an educational and clinical care problem. Academic Medicine, 89(5), 721–727.

    Article  Google Scholar 

  • Lavelle, E., Vuk, J., & Barber, C. (2013). Twelve tips for getting started using mixed methods in medical education research. Medical Teacher, 35, 272–276.

    Article  Google Scholar 

  • Lee, V., Brain, K., & Martin, J. (2017). Factors influencing Mini-CEX rater judgments and their practical implications. Academic Medicine, 92(6), 880–887.

    Article  Google Scholar 

  • Maudsley, G. (2011). Mixing it but not mixed-up: Mixed methods research in medical education (a critical narrative review). Medical Teacher, 33(2), e92–e104.

    Article  Google Scholar 

  • Mertens, F., de Groot, E., Meijer, L., Wens, J., Gemma Cherry, M., Deveugele, M., et al. (2017). Workplace learning through collaboration in primary healthcare: A BEME realist review of what works, for whom and in what circumstances: BEME Guide No. 46. Medical Teacher, 40(2), 117–134.

    Article  Google Scholar 

  • Norcini, J. J., Blank, L. L., Arnold, G. K., & Kimball, H. R. (1995). The Mini-CEX (clinical evaluation exercise): A preliminary investigation. Annals of Internal Medicine, 123(10), 795–799.

    Article  Google Scholar 

  • Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The Mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476–481.

    Article  Google Scholar 

  • Pangaro, L., & ten Cate, O. (2013). Frameworks for learner assessment in medicine: AMEE Guide No. 78. Medical Teacher, 35(6), e1197–e1210.

    Article  Google Scholar 

  • Pelgrim, E. A. M., Kramer, A. W. M., Mokkink, H. G. A., & van der Vleuten, C. P. M. (2012). The process of feedback in workplace-based assessment: Organisation, delivery, continuity. Medical Education, 46(6), 604–612.

    Article  Google Scholar 

  • Ramani, S., Könings, K. D., Ginsburg, S., & van der Vleuten, C. P. M. (2018). Twelve tips to promote a feedback culture with a growth mind-set: Swinging the feedback pendulum from recipes to relationships. Medical Teacher, 7, 1–7. https://doi.org/10.1080/0142159X.2018.1432850.

    Article  Google Scholar 

  • Rogausch, A., Beyeler, C., Montagne, S., Jucker-Kupper, P., Berendonk, C., Huwendiek, S., et al. (2015). The influence of students’ prior clinical skills and context characteristics on Mini-CEX scores in clerkships—a multilevel analysis. BMC Medical Education, 15(1), 208.

    Article  Google Scholar 

  • Sagasser, M. H., Kramer, A. W. M., Fluit, C. R. M. G., van Weel, C., & van der Vleuten, C. P. M. (2017). Self-entrustment: How trainees’ self-regulated learning supports participation in the workplace. Advances in Health Sciences Education, 22(4), 931–949.

    Article  Google Scholar 

  • Schifferdecker, K. E., & Reed, V. A. (2009). Using mixed methods research in medical education: Basic guidelines for researchers. Medical Education, 43(7), 637–644.

    Article  Google Scholar 

  • Schuh, L. A., London, Z., Neel, R., Brock, C., Kissela, B. M., Schultz, L., et al. (2009). Education research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination. Neurology, 73(11), 904–908.

    Article  Google Scholar 

  • Schuwirth, L. W. T., & Van der Vleuten, C. P. M. (2011a). Programmatic assessment: From assessment of learning to assessment for learning. Medical Teacher, 33(6), 478–485.

    Article  Google Scholar 

  • Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2011b). General overview of the theories used in assessment: AMEE Guide No. 57. Medical Teacher, 33(10), 783–797.

    Article  Google Scholar 

  • Sheehan, D., Jowsey, T., Parwaiz, M., Birch, M., Seaton, P., Shaw, S., et al. (2017). Clinical learning environments: Place, artefacts and rhythm. Medical Education, 51(10), 1049–1060.

    Article  Google Scholar 

  • Stalmeijer, R. E., Mcnaughton, N., & Van Mook, W. N. K. A. (2014). Using focus groups in medical education research: AMEE Guide No. 91. Medical Teacher, 36(11), 923–939.

    Article  Google Scholar 

  • Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education, 18(2), 291–303.

    Article  Google Scholar 

  • Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.

    Article  Google Scholar 

  • Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292.

    Article  Google Scholar 

  • Winstone, N. E., Nash, R. A., Parker, M., & Rowntree, J. (2017). Supporting learners’ agentic engagement with feedback: A systematic review and a taxonomy of recipience processes. Educational Psychologist, 52(1), 17–37.

    Article  Google Scholar 

  • Wood, T. J. (2013). Mental workload as a tool for understanding dual processes in rater-based assessments. Advances in Health Sciences Education, 18(3), 523–525.

    Article  Google Scholar 

  • Wood, T. J. (2014). Exploring the role of first impressions in rater-based assessments. Advances in Health Sciences Education, 19(3), 409–427.

    Article  Google Scholar 

  • Yeates, P., Cardell, J., Byrne, G., & Eva, K. W. (2015). Relatively speaking: Contrast effects influence assessors’ scores and narrative feedback. Medical Education, 49(9), 909–919.

    Article  Google Scholar 

  • Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2013a). Seeing the same thing differently. Advances in Health Sciences Education, 18(3), 325–341.

    Article  Google Scholar 

  • Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2013b). “You”re certainly relatively competent: Assessor bias due to recent experiences. Medical Education, 47(9), 910–922.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Marjan Govaerts and Andrea Gingerich for their time and expertise in providing feedback regarding our survey and focus group questions. We would also like to thank staff at Monash University Eastern Health Clinical School for their assistance, the Australasian College for Emergency Medicine for permitting us to conduct this research and the Fellows of the College who participated.

Author information

Authors and Affiliations

  1. Department of Emergency Medicine, Austin Health, P.O. Box 5555, Heidelberg, VIC, 3084, Australia

    Victor Lee

  2. Western Hospital, Footscray, VIC, Australia

    Keira Brain

  3. Eastern Health Clinical School, Monash University and Deakin University, Box Hill, VIC, Australia

    Jenepher Martin

Authors
  1. Victor Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Keira Brain
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jenepher Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Victor Lee.

Ethics declarations

Conflict of interest

K Brain and J. Martin declare they have no conflicts of interest. V. Lee was chair of the Workplace-based Assessment working group during the period of this research, but this was a voluntary and unpaid position for the Australasian College for Emergency Medicine. This research was conducted independently without funding or coercion as part of his Master of Health Professional Education thesis at Monash University.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 86 kb)

Supplementary material 2 (DOCX 24 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, V., Brain, K. & Martin, J. From opening the ‘black box’ to looking behind the curtain: cognition and context in assessor-based judgements. Adv in Health Sci Educ 24, 85–102 (2019). https://doi.org/10.1007/s10459-018-9851-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-018-9851-0

Keywords

Search

Navigation