Skip to main content
SpringerLink
Log in

(Mis)Alignment of Medical Education Validation Research with Contemporary Validity Theory: The Mini-CEX as an Example

  • Chapter
  • First Online:
Validity and Validation in Social, Behavioral, and Health Sciences

Part of the book series: Social Indicators Research Series ((SINS,volume 54))

Abstract

The defensibility of inferences and decisions based on any educational or psychological assessment is derived from the alignment of its validation research with contemporary validity theory. Using the Mini-CEX as an example, this study investigated the degree to which medical education assessments are aligned with contemporary validity theory as set out in the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing. American Educational Research Association, Washington, DC, 1999). Results of a systematic review of Mini-CEX validation research indicate that the current body of Mini-CEX validation research is not well aligned with the Standards particularly with respect to identifying the construct being assessed by the Mini-CEX and clarifying the types of inferences that can be drawn from Mini-CEX scores. These findings have implications for researchers, journal editors, medical education programs and governing bodies that set policies or mandate the use of the Mini-CEX.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • American Board of Internal Medicine. (2009). Assessment tools. Retrieved November 20, 2010, from http://www.abim.org/program-directors-administrators/assessment-tools/mini-cex.aspx

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.

    Article  Google Scholar 

  • Cizek, G. J., Rosenberg, S. L., & Koons, H. H. (2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68(3), 397–412.

    Article  Google Scholar 

  • Cizek, G. J., Bowen, D., & Church, K. (2010). Sources of validity evidence for educational and psychological tests: A follow-up study. Educational and Psychological Measurement, 70(5), 732–743.

    Article  Google Scholar 

  • Cohen, S. N., Farrant, P. B. J., & Taibjee, S. M. (2009). Assessing the assessments: U.K. dermatology trainees’ views of the workplace assessment tools. British Journal of Dermatology, 161(1), 34–39.

    Article  Google Scholar 

  • Cronbach, L. J. (1988). Five perspectives on validity argument. In H. Wainer & H. Braun (Eds.), Test validity (pp. 3–17). Hillside: Lawrence Erlbaum.

    Google Scholar 

  • Dewi, S. P., & Achmad, T. H. (2010). Optimising feedback using the mini-CEX during the final semester programme. Medical Education, 44(5), 509–509.

    Article  Google Scholar 

  • Hawkins, R. E., Margolis, M. J., Durning, S. J., & Norcini, J. J. (2010). Constructing a validity argument for the mini-clinical evaluation exercise: A review of the research. Academic Medicine: Journal of the Association of American Medical Colleges, 85(9), 1453–1461.

    Article  Google Scholar 

  • Hill, F., & Kendall, K. (2007). Adopting and adapting the mini-CEX as an undergraduate assessment and learning tool. The Clinical Teacher, 4(4), 244–248.

    Article  Google Scholar 

  • Hill, F., Kendall, K., Galbraith, K., & Crossley, J. (2009). Implementing the undergraduate mini-CEX: A tailored approach at Southampton University. Medical Education, 43(4), 326–334.

    Article  Google Scholar 

  • Hogan, T. P., & Agnello, J. (2004). An empirical study of reporting practices concerning measurement validity. Educational and Psychological Measurement, 64(5), 802–812.

    Article  Google Scholar 

  • Holmboe, E. S., Huot, S., Chung, J., Norcini, J., & Hawkins, R. E. (2003). Construct validity of the MiniClinical evaluation exercise (MiniCEX). Academic Medicine, 78(8), 826–830.

    Article  Google Scholar 

  • Hubley, A. M., & Zumbo, B. D. (1996). A dialectic on validity: Where we have been and where we are going. The Journal of General Psychology, 123, 207–215.

    Article  Google Scholar 

  • Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535.

    Article  Google Scholar 

  • Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319–342.

    Article  Google Scholar 

  • Kane, M. T. (2006). Validation. In R. Brennan (Ed.), Educational measurement (4th ed.). Washington, DC: American Council on Education.

    Google Scholar 

  • Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. doi:10.1111/jedm.12000.

    Article  Google Scholar 

  • Kogan, J. R., Bellini, L. M., & Shea, J. A. (2003). Feasibility, reliability, and validity of the mini-clinical evaluation exercise (mCEX) in a medicine core clerkship. Academic Medicine: Journal of the Association of American Medical Colleges, 78(10), S33–S35.

    Article  Google Scholar 

  • Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. JAMA, the Journal of the American Medical Association, 302(12), 1316–1326.

    Article  Google Scholar 

  • Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048–1060.

    Article  Google Scholar 

  • Kogan, J. R., Conforti, L. N., Bernabeo, E. C., Durning, S. J., Hauer, K. E., & Holmboe, E. S. (2012). Faculty staff perceptions of feedback to residents after direct observation of clinical skills. Medical Education, 46(2), 201–215.

    Article  Google Scholar 

  • Lie, D., Encinas, J., Stephens, F., & Prislin, M. (2010). Do faculty show the ‘halo effect’ in rating students compared with standardized patients during a clinical examination? Internet Journal of Family Practice, 8(2), 1–1.

    Google Scholar 

  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: Macmillan.

    Google Scholar 

  • Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13–23.

    Article  Google Scholar 

  • Miller, M. D., & Linn, R. L. (2000). Validation of performance-based assessments. Applied Psychological Measurement, 24(4), 367.

    Article  Google Scholar 

  • Nair, B. R., Alexander, H. G., McGrath, B. P., Parvathy, M. S., Kilsby, E. C., Wenzel, J., et al. (2008). The mini clinical evaluation exercise (mini-CEX) for assessing clinical performance of international medical graduates. The Medical Journal of Australia, 189(3), 159–161.

    Google Scholar 

  • Ney, E. M., Shea, J. A., & Kogan, J. R. (2009). Predictive validity of the mini-clinical evaluation exercise (mcex): Do medical students’ mCEX ratings correlate with future clinical exam performance? Academic Medicine: Journal of the Association of American Medical Colleges, 84(10), S17–S20.

    Article  Google Scholar 

  • Norcini, J. J., & Blank, L. L. (1995). The mini-CEX (clinical evaluation exercise): A preliminary investigation. Annals of Internal Medicine, 123(10), 795–799.

    Article  Google Scholar 

  • Pelgrim, E. A., Kramer, A. W., Mokkink, H. G., van den Elsen, L., Grol, R. P., & van der Vleuten, C. P. (2010). In-training assessment using direct observation of single-patient encounters: A literature review. Advances in Health Sciences Education. doi:10.1007/s10459-010-9235-6.

    Google Scholar 

  • Reckase, M. D. (1998). The interaction of values and validity assessment: Does a test’s level of validity depend on a researcher’s values? Social Indicators Research, 45(1/3, Validity Theory and the Methods Used in Validation: Perspectives from Social and Behavioral Sciences), 45–54.

    Google Scholar 

  • Sidhu, R. S., Hatala, R., Barron, S., Broudo, M., Pachev, G., & Page, G. (2009). Reliability and acceptance of the mini-clinical evaluation exercise as a performance assessment of practicing physicians. Academic Medicine: Journal of the Association of American Medical Colleges, 84(10), S113–S115.

    Article  Google Scholar 

  • Sireci, S. G. (1998). The construct of content validity. In B. D. Zumbo (Ed.), Validity theory and the methods used in validation: Perspectives from the social and behavioral sciences (pp. 83–117). Amsterdam: Kluwer Academic Press.

    Google Scholar 

  • Sireci, S. G. (2009). Packing and unpacking sources of validity evidence: History repeats itself again. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 19–37). Charlotte: Information Age Publishing.

    Google Scholar 

  • Sireci, S. G., & Parker, P. (2006). Validity on trial: Psychometric and legal conceptualizations of validity. Educational Measurement: Issues and Practice, 25(3), 27–34.

    Article  Google Scholar 

  • Weller, J. M., Jones, A., Merry, A. F., Jolly, B., & Saunders, D. (2009). Investigation of trainee and specialist reactions to the mini-clinical evaluation exercise in anaesthesia: Implications for implementation. British Journal of Anaesthesia, 103(4), 524–530.

    Article  Google Scholar 

  • Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte: Information Age Publishing.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Education, Department of Educational and Counseling Psychology, and Special Education, The University of British Columbia, 2125 Main Mall, Vancouver, BC, V6T 1Z4, Canada

    Debra (Dallie) Sandilands Ph.D.

  2. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counseling Psychology, and Special Education, The University of British Columbia, 2125 Main Mall, Vancouver, BC, V6T 1Z4, Canada

    Bruno D. Zumbo Ph.D.

Authors
  1. Debra (Dallie) Sandilands Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruno D. Zumbo Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Debra (Dallie) Sandilands Ph.D. or Bruno D. Zumbo Ph.D. .

Editor information

Editors and Affiliations

  1. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counseling, Psychology, and Special Education, The University of British Columbia, Vancouver, British Columbia, Canada

    Bruno D. Zumbo

  2. Measurement, Evaluation, and Research Methodology (MERM) Program Department of Educational and Counseling, Psychology, and Special Education, The University of British Columbia, Vancouver, British Columbia, Canada

    Eric K.H. Chan

Appendix: List of Included Studies

Appendix: List of Included Studies

Alves de Lima, A., Henquin, R., Thierer, J., Paulin, J., Lamari, S., Belcastro, F., & Van der Vleuten, C. P. M. (2005). A qualitative study of the impact on learning of the mini clinical evaluation exercise in postgraduate training. Medical Teacher, 27(1), 46–52.

Alves de Lima, A., Barrero, C., Baratta, S., Castillo Costa, Y., Bortman, G., Carabajales, J., Conde, D., Galli, A., Degrange, G., & van der Vleuten, C. (2007). Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training. Medical Teacher, 29(8), 785–790.

Alves de Lima, A. E., Conde, D., Aldunate, L., & van der Vleuten, C. P. (2010). Teachers’ experiences of the role and function of the mini clinical evaluation exercise in post-graduate training. International Journal of Medical Education, 1, 68–73.

Brazil, V., Ratcliffe, L., Zhang, J., & Davin, L. (2012). Mini-CEX as a workplace-based assessment tool for interns in an emergency department – Does cost outweigh value? Medical Teacher, 34(12), 1017–1023. doi:10.3109/0142159X.2012.719653.

Chen, W., Lai, M.-M., Li, T.-C., Chen, P. J., Chan, C.-Y., & Lin, C.-C. (2011). Professional development is enhanced by serving as a mini-CEX preceptor. Journal of Continuing Education in the Health Professions, 31(4), 225–230.

Cohen, S. N., Farrant, P. B. J., & Taibjee, S. M. (2009). Assessing the assessments: UK dermatology trainees’ views of the workplace assessment tools. British Journal of Dermatology, 161(1), 34–39.

Cook, D. A., & Beckman, T. J. (2009). Does scale length matter? A comparison of nine-versus five-point rating scales for the mini-CEX. Advances in Health Sciences Education, 14(5), 655–664.

Cook, D. A., Dupras, D. M., Beckman, T. J., & Thomas, K. G. (2009). Effect of rater training on reliability and accuracy of mini-CEX scores: A randomized, controlled trial. Journal of General Internal Medicine, 24(1), 74–79.

Cook, D. A., Beckman, T. J., Mandrekar, J. N., & Pankratz, V. S. (2010). Internal structure of mini-CEX scores for internal medicine residents: Factor analysis and generalizability. Advances in Health Sciences Education, 15(5), 633–645.

Crossley, J., Johnson, G., Booth, J., & Wade, W. (2011). Good questions, good answers: Construct alignment improves the performance of workplace-based assessment scales. Medical Education, 45(6), 560–569.

Dewi, S. P., & Achmad, T. H. (2010). Optimising feedback using the mini-CEX during the final semester programme. Medical Education, 44(5), 509–509.

Durning, S. J., Cation, L. J., Markert, R. J., & Pangaro, L. N. (2002). Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Academic Medicine, 77(9), 900.

Fernando, N., Cleland, J., McKenzie, H., & Cassar, K. (2008). Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments. Medical Education, 42(1), 89–95.

Hatala, R., Ainslie, M., Kassen, B. O., Mackie, I., & Roberts, J. M. (2006). Assessing the mini-Clinical Evaluation Exercise in comparison to a national specialty examination. Medical Education, 40(10), 950–956.

Hauer, K. E. (2000). Enhancing feedback to students using the Mini-CEX (Clinical Evaluation Exercise). Academic Medicine, 75(5), 524.

Hill, F., & Kendall, K. (2007). Adopting and adapting the mini-CEX as an undergraduate assessment and learning tool. The Clinical Teacher, 4(4), 244–248.

Hill, F., Kendall, K., Galbraith, K., & Crossley, J. (2009). Implementing the undergraduate mini-CEX: A tailored approach at Southampton University. Medical Education, 43(4), 326–334.

Holmboe, E. S., Huot, S., Chung, J., Norcini, J., & Hawkins, R. E. (2003). Construct validity of the miniClinical Evaluation Exercise (miniCEX). Academic Medicine, 78(8), 826.

Holmboe, E. S., Yepes, M., Williams, F., & Huot, S. J. (2004). Feedback and the Mini Clinical Evaluation Exercise. Journal of General Internal Medicine, 19(5p2), 558–561.

Jackson, D., & Wall, D. (2010). An evaluation of the use of the mini-CEX in the foundation programme. British Journal of Hospital Medicine, 71(10), 584–588.

Kogan, J. R., & Hauer, K. E. (2006). Brief report: Use of the Mini-Clinical Evaluation Exercise in internal medicine core clerkships. Journal of General Internal Medicine, 21(5), 501–502.

Kogan, J. R., Bellini, L. M., & Shea, J. A. (2002). Implementation of the mini-CEX to evaluate medical students’ clinical skills. Academic Medicine, 77(11), 1156–1157.

Kogan, J. R., Bellini, L. M., & Shea, J. A. (2003). Feasibility, reliability, and validity of the mini-clinical evaluation exercise (mCEX) in a medicine core clerkship. Academic Medicine, 78(10), S33–S35.

Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048–1060.

Kogan, J. R., Conforti, L. N., Bernabeo, E. C., Durning, S. J., Hauer, K. E., & Holmboe, E. S. (2012). Faculty staff perceptions of feedback to residents after direct observation of clinical skills. Medical Education, 46(2), 201–215.

Lie, D., Encinas, J., Stephens, F., & Prislin, M. (2010). Do faculty show the “halo effect” in rating students compared with standardized patients during a clinical examination. The Internet Journal of Family Practice, 8(2). Retrieved from http://www.ispub.com/journal/the_internet_journal_of_family_practice/volume_8_number_2_20/article/do-faculty-show-the-halo-effect-in-rating-students-compared-with-standardized-patients-during-a-clinical-examination.html

Malhotra, S., Hatala, R., & Courneya, C.-A. (2008). Internal medicine residents’ perceptions of the Mini-Clinical Evaluation Exercise. Medical Teacher, 30(4), 414–419.

Margolis, M. J., Clauser, B. E., Cuddy, M. M., Ciccone, A., Mee, J., Harik, P., & Hawkins, R. E. (2006). Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: A validity study. Academic Medicine, 81(10), S56–S60.

Mitchell, C., Bhat, S., Herbert, A., & Baker, P. (2011). Workplace-based assessments of junior doctors: Do scores predict training difficulties? Medical Education, 45(12), 1190–1198.

Nair, B. R., Alexander, H. G., McGrath, B. P., Parvathy, M. S., Kilsby, E. C., Wenzel, J., Frank, I. B., Pachev, G. S., & Page, G. G. (2008). The mini clinical evaluation exercise (mini-CEX) for assessing clinical performance of international medical graduates. The Medical Journal of Australia, 189(3), 159–161.

Ney, E. M., Shea, J. A., & Kogan, J. R. (2009). Predictive validity of the mini-Clinical Evaluation Exercise (mCEX): Do medical students’ mCEX ratings correlate with future clinical exam performance? Academic Medicine, 84(10), S17–S20.

Norcini, J. J., Blank, L. L., Arnold, G. K., & Kimball, H. R. (1995). The Mini-CEX (Clinical Evaluation Exercise): A preliminary investigation. Annals of Internal Medicine, 123(10), 795–799.

Norcini, J. J., Blank, L. L., Arnold, G. K., & Kimball, H. R. (1997). Examiner differences in the mini-CEX. Advances in Health Sciences Education, 2(1), 27–33.

Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476–481.

Ogunbanjo, G. A. (2009). Adapting mini-CEX scoring to improve inter-rater reliability. Medical Education, 43(5), 484–485.

Quantrill, S. J., & Tun, J. K. (2012). Workplace-based assessment as an educational tool. Guide supplement 31.5-Viewpoint. Medical Teacher, 34(5), 417–418.

Sidhu, R. S., Hatala, R., Barron, S., Broudo, M., Pachev, G., & Page, G. (2009). Reliability and acceptance of the Mini-Clinical Evaluation Exercise as a performance assessment of practicing physicians. Academic Medicine, 84(10), S113–S115.

Torre, D. M., Simpson, D. E., Elnicki, D. M., Sebastian, J. L., & Holmboe, E. S. (2007). Feasibility, reliability and user satisfaction with a PDA-based mini-CEX to evaluate the clinical skills of third-year medical students. Teaching and Learning in Medicine, 19(3), 271–277.

Van Lohuizen, M. T., Kuks, J. B., van Hell, E. A., Raat, A. N., Stewart, R. E., & Cohen-Schotanus, J. (2010). The reliability of in-training assessment when performance improvement is taken into account. Advances in Health Sciences Education, 15(5), 659–669.

Weller, J. M., Jolly, B., Misur, M. P., Merry, A. F., Jones, A., Crossley, J. M., Pedersen, K., & Smith, K. (2009a). Mini-clinical evaluation exercise in anaesthesia training. British Journal of Anaesthesia, 102(5), 633–641.

Weller, J. M., Jones, A., Merry, A. F., Jolly, B., & Saunders, D. (2009b). Investigation of trainee and specialist reactions to the mini-Clinical Evaluation Exercise in anaesthesia: Implications for implementation. British Journal of Anaesthesia, 103(4), 524–530.

Wiles, C. M., Dawson, K., Hughes, T. A. T., Llewelyn, J. G., Morris, H. R., Pickersgill, T. P., Robertson, N. P., & Smith, P. E. M. (2007). Clinical skills evaluation of trainees in a neurology department. Clinical Medicine, 7(4), 365–369.

Wilkinson, J. R., Crossley, J. G., Wragg, A., Mills, P., Cowan, G., & Wade, W. (2008). Implementing workplace-based assessment across the medical specialties in the United Kingdom. Medical Education, 42(4), 364–373.

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Sandilands, D.(., Zumbo, B.D. (2014). (Mis)Alignment of Medical Education Validation Research with Contemporary Validity Theory: The Mini-CEX as an Example. In: Zumbo, B., Chan, E. (eds) Validity and Validation in Social, Behavioral, and Health Sciences. Social Indicators Research Series, vol 54. Springer, Cham. https://doi.org/10.1007/978-3-319-07794-9_17

Download citation

Publish with us

Policies and ethics

Search

Navigation