A Comparison of Conjoint Analysis with Other Approaches to Model Physician Policies in Scoring Complex Performance-Based Assessment

Summary

Scoring performance assessment requires expert judgments that combine and weight various features of performance to produce a measure. Building models of these policies and mechanically applying them is potentially advantageous where the number of judgments would exhaust the number of available raters. Conjoint analysis is presented as a technique to score computer-based case simulations (CCS) developed by the National Board of Medical Examiners and intended for use in licensure assessment in the United States. To study this technique, four CCSs were selected from those administered to medical students. A four-member clinician committee defined performance attributes (diagnosis, therapy, monitoring, timing) and levels of each; from this, performance profiles were generated for review. Then, each expert individually assigned an anchored rating from 1 (worst) to 9 (best) to each profile; individual rating disagreements on the same examinee performance of more than one were discussed. From this, expert policies were applied to produce examinee case scores.

Correlation of “conjoint scores ” with content experts’ ratings ranged from.81 to.94. This was comparable to previously reported results using other policy modelling techniques; however, this technique was less time consuming than others. This study suggests the potential usefulness of conjoint analysis as a scoring approach for complex performance assessment.