A Comparison of Diagnostic Pattern Recognition Skills to Performance on a “High Stakes” OSCE

Summary

The use of the diagnostic pattern recognition test (DPRT) for assessing diagnostic competency has been increasing over the last few years. The main advantages of this format are ease of creation, generation of reliable scores, and assessment of problem-solving (diagnostic) skills. While the use of this format is compelling, the evidence of its validity required further investigation. This study examined whether a 20-item DPRT could predict physicians’ diagnostic behaviour as measured in an OSCE. Twenty-five candidates wrote the Canadian Academy of Sport Medicine (CASM) Diploma Examination. At one rest station, the candidates wrote a twenty-item DPRT. The DPRT was divided into two subtests of 10 items each: diagnosis and treatment. Each of the pencil and paper case vignettes corresponded to an OSCE station and the answer to a specific behaviour on each station checklist. Reliability estimates for the DPRT (diagnosis was 0.11 and treatment was 0.54). OSCE reliability estimates varied from scores were 0.83 – 0.95 depending on the way the estimates were calculated. The analyses of the data showed that the scores obtained on the DPRT have some relationship to those specific behaviours assessed by the checklist but not to the corresponding behaviour (diagnosis to diagnosis). Although efforts were made to standards the content between the DPRT and OSCE, it appears that the knowledge assessed by the DPRT and OSCE are different.