How Reliable Are Therapeutic Competence Ratings? Results of a Systematic Review and Meta-Analysis
Assessments of psychotherapeutic competencies play a crucial role in research and training. However, research on the reliability and validity of such assessments is sparse. This study aimed to provide an overview of the current evidence and to provide an average interrater reliability (IRR) of psychotherapeutic competence ratings. A systematic review was conducted, and 20 studies reported in 32 publications were collected. These 20 studies were included in a narrative synthesis, and 20 coefficients were entered into the meta-analysis. Most primary studies referred to cognitive-behavioral therapies and the treatment of depression, used the Cognitive Therapy Scale, based ratings on videos, and trained the raters. Our meta-analysis revealed a pooled ICC of 0.82, but at the same time severe heterogeneity. The evidence map highlighted a variety of variables related to competence assessments. Further aspects influencing the reliability of competence ratings and regarding the considerable heterogeneity are discussed in detail throughout the manuscript.
KeywordsCompetency Therapist competence Adherence Psychotherapy Assessment
We would like to thank, Ricarda Löscher B.Sc. psych., for her assistance with study screening and data extraction.
No external funding.
Compliance with Ethical Standards
Conflict of interest
Florian Weck is an author of six of the publications included in the review. Franziska Kühne, Ramona Meister, Ulrike Maaß and Tatjana Paunov declare that they have no conflict of interest.
This article does not contain any studies involving human participants.
Research Involving Animal Rights
This article does not contain any studies with animals.
References marked with an asterisk indicate studies included in the qualitative summary
- American Psychiatric Association (APA). (2017). What is psychotherapy? Retrieved from https://www.apa.org/ptsd-guideline/patients-and-families/psychotherapy.
- * Barber, J. P., Krakauer, I., Calvo, N., & Badgio, P. C. (1997). Measuring adherence and competence of dynamic therapists in the treatment of cocaine dependence. Journal of Psychotherapy Practice & Research, 6(1), 12–24.Google Scholar
- * Barber, J. P., Liese, B. S., & Abrams, M. J. (2003). Development of the cognitive therapy adherence and competence scale. Psychotherapy Research, 13(2), 205–221.Google Scholar
- Barber, J. P., Sharpless, B. A., Klostermann, S., & McCarthy, K. S. (2007). Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature. Professional Psychology: Research and Practice,38, 493–500. https://doi.org/10.1037/0735-7028.38.5.493.CrossRefGoogle Scholar
- Beck Institute for Cognitive Behavior Therapy. (2019, October 10). Cognitive Therapy Rating Scale (CTRS). Retrieved from https://beckinstitute.org/wp-content/uploads/2017/03/CTRS-Scale-and-Score-Report-2016.pdf.
- * Blackburn, I. M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A., & Reichelt, F. K. (2001). The revised cognitive therapy scale (CTS-R): Psychometric properties. Behavioural and cognitive psychotherapy, 29(4), 431–446.Google Scholar
- * Brueck, R. K., Frick, K., Loessl, B., Kriston, L., Schondelmaier, S., Go, C.,…, Berner, M. (2009). Psychometric properties of the German version of the motivational interviewing treatment integrity code. Journal of Substance Abuse Treatment, 36(1), 44–48. https://doi.org/10.1016/j.jsat.2008.04.004.CrossRefGoogle Scholar
- * Chevron, E. S., & Rounsaville, B. J. (1983). Evaluating the clinical skills of psychotherapists: A comparison of techniques. Archives of General Psychiatry, 40(10), 1129–1132. https://doi.org/10.1001/archpsyc.1983.01790090091014.CrossRefGoogle Scholar
- * Dennhag, I., Gibbons, M. B. C., Barber, J. P., Gallop, R., & Crits-Christoph, P. (2012a). Do supervisors and independent judges agree on evaluations of therapist adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22(6), 720–730. https://doi.org/10.1080/10503307.2012.716528.CrossRefGoogle Scholar
- * Dennhag, I., Gibbons, M. B. C., Barber, J. P., Gallop, R., & Crits-Christoph, P. (2012b). How many treatment sessions and patients are needed to create a stable score of adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22(4), 475–488. https://doi.org/10.1080/10503307.2012.674790.CrossRefGoogle Scholar
- * Dittmann, C., Müller‐Engelmann, M., Stangier, U., Priebe, K., Fydrich, T., Görg, N., …, Steil, R. (2017). Disorder‐ and treatment‐specific therapeutic competence scales for posttraumatic stress disorder intervention: Development and psychometric properties. Journal of Traumatic Stress. https://doi.org/10.1002/jts.22236.CrossRefGoogle Scholar
- * Dobson, K. S., Shaw, B. F., & Vallis, T. M. (1985). Reliability of a measure of the quality of cognitive therapy. British Journal of Clinical Psychology, 24(4), 295–300. https://doi.org/10.1111/j.2044-8260.1985.tb00662.x.CrossRefGoogle Scholar
- Duffy, L., Gajree, S., Langhorne, P., Stott, D. J., & Quinn, T. J. (2013). Reliability (inter-rater agreement) of the Barthel Index for assessment of stroke survivors: Systematic review and meta-analysis. Stroke,44(2), 462–468. https://doi.org/10.1161/strokeaha.112.678615.CrossRefPubMedGoogle Scholar
- Fisher, Z., & Tipton, E. (2015). robumeta: An R-package for robust variance estimation in meta-analysis. http://arxiv.org/abs/1503.02220.
- Higgins, J., & Green, S. (2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1. 0 [updated March 2011]. The Cochrane Collaboration, 2011. Retrieved August, 29 from www.cochrane-handbook.org.
- * Karterud, S., Pedersen, G., Engen, M., Johansen, M. S., Johansson, P. N., Schlüter, C., …, Bateman, A. W. (2013). The MBT Adherence and Competence Scale (MBT-ACS): Development, structure and reliability. Psychotherapy Research, 23(6), 705–717. https://doi.org/10.1080/10503307.2012.708795.CrossRefGoogle Scholar
- * Kazantzis, N., Clayton, X., Cronin, T. J., Farchione, D., Limburg, K., & Dobson, K. S. (2018). The Cognitive Therapy Scale and Cognitive Therapy Scale-Revised as measures of therapist competence in cognitive behavior therapy for depression: Relations with short and long term outcome. Cognitive Therapy and Research, 42(4), 385–397. https://doi.org/10.1007/s10608-018-9919-4.CrossRefGoogle Scholar
- Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajeweski, B. J., Hróbjartsson, A., et al. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. International Journal of Nursing Studies,48, 661–671. https://doi.org/10.1016/j.ijnurstu.2011.01.016.CrossRefPubMedGoogle Scholar
- Machmutow, K., Holtforth, M. G., Krieger, T., & Watzke, B. (2018). Identifying relapse prevention elements during psychological treatment of depression: Development of an observer-based rating instrument. Journal of Affective Disorders,227, 358–365. https://doi.org/10.1016/j.jad.2017.11.009.CrossRefPubMedGoogle Scholar
- * McGrath, K. B. (2013). Validation of the Drexel University ACT/tCBT Adherence and Competence Rating Scale: Revised for use in a clinical population. Doctoral dissertation. Retrieved from https://idea.library.drexel.edu/islandora/object/idea%3A3803.
- Muse, K., & McManus, F. (2016). Expert insight into the assessment of competence in cognitive-behavioural therapy: A qualitative exploration of experts’ experiences, opinions and recommendations. Clinical Psychology and Psychotherapy,23, 246–259. https://doi.org/10.1002/cpp.1952.CrossRefPubMedGoogle Scholar
- Muse, K., McManus, F., Rakovshik, S., & Thwaites, R. (2017). Development and psychometric evaluation of the Assessment of Core CBT Skills (ACCS): An observation-based tool for assessing cognitive behavioral therapy competence. Psychological Assessment,29(5), 542–555. https://doi.org/10.1037/pas0000372.CrossRefPubMedGoogle Scholar
- Portney, L. G., & Watkins, M. P. (2009). Foundations of clinical research: Applications to practice. Upper Saddle River, NJ: Pearson/Prentice Hall.Google Scholar
- R-Core-Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. Retrieved from https://www.R-project.org/.
- * Reichelt, F. K., James, I. A., & Blackburn, I. M. (2003). Impact of training on rating competence in cognitive therapy. Journal of Behavior Therapy and Experimental Psychiatry, 34(2), 87–99.Google Scholar
- Roth, A. D., & Pilling, S. (2007). The competences required to deliver effective cognitive and behavioural therapy for people with depression and with anxiety disorders. Retrieved from https://www.ucl.ac.uk/pals/research/clinical-ed.
- Santelmann, H., Franklin, J., Bußhoff, J., & Baethge, C. (2016). Interrater reliability of schizoaffective disorder compared with schizophrenia, bipolar disorder, and unipolar depression—A systematic review and meta-analysis. Schizophrenia Research,176(2), 357–363. https://doi.org/10.1016/j.schres.2016.07.012.CrossRefPubMedGoogle Scholar
- * Soygüt, G., Uluç, S., & Tüzün, Z. (2008). A pilot study of the reliability and validity of the turkish cognitive therapy adherence and competence scale. Turkish Journal of Psychiatry, 19(2).Google Scholar
- * Tadic, M., Drapeau, M., Solai, S., de Roten, Y., & Despland, J. N. (2003). Development of a competence scale for brief psychodynamic investigation: A pilot study. Schweizer Archiv für Neurologie und Psychiatrie, 154(1), 28–35. https://doi.org/10.4414/sanp.2003.01333.
- Trajković, G., Starčević, V., Latas, M., Leštarević, M., Ille, T., Bukumirić, Z., et al. (2011). Reliability of the Hamilton Rating Scale for depression: A meta-analysis over a period of 49years. Psychiatry Research,189(1), 1–9. https://doi.org/10.1016/j.psychres.2010.12.007.CrossRefPubMedGoogle Scholar
- * Vallis, T. M., Shaw, B. F., & McCabe, S. B. (1988). The relationship between therapist competency and cognitive therapy and general therapy skill. Journal of Cognitive Psychotherapy: An International Quarterly, 2(4), 237–249.Google Scholar
- * von Consbruch, K., Clark, D. M., & Stangier, U. (2012). Assessing therapeutic competence in cognitive therapy for social phobia: Psychometric properties of the Cognitive Therapy Competence Scale for Social Phobia (CTCS-SP). Behavioural and Cognitive Psychotherapy, 40(2), 149–161. https://doi.org/10.1017/s1352465811000622.CrossRefGoogle Scholar
- Warshaw, M. G., Dyck, I., Allsworth, J., Stout, R. L., & Keller, M. B. (2001). Maintaining reliability in a long-term psychiatric study: An ongoing inter-rater reliability monitoring program using the longitudinal interval follow-up evaluation. Journal of Psychiatric Research,35(5), 297–305.CrossRefGoogle Scholar
- * Weck, F., Hautzinger, M., Heidenreich, T., & Stangier, U. (2011). Psychoedukation bei depressiven störungen -erfassung von interventionsmerkmalen und behandlungskompetenzen = Psychoeducation for depression - features of interventions and therapeutic competencies. PPmP: Psychotherapie Psychosomatik Medizinische Psychologie, 61(3-4), 148–153. https://doi.org/10.1055/s-0030-1269902.CrossRefGoogle Scholar
- * Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U. (2011). Reliability of adherence and competence assessment in cognitive behavioral therapy: Influence of clinical experience. Journal of Nervous and Mental Disease, 199(4), 276–279. https://doi.org/10.1097/nmd.0b013e318212461.
- * Weck, F., Weigel, M., Richtberg, S., & Stangier, U. (2011). Reliability of adherence and competence assessment in psychoeducational treatment: Influence of clinical experience. Journal of Nervous and Mental Disease, 199(12), 983–986. https://doi.org/10.1097/nmd.0b013e3182392da1.CrossRefGoogle Scholar
- WHO. (1992). The ICD-10 classification of mental and behavioural disorders: Clinical descriptions and diagnostic guidelines (6th ed.). Geneva: World Health Organization.Google Scholar
- Wickham, H., Francois, R., Henry, L., & Müller, K. (2015). dplyr: A grammar of data manipulation. R package version 0.4, 3.Google Scholar
- Wirtz, M. A. (2017). Interrater Reliability. In V. Zeigler-Hill & T. K. Shackelford (Eds.), Encyclopedia of personality and individual differences (pp. 1–4). New York: Springer.Google Scholar
- Wirtz, M., & Caspar, F. (2002). Interrater agreement and interrater reliability. Göttingen: Hogrefe.Google Scholar
- * Wittorf, A., Jakobi-Malterre, U. E., Beulen, S., Bechdolf, A., Müller, B. W., Sartory, G., …, Klingberg, S. (2013). Associations between therapy skills and patient experiences of change processes in cognitive behavioral therapy for psychosis. Psychiatry Research, 210(3), 702–709. https://doi.org/10.1016/j.psychres.2013.08.01.
- Young, J. E., & Beck, A. T. (1980). Cognitive Therapy Scale: Rating manual. Unpublished Manuscript, University of Pennsylvania, Philadelphia, PA.Google Scholar