Advertisement

How Reliable Are Therapeutic Competence Ratings? Results of a Systematic Review and Meta-Analysis

  • Franziska KühneEmail author
  • Ramona Meister
  • Ulrike Maaß
  • Tatjana Paunov
  • Florian Weck
Original Article
First Online:

Abstract

Assessments of psychotherapeutic competencies play a crucial role in research and training. However, research on the reliability and validity of such assessments is sparse. This study aimed to provide an overview of the current evidence and to provide an average interrater reliability (IRR) of psychotherapeutic competence ratings. A systematic review was conducted, and 20 studies reported in 32 publications were collected. These 20 studies were included in a narrative synthesis, and 20 coefficients were entered into the meta-analysis. Most primary studies referred to cognitive-behavioral therapies and the treatment of depression, used the Cognitive Therapy Scale, based ratings on videos, and trained the raters. Our meta-analysis revealed a pooled ICC of 0.82, but at the same time severe heterogeneity. The evidence map highlighted a variety of variables related to competence assessments. Further aspects influencing the reliability of competence ratings and regarding the considerable heterogeneity are discussed in detail throughout the manuscript.

Keywords

Competency Therapist competence Adherence Psychotherapy Assessment 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgements

We would like to thank, Ricarda Löscher B.Sc. psych., for her assistance with study screening and data extraction.

Funding

No external funding.

Compliance with Ethical Standards

Conflict of interest

Florian Weck is an author of six of the publications included in the review. Franziska Kühne, Ramona Meister, Ulrike Maaß and Tatjana Paunov declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies involving human participants.

Research Involving Animal Rights

This article does not contain any studies with animals.

Supplementary material

10608_2019_10056_MOESM1_ESM.pdf (31 kb)
Supplementary material 1 (PDF 30 kb)
10608_2019_10056_MOESM2_ESM.pdf (5 kb)
Supplementary material 2 (PDF 4 kb)

References

References marked with an asterisk indicate studies included in the qualitative summary

  1. American Psychiatric Association (APA). (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington VA: American Psychiatric Association.CrossRefGoogle Scholar
  2. American Psychiatric Association (APA). (2017). What is psychotherapy? Retrieved from https://www.apa.org/ptsd-guideline/patients-and-families/psychotherapy.
  3. * Barber, J. P., & Crits-Christoph, P. (1996). Development of a therapist adherence/competence rating scale for supportive-expressive dynamic psychotherapy: A preliminary report. Psychotherapy Research, 6(2), 81–94.  https://doi.org/10.1080/10503309612331331608.CrossRefGoogle Scholar
  4. * Barber, J. P., Crits-Christoph, P., & Luborsky, L. (1996). Effects of therapist adherence and competence on patient outcome in brief dynamic therapy. Journal of Consulting and Clinical Psychology, 64(3), 619–622.  https://doi.org/10.1037/0022-006x.64.3.619.CrossRefGoogle Scholar
  5. * Barber, J. P., Foltz, C., Crits-Christoph, P., & Chittams, J. (2004). Therapists’ adherence and competence and treatment discrimination in the NIDA Collaborative Cocaine Treatment Study. Journal of Clinical Psychology, 60(1), 29–41.  https://doi.org/10.1002/jclp.10186 CrossRefGoogle Scholar
  6. * Barber, J. P., Krakauer, I., Calvo, N., & Badgio, P. C. (1997). Measuring adherence and competence of dynamic therapists in the treatment of cocaine dependence. Journal of Psychotherapy Practice & Research, 6(1), 12–24.Google Scholar
  7. * Barber, J. P., Liese, B. S., & Abrams, M. J. (2003). Development of the cognitive therapy adherence and competence scale. Psychotherapy Research, 13(2), 205–221.Google Scholar
  8. Barber, J. P., Sharpless, B. A., Klostermann, S., & McCarthy, K. S. (2007). Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature. Professional Psychology: Research and Practice,38, 493–500.  https://doi.org/10.1037/0735-7028.38.5.493.CrossRefGoogle Scholar
  9. Baujat, B., Mahé, C., Pignon, J. P., & Hill, C. (2002). A graphical method for exploring heterogeneity in meta-analyses: Application to a meta-analysis of 65 trials. Statistics in Medicine,21(18), 2641–2652.CrossRefGoogle Scholar
  10. Beck Institute for Cognitive Behavior Therapy. (2019, October 10). Cognitive Therapy Rating Scale (CTRS). Retrieved from https://beckinstitute.org/wp-content/uploads/2017/03/CTRS-Scale-and-Score-Report-2016.pdf.
  11. * Blackburn, I. M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A., & Reichelt, F. K. (2001). The revised cognitive therapy scale (CTS-R): Psychometric properties. Behavioural and cognitive psychotherapy, 29(4), 431–446.Google Scholar
  12. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. Hoboken: Wiley.CrossRefGoogle Scholar
  13. * Brueck, R. K., Frick, K., Loessl, B., Kriston, L., Schondelmaier, S., Go, C.,…, Berner, M. (2009). Psychometric properties of the German version of the motivational interviewing treatment integrity code. Journal of Substance Abuse Treatment, 36(1), 44–48.  https://doi.org/10.1016/j.jsat.2008.04.004.CrossRefGoogle Scholar
  14. * Chevron, E. S., & Rounsaville, B. J. (1983). Evaluating the clinical skills of psychotherapists: A comparison of techniques. Archives of General Psychiatry, 40(10), 1129–1132.  https://doi.org/10.1001/archpsyc.1983.01790090091014.CrossRefGoogle Scholar
  15. * Dennhag, I., Gibbons, M. B. C., Barber, J. P., Gallop, R., & Crits-Christoph, P. (2012a). Do supervisors and independent judges agree on evaluations of therapist adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22(6), 720–730.  https://doi.org/10.1080/10503307.2012.716528.CrossRefGoogle Scholar
  16. * Dennhag, I., Gibbons, M. B. C., Barber, J. P., Gallop, R., & Crits-Christoph, P. (2012b). How many treatment sessions and patients are needed to create a stable score of adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22(4), 475–488.  https://doi.org/10.1080/10503307.2012.674790.CrossRefGoogle Scholar
  17. * Dittmann, C., Müller‐Engelmann, M., Stangier, U., Priebe, K., Fydrich, T., Görg, N., …, Steil, R. (2017). Disorder‐ and treatment‐specific therapeutic competence scales for posttraumatic stress disorder intervention: Development and psychometric properties. Journal of Traumatic Stress.  https://doi.org/10.1002/jts.22236.CrossRefGoogle Scholar
  18. * Dobson, K. S., Shaw, B. F., & Vallis, T. M. (1985). Reliability of a measure of the quality of cognitive therapy. British Journal of Clinical Psychology, 24(4), 295–300.  https://doi.org/10.1111/j.2044-8260.1985.tb00662.x.CrossRefGoogle Scholar
  19. Duffy, L., Gajree, S., Langhorne, P., Stott, D. J., & Quinn, T. J. (2013). Reliability (inter-rater agreement) of the Barthel Index for assessment of stroke survivors: Systematic review and meta-analysis. Stroke,44(2), 462–468.  https://doi.org/10.1161/strokeaha.112.678615.CrossRefPubMedGoogle Scholar
  20. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal Open,315, 629–634.CrossRefGoogle Scholar
  21. Fairburn, C. G., & Cooper, Z. (2011). Therapist competence, therapy quality, and therapist training. Behaviour Research and Therapy,49, 373–378.  https://doi.org/10.1016/j.brat.2011.03.005.CrossRefPubMedPubMedCentralGoogle Scholar
  22. Fisher, Z., & Tipton, E. (2015). robumeta: An R-package for robust variance estimation in meta-analysis. http://arxiv.org/abs/1503.02220.
  23. Hempel, S., Miles, J. N., Booth, M. J., Wang, Z., Morton, S. C., & Shekelle, P. G. (2013). Risk of bias: A simulation study of power to detect study-level moderator effects in meta-analysis. Systematic Reviews,2, 107.  https://doi.org/10.1186/2046-4053-2-107.CrossRefPubMedPubMedCentralGoogle Scholar
  24. Higgins, J., & Green, S. (2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1. 0 [updated March 2011]. The Cochrane Collaboration, 2011. Retrieved August, 29 from www.cochrane-handbook.org.
  25. * Hoffart, A., Sexton, H., Nordahl, H. M., & Stiles, T. C. (2005). Connection between patient and therapist and therapist’s competence in schema-focused therapy of personality problems. Psychotherapy Research, 15(4), 409–419.  https://doi.org/10.1080/10503300500091702.CrossRefGoogle Scholar
  26. * Karterud, S., Pedersen, G., Engen, M., Johansen, M. S., Johansson, P. N., Schlüter, C., …, Bateman, A. W. (2013). The MBT Adherence and Competence Scale (MBT-ACS): Development, structure and reliability. Psychotherapy Research, 23(6), 705–717.  https://doi.org/10.1080/10503307.2012.708795.CrossRefGoogle Scholar
  27. Kazantzis, N. (2003). Therapist competence in cognitive-behavioural therapies: Review of the contemporary empirical evidence. Behaviour Change,20(1), 1–12.  https://doi.org/10.1375/bech.20.1.1.24845.CrossRefGoogle Scholar
  28. * Kazantzis, N., Clayton, X., Cronin, T. J., Farchione, D., Limburg, K., & Dobson, K. S. (2018). The Cognitive Therapy Scale and Cognitive Therapy Scale-Revised as measures of therapist competence in cognitive behavior therapy for depression: Relations with short and long term outcome. Cognitive Therapy and Research, 42(4), 385–397.  https://doi.org/10.1007/s10608-018-9919-4.CrossRefGoogle Scholar
  29. Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine,15(2), 155–163.  https://doi.org/10.1016/j.jcm.2016.02.012.CrossRefPubMedPubMedCentralGoogle Scholar
  30. Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajeweski, B. J., Hróbjartsson, A., et al. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. International Journal of Nursing Studies,48, 661–671.  https://doi.org/10.1016/j.ijnurstu.2011.01.016.CrossRefPubMedGoogle Scholar
  31. Kuyken, W., & Tsivrikos, D. (2009). Therapist competence, comorbidity and cognitive-behavioral therapy for depression. Psychotherapy and Psychosomatics,78(1), 42–48.  https://doi.org/10.1159/000172619.CrossRefPubMedGoogle Scholar
  32. Machmutow, K., Holtforth, M. G., Krieger, T., & Watzke, B. (2018). Identifying relapse prevention elements during psychological treatment of depression: Development of an observer-based rating instrument. Journal of Affective Disorders,227, 358–365.  https://doi.org/10.1016/j.jad.2017.11.009.CrossRefPubMedGoogle Scholar
  33. * McGrath, K. B. (2013). Validation of the Drexel University ACT/tCBT Adherence and Competence Rating Scale: Revised for use in a clinical population. Doctoral dissertation. Retrieved from https://idea.library.drexel.edu/islandora/object/idea%3A3803.
  34. Meister, R., Jansen, A., Härter, M., Nestoriuc, Y., & Kriston, L. (2017). Placebo and nocebo reactions in randomized trials of pharmacological treatments for persistent depressive disorder. A meta-regression analysis. Journal of Affective Disorders,215, 288–298.CrossRefGoogle Scholar
  35. Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med,6(7), e1000097.  https://doi.org/10.1371/journal.pmed.1000097.CrossRefPubMedPubMedCentralGoogle Scholar
  36. Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review,33(3), 484–499.  https://doi.org/10.1016/j.cpr.2013.01.010.CrossRefPubMedGoogle Scholar
  37. Muse, K., & McManus, F. (2016). Expert insight into the assessment of competence in cognitive-behavioural therapy: A qualitative exploration of experts’ experiences, opinions and recommendations. Clinical Psychology and Psychotherapy,23, 246–259.  https://doi.org/10.1002/cpp.1952.CrossRefPubMedGoogle Scholar
  38. Muse, K., McManus, F., Rakovshik, S., & Thwaites, R. (2017). Development and psychometric evaluation of the Assessment of Core CBT Skills (ACCS): An observation-based tool for assessing cognitive behavioral therapy competence. Psychological Assessment,29(5), 542–555.  https://doi.org/10.1037/pas0000372.CrossRefPubMedGoogle Scholar
  39. Portney, L. G., & Watkins, M. P. (2009). Foundations of clinical research: Applications to practice. Upper Saddle River, NJ: Pearson/Prentice Hall.Google Scholar
  40. Quintana, D. S. (2015). From pre-registration to publication: A non-technical primer for conducting a meta-analysis to synthesize correlational data. Frontiers in Psychology,6, 1549.  https://doi.org/10.3389/fpsyg.2015.01549.CrossRefPubMedPubMedCentralGoogle Scholar
  41. R-Core-Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. Retrieved from https://www.R-project.org/.
  42. * Reichelt, F. K., James, I. A., & Blackburn, I. M. (2003). Impact of training on rating competence in cognitive therapy. Journal of Behavior Therapy and Experimental Psychiatry, 34(2), 87–99.Google Scholar
  43. Roth, A. D., & Pilling, S. (2007). The competences required to deliver effective cognitive and behavioural therapy for people with depression and with anxiety disorders. Retrieved from https://www.ucl.ac.uk/pals/research/clinical-ed.
  44. Santelmann, H., Franklin, J., Bußhoff, J., & Baethge, C. (2016). Interrater reliability of schizoaffective disorder compared with schizophrenia, bipolar disorder, and unipolar depression—A systematic review and meta-analysis. Schizophrenia Research,176(2), 357–363.  https://doi.org/10.1016/j.schres.2016.07.012.CrossRefPubMedGoogle Scholar
  45. * Schmidt, I. D., Strunk, D. R., DeRubeis, R. J., Conklin, L. R., & Braun, J. D. (2018). Revisiting how we assess therapist competence in cognitive therapy. Cognitive Therapy and Research, 42(4), 369–384.  https://doi.org/10.1007/s10608-018-9908-7.CrossRefGoogle Scholar
  46. * Strunk, D. R., Brotman, M. A., DeRubeis, R. J., & Hollon, S. D. (2010). Therapist competence in cognitive therapy for depression: Predicting subsequent symptom change. J Consult Clin Psychol, 78(3), 429–437.  https://doi.org/10.1037/a0019631.CrossRefGoogle Scholar
  47. * Soygüt, G., Uluç, S., & Tüzün, Z. (2008). A pilot study of the reliability and validity of the turkish cognitive therapy adherence and competence scale. Turkish Journal of Psychiatry, 19(2).Google Scholar
  48. * Svartberg, M. (1989). Manualization and competence monitoring of short-term anxiety-provoking psychotherapy. Psychotherapy: Theory, Research, Practice, Training, 26(4), 564–571.  https://doi.org/10.1037/h0085477.CrossRefGoogle Scholar
  49. * Tadic, M., Drapeau, M., Solai, S., de Roten, Y., & Despland, J. N. (2003). Development of a competence scale for brief psychodynamic investigation: A pilot study. Schweizer Archiv für Neurologie und Psychiatrie, 154(1), 28–35.  https://doi.org/10.4414/sanp.2003.01333.
  50. Trajković, G., Starčević, V., Latas, M., Leštarević, M., Ille, T., Bukumirić, Z., et al. (2011). Reliability of the Hamilton Rating Scale for depression: A meta-analysis over a period of 49years. Psychiatry Research,189(1), 1–9.  https://doi.org/10.1016/j.psychres.2010.12.007.CrossRefPubMedGoogle Scholar
  51. * Vallis, T. M., Shaw, B. F., & Dobson, K. S. (1986). The Cognitive Therapy Scale: Psychometric properties. Journal of Consulting and Clinical Psychology, 54(3), 381–385.  https://doi.org/10.1037/0022-006x.54.3.381.CrossRefGoogle Scholar
  52. * Vallis, T. M., Shaw, B. F., & McCabe, S. B. (1988). The relationship between therapist competency and cognitive therapy and general therapy skill. Journal of Cognitive Psychotherapy: An International Quarterly, 2(4), 237–249.Google Scholar
  53. Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software,36, 1–48.CrossRefGoogle Scholar
  54. * von Consbruch, K., Clark, D. M., & Stangier, U. (2012). Assessing therapeutic competence in cognitive therapy for social phobia: Psychometric properties of the Cognitive Therapy Competence Scale for Social Phobia (CTCS-SP). Behavioural and Cognitive Psychotherapy, 40(2), 149–161.  https://doi.org/10.1017/s1352465811000622.CrossRefGoogle Scholar
  55. Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology,61(4), 620–630.CrossRefGoogle Scholar
  56. Warshaw, M. G., Dyck, I., Allsworth, J., Stout, R. L., & Keller, M. B. (2001). Maintaining reliability in a long-term psychiatric study: An ongoing inter-rater reliability monitoring program using the longitudinal interval follow-up evaluation. Journal of Psychiatric Research,35(5), 297–305.CrossRefGoogle Scholar
  57. Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology,78, 200–211.  https://doi.org/10.1037/a0018912.CrossRefPubMedPubMedCentralGoogle Scholar
  58. * Weck, F., Bohn, C., Ginzburg, D. M., & Stangier, U. (2011). Assessment of adherence and competence in cognitive therapy: Comparing session segments with entire sessions. Psychotherapy Research, 21(6), 658–669.  https://doi.org/10.1080/10503307.2011.602751.CrossRefGoogle Scholar
  59. * Weck, F., Grikscheit, F., Höfling, V., & Stangier, U. (2014). Assessing treatment integrity in cognitive-behavioral therapy: Comparing session segments with entire sessions. Behavior Therapy, 45(4), 541–552.  https://doi.org/10.1016/j.beth.2014.03.003.CrossRefGoogle Scholar
  60. * Weck, F., Hautzinger, M., Heidenreich, T., & Stangier, U. (2011). Psychoedukation bei depressiven störungen -erfassung von interventionsmerkmalen und behandlungskompetenzen = Psychoeducation for depression - features of interventions and therapeutic competencies. PPmP: Psychotherapie Psychosomatik Medizinische Psychologie, 61(3-4), 148–153.  https://doi.org/10.1055/s-0030-1269902.CrossRefGoogle Scholar
  61. * Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U. (2011). Reliability of adherence and competence assessment in cognitive behavioral therapy: Influence of clinical experience. Journal of Nervous and Mental Disease, 199(4), 276–279.  https://doi.org/10.1097/nmd.0b013e318212461.
  62. * Weck, F., Weigel, M., Richtberg, S., & Stangier, U. (2011). Reliability of adherence and competence assessment in psychoeducational treatment: Influence of clinical experience. Journal of Nervous and Mental Disease, 199(12), 983–986.  https://doi.org/10.1097/nmd.0b013e3182392da1.CrossRefGoogle Scholar
  63. WHO. (1992). The ICD-10 classification of mental and behavioural disorders: Clinical descriptions and diagnostic guidelines (6th ed.). Geneva: World Health Organization.Google Scholar
  64. Wickham, H., Francois, R., Henry, L., & Müller, K. (2015). dplyr: A grammar of data manipulation. R package version 0.4, 3.Google Scholar
  65. Wirtz, M. A. (2017). Interrater Reliability. In V. Zeigler-Hill & T. K. Shackelford (Eds.), Encyclopedia of personality and individual differences (pp. 1–4). New York: Springer.Google Scholar
  66. Wirtz, M., & Caspar, F. (2002). Interrater agreement and interrater reliability. Göttingen: Hogrefe.Google Scholar
  67. * Wittorf, A., Jakobi-Malterre, U. E., Beulen, S., Bechdolf, A., Müller, B. W., Sartory, G., …, Klingberg, S. (2013). Associations between therapy skills and patient experiences of change processes in cognitive behavioral therapy for psychosis. Psychiatry Research, 210(3), 702–709.  https://doi.org/10.1016/j.psychres.2013.08.01.
  68. Young, J. E., & Beck, A. T. (1980). Cognitive Therapy Scale: Rating manual. Unpublished Manuscript, University of Pennsylvania, Philadelphia, PA.Google Scholar
  69. Zarafonitis-Müller, S., Kuhr, K., & Bechdorf, A. (2014). Der Zusammenhang der Therapeutenkompetenz und Adhärenz zum Therapieerfolg in der Kognitiven Verhaltenstherapie - metaanalytische Ergebnisse. Fortschritte der Neurologie-Psychiatrie,82, 502–510.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Psychology, Clinical Psychology and PsychotherapyUniversity of PotsdamPotsdamGermany
  2. 2.Department of Medical PsychologyUniversity Medical Center Hamburg EppendorfHamburgGermany

Personalised recommendations

Cite article

Buy options