Cognitive Therapy and Research

, Volume 43, Issue 3, pp 642–655 | Cite as

The Factor Structure of the Cognitive Therapy Rating Scale (CTRS) in a Sample of Community Mental Health Clinicians

  • Nicholas W. Affrunti
  • Torrey A. CreedEmail author
Original Article


Treatment fidelity is an essential outcome of implementation research. The gold standard measure for Cognitive-Behavioral Therapy (CBT) fidelity is the Cognitive Therapy Rating Scale (CTRS). Despite its widespread use in research and training programs, the structure of the CTRS has not been examined in a sample of community mental health clinicians with adult and child clients. The current study addressed this gap. The sample consisted of 355 clinicians and 1298 CBT sessions scored using the CTRS. High interrater reliability was observed and factor analysis yielded separate structures for child and adult treatment sessions. These structures were not consistent with previous factor analyses conducted on the scale. Findings suggest that the CTRS is a reliable measure of CBT in community mental health settings but that its structure may depend on the clinical population measured. Additionally, the factor structure can provide guidance for delivering feedback in training and supervision settings.


Treatment fidelity Cognitive Therapy Rating Scale Community mental health Factor analysis Psychometrics 


Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Research Involving Animal Rights

This article does not contain any studies with animals performed by any of the authors.

Informed Consent

Informed consent was obtained from all individual participants included in the study.


  1. Aarons, G. A., Miller, E. A., Green, A. E., Perrott, J. A., & Bradway, R. (2012). Adaptation happens: A qualitative case study of implementation of the incredible years evidence-based parent training programme in a residential substance abuse treatment programme. Journal of Children’s Services, 7, 233–245.Google Scholar
  2. Arbuckle, J. L. (2012). Amos (Version 21.0) [Computer Program]. Chicago: IBM SPSS.Google Scholar
  3. Beavers, A. S., Lounsbury, J. W., Richards, J. K., Huck, S. W., Skolits, G. J., & Esquivel, S. L. (2013). Practical considerations for using exploratory factor analysis in educational research. Practical Assessment, Research & Evaluation, 18, 1–13.Google Scholar
  4. Beck, A. T., & Haigh, E. A. (2014). Advances in cognitive theory and therapy: The generic cognitive model. Annual Review of Clinical Psychology, 10, 1–24.Google Scholar
  5. Borkovec, T. D., Newman, M. G., Pincus, A. L., & Lytle, R. (2002). A component analysis of cognitive-behavioral therapy for generalized anxiety disorder and the role of interpersonal problems. Journal of Consulting and Clinical Psychology, 70, 288–298. Scholar
  6. Branson, A., Shafran, R., & Myles, P. (2015). Investigating the relationship between competence and patient outcome with CBT. Behaviour research and therapy, 68, 19–26.Google Scholar
  7. Byrne, B. M. (2004). Testing for multigroup invariance using AMOS graphics: A road less traveled. Structural Equation Modeling, 11, 272–300.Google Scholar
  8. Chavira, D. A., Golinelli, D., Sherbourne, C., Stein, M. B., Sullivan, G., Bystritsky, A., … Bumgardner, K. (2014). Treatment engagement and response to CBT among Latinos with anxiety disorders in primary care. Journal of Consulting and Clinical Psychology, 82, 392–403.Google Scholar
  9. Chu, B. C., & Kendall, P. C. (2009). Therapist responsiveness to child engagement: flexibility within manual-based CBT for anxious youth. Journal of Clinical Psychology, 65, 736–754.Google Scholar
  10. Cook, J. M., Dinnen, S., Thompson, R., Simiola, V., & Schnurr, P. P. (2014). Changes in implementation of two evidence-based psychotherapies for PTSD in VA residential treatment programs: A national investigation. Journal of Traumatic Stress, 27, 137–143.Google Scholar
  11. Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10, 1–9.Google Scholar
  12. Creed, T. A., Frankel, S. A., German, R. E., Green, K. L., Jager-Hyman, S., Taylor, K. P., … Williston, M. A. (2016a). Implementation of transdiagnostic cognitive therapy in community behavioral health: The Beck Community Initiative. Journal of Consulting and Clinical Psychology, 84, 1116–1126.Google Scholar
  13. Creed, T. A., Jager-Hyman, S., Pontoski, K., Feinberg, B., Rosenberg, Z., Evans, A. C., Hurford, M. O., & Beck, A. T. (2013). The Beck Initiative: A strength-based approach to training school-based mental health staff in cognitive therapy. International Journal of Emotional Education, 5, 49–66.Google Scholar
  14. Creed, T. A., Stirman, S. W., Evans, A. C., & Beck, A. T. (2014). A model for implementation of cognitive therapy in community mental health: The Beck Initiative. Behavior Therapist, 37, 56–64.Google Scholar
  15. Creed, T. A., Wolk, C. B., Feinberg, B., Evans, A. C., & Beck, A. T. (2016b). Beyond the label: Relationship between community therapists’ self-report of a cognitive behavioral therapy orientation and observed skills. Administration and Policy in Mental Health and Mental Health Services Research, 43, 36–43. Scholar
  16. Dimidjian, S., Hollon, S. D., Dobson, K. S., Schmaling, K. B., Kohlenberg, R. J., Addis, M. E., … Atkins, D. C. (2006). Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of Consulting and Clinical Psychology, 74, 658–670.Google Scholar
  17. Dobson, K. S., Shaw, B. F., & Vallis, T. M. (1985). Reliability of a measure of the quality of cognitive therapy. British Journal of Clinical Psychology, 24, 295–300.Google Scholar
  18. Essock, S. M., Nossel, I. R., McNamara, K., Bennett, M. E., Buchanan, R. W., Kreyenbuhl, J. A., … Dixon, L. B. (2015). Practical monitoring of treatment fidelity: examples from a team-based intervention for people with early psychosis. Psychiatric Services, 66, 674–676.Google Scholar
  19. Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.Google Scholar
  20. Forand, N. R., Evans, S., Haglin, D., & Fishman, B. (2011). Cognitive behavioral therapy in practice: Treatment delivered by trainees at an outpatient clinic is clinically effective. Behavior Therapy, 42, 612–623.Google Scholar
  21. Fuggle, P., Dunsmuir, S., & Curry, V. (2012). CBT with children, young people and families. London: Sage.Google Scholar
  22. Garety, P. A., Craig, T. K., Iredale, C. H., Basit, N., Fornells-Ambrojo, M., Halkoree, R., … Zala, D. (2017). Training the frontline workforce to deliver evidence-based therapy to people with psychosis: Challenges in the GOALS study. Psychiatric Services, 69, 9–11.Google Scholar
  23. Gaudiano, B. A. (2008). Cognitive-behavioral therapies: Achievements and challenges. Evidence Based Mental Health, 11, 5–7. Scholar
  24. German, R. E., Adler, A., Frankel, S. A., Stirman, S. W., Pinedo, P., Evans, A. C., … Creed, T. A. (2017). Testing a web-based, trained-peer model to build capacity for evidence-based practices in community mental health systems. Psychiatric Services, 69, 286–292.Google Scholar
  25. Ginsburg, G. S., Becker, K. D., Drazdowski, T. K., & Tein, J. Y. (2012). Treating anxiety disorders in inner city schools: Results from a pilot randomized controlled trial comparing CBT and usual care. Child & Youth Care Forum, 41, 1–19.Google Scholar
  26. Hamilton, J. D., Kendall, P. C., Gosch, E., Furr, J. M., & Sood, E. (2008). Flexibility within fidelity. Journal of the American Academy of Child & Adolescent Psychiatry, 47, 987–993.Google Scholar
  27. Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30, 448–466.Google Scholar
  28. Hofmann, S. G., Asmundson, G. J., & Beck, A. T. (2013). The science of cognitive therapy. Behavior Therapy, 44, 199–212.Google Scholar
  29. Hofmann, S. G., Asnaani, A., Vonk, I. J., Sawyer, A. T., & Fang, A. (2012). The efficacy of cognitive behavioral therapy: A review of meta-analyses. Cognitive Therapy and Research, 36, 427–440.Google Scholar
  30. Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555.Google Scholar
  31. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55.Google Scholar
  32. Huang, F. L. (2016). Alternatives to multilevel modeling for the analysis of clustered data. The Journal of Experimental Education, 84, 175–196.Google Scholar
  33. Jacobson, N. S., & Gortner, E. T. (2000). Can depression be de-medicalized in the 21st century: Scientific revolutions, counter-revolutions and the magnetic field of normal science. Behaviour Research and Therapy, 38, 103–117.Google Scholar
  34. James, A. C., James, G., Cowdrey, F. A., Soler, A., & Choke, A. (2013). Cognitive behavioural therapy for anxiety disorders in children and adolescents. Cochrane Database of Systematic Reviews, 2013(6), CD004790. Scholar
  35. Kazantzis, N., Clayton, X., Cronin, T. J., Farchione, D., Limburg, K., & Dobson, K. S. (2018). The Cognitive Therapy Scale and Cognitive Therapy Scale-Revised as measures of therapist competence in cognitive behavior therapy for depression: Relations with short and long term outcome. Cognitive Therapy and Research, 42, 385–397.Google Scholar
  36. Keen, A. J., & Freeston, M. H. (2008). Assessing competence in cognitive-behavioural therapy. The British Journal of Psychiatry, 193, 60–64.Google Scholar
  37. Landis, R., Edwards, B. D., & Cortina, J. (2009). Correlated residuals among items in the estimation of measurement models. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity, and fable in the organizational and social sciences (pp. 195–214). New York, NY: Routledge.Google Scholar
  38. Lewis, C. C., Scott, K. E., & Hendricks, K. E. (2014). A model and guide for evaluating supervision outcomes in cognitive–behavioral therapy-focused training programs. Training and Education in Professional Psychology, 8, 165–173.Google Scholar
  39. Matsunaga, M. (2010). How to factor-analyze your data right: Do’s, don’ts, and how-to’s. International Journal of Psychological Research, 3, 97–110.Google Scholar
  40. McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65, 73–84.Google Scholar
  41. McHugh, R. K., Murray, H. W., & Barlow, D. H. (2009). Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventions. Behaviour Research and Therapy, 47, 946–953.Google Scholar
  42. McManus, F., Westbrook, D., Vazquez-Montes, M., Fennell, M., & Kennerley, H. (2010). An evaluation of the effectiveness of diploma-level training in cognitive behaviour therapy. Behaviour Research and Therapy, 48, 1123–1132.Google Scholar
  43. Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., & Pirritano, M. (2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of consulting and Clinical Psychology, 72, 1050–1062.Google Scholar
  44. Morse, G., Salyers, M. P., Rollins, A. L., Monroe-DeVita, M., & Pfahler, C. (2012). Burnout in mental health services: A review of the problem and its remediation. Administration and Policy in Mental Health and Mental Health Services Research, 39, 341–352.Google Scholar
  45. Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review, 33, 484–499.Google Scholar
  46. Patil, V. H., Singh, S. N., Mishra, S., & Donavan, D. T. (2008). Efficient theory development and factor retention criteria: Abandon the ‘eigenvalue greater than one’ criterion. Journal of Business Research, 61, 162–170.Google Scholar
  47. Pontoski, K., Jager-Hyman, S., Cunningham, A., Sposato, R., Schultz, L., Evans, A. C., Beck, A. T., & Creed, T. A. (2016). Using a Cognitive Behavioral framework to train staff serving individuals who experience chronic homelessness. Journal of Community Psychology, 44, 674–680.Google Scholar
  48. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., … Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38, 65–76.Google Scholar
  49. Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36, 24–34.Google Scholar
  50. Riggs, S. E., & Creed, T. A. (2017). A model to transform psychosis milieu treatment using CBT-informed interventions. Cognitive and Behavioral Practice, 24, 353–362.Google Scholar
  51. Rollins, A. L., McGrew, J. H., Kukla, M., McGuire, A. B., Flanagan, M. E., Hunt, M. G., … Salyers, M. P. (2016). Comparison of assertive community treatment fidelity assessment methods: Reliability and validity. Administration and Policy in Mental Health and Mental Health Services Research, 43, 157–167.Google Scholar
  52. Roth, A. D., Calder, F., & Pilling, S. (2011). A competence framework for child and adolescent mental health services. Edinburgh: NHS Education for Scotland.Google Scholar
  53. Rozek, D. C., Serrano, J. L., Marriott, B. R., Scott, K. S., Hickman, L. B., Brothers, B. M., … Simons, A. D. (2018). Cognitive behavioural therapy competency: Pilot data from a comparison of multiple perspectives. Behavioural and Cognitive Psychotherapy, 46, 244–250.Google Scholar
  54. Schoenwald, S. K., Carter, R. E., Chapman, J. E., & Sheidow, A. J. (2008). Therapist adherence and organizational effects on change in youth behavior problems one year after multisystemic therapy. Administration and Policy in Mental Health and Mental Health Services Research, 35, 379–394.Google Scholar
  55. Shaw, B. F., Elkin, I., Yamaguchi, J., Olmsted, M., Vallis, T. M., Dobson, K. S., … Imber, S. D. (1999). Therapist competence ratings in relation to clinical outcome in cognitive therapy of depression. Journal of Consulting and Clinical Psychology, 67, 837–846.Google Scholar
  56. Sholomskas, D. E., Syracuse-Siewert, G., Rounsaville, B. J., Ball, S. A., Nuro, K. F., & Carroll, K. M. (2005). We don’t train in vain: a dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology, 73, 106–115.Google Scholar
  57. Simons, A. D., Padesky, C. A., Montemarano, J., Lewis, C. C., Murakami, J., Lamb, K., … Beck, A. T. (2010). Training and dissemination of cognitive behavior therapy for depression in adults: A preliminary examination of therapist competence and client outcomes. Journal of Consulting and Clinical Psychology, 78, 751–756.Google Scholar
  58. Stallard, P. (2005). A clinicians guide to think good feel good: Using CBT with children and young people. Winchester: Wiley.Google Scholar
  59. Stallard, P., Myles, P., & Branson, A. (2014). The cognitive behaviour therapy scale for children and young people (CBTS-CYP): Development and psychometric properties. Behavioural and Cognitive Psychotherapy, 42, 269–282.Google Scholar
  60. Stirman, S. W., Calloway, A., Toder, K., Miller, C. J., DeVito, A. K., Meisel, S. N., … Crits-Christoph, P. (2013a). Community mental health provider modifications to cognitive therapy: Implications for sustainability. Psychiatric Services, 64, 1056–1059.Google Scholar
  61. Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science, 7, 1–19.Google Scholar
  62. Stirman, S. W., Marques, L., Creed, T. A., Gutner, C. A., DeRubeis, R., Barnett, P. G., … Jo, B. (2018). Leveraging routine clinical materials and mobile technology to assess CBT fidelity: The Innovative Methods to Assess Psychotherapy Practices (imAPP) study. Implementation Science, 13, 1–11.Google Scholar
  63. Stirman, S. W., Miller, C. J., Toder, K., & Calloway, A. (2013b). Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implementation Science, 8, 1–12.Google Scholar
  64. Strunk, D. R., Brotman, M. A., & DeRubeis, R. J. (2010). The process of change in cognitive therapy for depression: Predictors of early inter-session symptom gains. Behaviour Research and Therapy, 48, 599–606.Google Scholar
  65. Thomas, S. L., & Heck, R. H. (2001). Analysis of large-scale secondary data in higher education research: Potential perils associated with complex sampling designs. Research in Higher Education, 42, 517–540.Google Scholar
  66. Trepka, C., Rees, A., Shapiro, D. A., Hardy, G. E., & Barkham, M. (2004). Therapist competence and outcome of cognitive therapy for depression. Cognitive Therapy and Research, 28, 143–157.Google Scholar
  67. Vallis, T. M., Shaw, B. F., & Dobson, K. S. (1986). The cognitive therapy scale: Psychometric properties. Journal of Consulting and Clinical Psychology, 54, 381–385.Google Scholar
  68. Van Prooijen, J. W., & Van Der Kloot, W. A. (2001). Confirmatory analysis of exploratively obtained factor structures. Educational and Psychological Measurement, 61(5), 777–792.Google Scholar
  69. Waltman, S. H., Hall, B., McFarr, L., Beck, A. T., & Creed, T. A. (2017). In-session stuck points and pitfalls of community clinicians learning CBT: A qualitative investigation. Cognitive and Behavioral Practice, 24, 256–267. Scholar
  70. Waltman, S. H., Hall, B. C., McFarr, L. M., & Creed, T. A. (2018). Clinical case consultation and experiential learning in cognitive behavioral therapy implementation: Brief qualitative investigation. Journal of Cognitive Psychotherapy, 32, 112–127.Google Scholar
  71. Westbrook, D., Sedgwick-Taylor, A., Bennett-Levy, J., Butler, G., & McManus, F. (2008). A pilot evaluation of a brief CBT training course: Impact on trainees’ satisfaction, clinical skills and patient outcomes. Behavioural and Cognitive Psychotherapy, 36, 569–579.Google Scholar
  72. Westra, H. A., Arkowitz, H., & Dozois, D. J. (2009). Adding a motivational interviewing pretreatment to cognitive behavioral therapy for generalized anxiety disorder: A preliminary randomized controlled trial. Journal of Anxiety Disorders, 23, 1106–1117.Google Scholar
  73. Young, J., & Beck, A. (1980). Cognitive therapy scale: Rating manual. Unpublished manuscript. Philadelphia: Center for Cognitive Therapy.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Aaron T Beck Psychopathology Research CenterUniversity of Pennsylvania Perelman School of MedicinePhiladelphiaUSA
  2. 2.Aaron T. Beck Psychopathology Research CenterUniversity of Pennsylvania Perelman School of MedicinePhiladelphiaUSA

Personalised recommendations