Abstract
A key goal for implementation science is the identification of evidence-based consultation protocols and the active ingredients within these protocols that drive clinician behavior change. The current study examined clinicians’ self-coding of fidelity as a potential active ingredient of consultation for the Attachment and Biobehavioral Catch-up (ABC) intervention. It also examined two other potential predictors of clinician fidelity in response to consultation: dosage of consultation and working alliance. Twenty-nine clinicians (97% female, 62% White, M age = 34 years) participated in a year of weekly fidelity-focused ABC consultation sessions, for which clinicians self-coded fidelity and received consultant feedback on both their coding and their fidelity. Data from the ABC fidelity measure were available for 1067 sessions coded by consultants, and clinicians’ self-coding accuracy was calculated from 1044 sessions coded by both clinicians and consultants. Alliance was measured with the Working Alliance Inventory—Trainee and Supervisor Versions. The study was observational, and fidelity and self-coding accuracy were modeled across time using hierarchical linear modeling. Clinicians’ ABC fidelity, as well as their self-coding accuracy, increased over the course of consultation. Clinicians’ self-coding accuracy predicted their initial fidelity and growth in fidelity. Working alliance was also linked to fidelity and self-coding accuracy. These results suggest that clinician self-coding should be further examined as an active ingredient of consultation. The study has important implications for the design of consultation procedures and fidelity assessments.
Similar content being viewed by others
Data Availability
In order to maximize sample size in the current study, the dataset used in Caron and Dozier (2019) was included in the current study, representing 6 of the 29 clinicians. Preliminary data from the current study were published as a dissertation (Caron, 2017), but have not previously been published in a peer-reviewed journal.
References
Bahrick, A. S. (1989) Role induction for counselor trainees: Effects on the supervisory working alliance (Doctoral dissertation). Retrieved from OhioLINK Electronic Theses & Dissertations Center.
Barnett, M. L., Niec, L. N., & Acevedo-Polakovich, I. D. (2014). Assessing the key to effective coaching in parent–child interaction therapy: The therapist-parent interaction coding system. Journal of Psychopathology and Behavioral Assessment, 36(2), 211–223. https://doi.org/10.1007/s10862-013-9396-8
Beale, S., Liness, S., & Hirsch, C. R. (2020). Trainee self-assessment of cognitive behaviour therapy competence during and after training. The Cognitive Behaviour Therapist. https://doi.org/10.1017/S1754470X19000357
Bearman, S. K., Schneiderman, R. L., & Zoloth, E. (2017). Building an evidence base for effective supervision practices: An analogue experiment of supervision to increase EBT fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 44(2), 293–307. https://doi.org/10.1007/s10488-016-0723-8
Bearman, S. K., Weisz, J. R., Chorpita, B. F., Hoagwood, K., Ward, A., Ugueto, A. M., Bernstein, A. & The Research Network on Youth Mental Health. (2013). More practice, less preach? The role of supervision processes and therapist characteristics in EBP implementation. Administration and Policy in Mental Health and Mental Health Services Research, 40(6), 518–529. https://doi.org/10.1007/s10488-013-0485-5
Beidas, R. S., Edmunds, J. M., Marcus, S. C., & Kendall, P. C. (2012). Training and consultation to promote implementation of an empirically supported treatment: A randomized trial. Psychiatric Services, 63(7), 660–665. https://doi.org/10.1176/appi.ps.201100401
Bennett-Levy, J., Thwaites, R., Chaddock, A., & Davis, M. (2009). Reflective practice in cognitive behavioural therapy: The engine of lifelong learning. In J. Stedmon & R. Dallos (Eds.), Reflective practice in psychotherapy and counselling (pp. 115–135). McGraw Hill.
Bernard, K., Dozier, M., Bick, J., & Gordon, M. K. (2015). Intervening to enhance cortisol regulation among children at risk for neglect: Results of a randomized clinical trial. Development and Psychopathology, 27(3), 829–841. https://doi.org/10.1017/S095457941400073X
Bernard, K., Dozier, M., Bick, J., Lewis-Morrarty, E., Lindhiem, O., & Carlson, E. (2012). Enhancing attachment organization among maltreated children: Results of a randomized clinical trial. Child Development, 83(2), 623–636. https://doi.org/10.1111/j.1467-8624.2011.01712.x
Bhat, C. S., & Davis, T. E. (2007). Counseling supervisors’ assessment of race, racial identity, and working alliance in supervisory dyads. Journal of Multicultural Counseling and Development, 35(2), 80–91. https://doi.org/10.1002/j.2161-1912.2007.tb00051.x
Brookman-Frazee, L., Stadnick, N. A., Lind, T., Roesch, S., Terrones, L., Barnett, M. L., Regan, J., Kennedy, C. A., Garland, A. F., & Lau, A. S. (2021). Therapist-observer concordance in ratings of EBP strategy delivery: Challenges and targeted directions in pursuing pragmatic measurement in children’s mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 48, 155–170. https://doi.org/10.1007/s10488-020-01054-x
Brosan, L., Reynolds, S., & Moore, R. G. (2008). Self-evaluation of cognitive therapy performance: Do therapists know how competent they are? Behavioural and Cognitive Psychotherapy, 36(5), 581–587. https://doi.org/10.1017/s1352465808004438
Caron, E. (2017). Effects and processes of fidelity-focused consultation (Doctoral dissertation, University of Delaware). UDSpace Digital Archive. https://udspace.udel.edu/bitstream/handle/19716/23096/Caron_udel_0060D_13001.pdf?sequence=1
Caron, E., Bernard, K., & Dozier, M. (2018). In vivo feedback predicts parent behavior change in the Attachment and Biobehavioral Catch-up intervention. Journal of Clinical Child & Adolescent Psychology, 47(sup1), S35–S46. https://doi.org/10.1080/15374416.2016.1141359
Caron, E., & Dozier, M. (2019). Effects of fidelity-focused consultation on clinicians’ implementation: An exploratory multiple baseline design. Administration and Policy in Mental Health and Mental Health Services Research, 46(4), 445–457. https://doi.org/10.1007/s10488-019-00924-3
Caron, E., Lind, T. A., & Dozier, M. (2021). Strategies that promote therapist engagement in active and experiential learning: Micro-level sequential analysis. The Clinical Supervisor, 40(1), 112–133. https://doi.org/10.1080/07325223.2020.1870023
Caron, E., Muggeo, M. A., Souer, H. R., Pella, J. E., & Ginsburg, G. S. (2020). Concordance between clinician, supervisor and observer ratings of therapeutic competence in CBT and treatment as usual: Does clinician competence or supervisor session observation improve agreement? Behavioural and Cognitive Psychotherapy, 48(3), 350–363. https://doi.org/10.1017/s1352465819000699
Carroll, K., Nich, C., & Rounsaville, B. (1998). Utility of therapist session checklists to monitor delivery of coping skills treatment for cocaine abusers. Psychotherapy Research, 8(3), 307–320.
Child and Adolescent Health Measurement Initiative. 2018–2019 National Survey of Children’s Health (NSCH) data query. Data Resource Center for Child and Adolescent Health supported by the U.S. Department of Health and Human Services, Health Resources and Services Administration (HRSA), Maternal and Child Health Bureau (MCHB). Retrieved 05/03/21 from www.childhealthdata.org.
Collyer, H., Eisler, I., & Woolgar, M. (2020). Systematic literature review and meta-analysis of the relationship between adherence, competence and outcome in psychotherapy for children and adolescents. European Child & Adolescent Psychiatry, 29(4), 417–431.
Cox, J. R., McLeod, B. D., Jensen-Doss, A., Srivastava, V., Southam-Gerow, M. A., Kendall, P. C., & Weisz, J. R. (2020). Examining how CBT interventions for anxious youth are delivered across settings. Behavior Therapy, 51(6), 856–868. https://doi.org/10.1016/j.beth.2019.11.008
Dozier, M., Bernard, K., & Roben, C. K. P. (2017). Attachment and biobehavioral catch-up. In H. Steele & M. Steele (Eds.), The handbook of attachment-based interventions (pp. 27–49). The Guilford Press.
Dunn, C., Darnell, D., Atkins, D. C., Hallgren, K. A., Imel, Z. E., Bumgardner, K., Owens, M., & Roy-Byrne, P. (2016). Within-provider variability in motivational interviewing integrity for three years after MI training: Does time heal? Journal of Substance Abuse Treatment, 65, 74–82. https://doi.org/10.1016/j.jsat.2016.02.008
Edmunds, J. M., Beidas, R. S., & Kendall, P. C. (2013a). Dissemination and implementation of evidence–based practices: Training and consultation as implementation strategies. Clinical Psychology: Science and Practice, 20(2), 152–165. https://doi.org/10.1111/cpsp.12031
Edmunds, J. M., Kendall, P. C., Ringle, V. A., Read, K. L., Brodman, D. M., Pimentel, S. S., & Beidas, R. S. (2013b). An examination of behavioral rehearsal during consultation as a predictor of training outcomes. Administration and Policy in Mental Health and Mental Health Services Research, 40(6), 456–466. https://doi.org/10.1007/s10488-013-0490-8
Eiraldi, R., Mautone, J. A., Khanna, M. S., Power, T. J., Orapallo, A., Cacia, J., Schwartz, B. S., McCurdy, B., Keiffer, J., Paidipati, C., Kanine, R., Abraham, M., Tulio, S., Swift, L., Bressler, S. N., Cabello, B., & Jawad, A. F. (2018). Group CBT for externalizing disorders in urban schools: Effect of training strategy on treatment fidelity and child outcomes. Behavior Therapy, 49(4), 538–550. https://doi.org/10.1016/j.beth.2018.01.001
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (FMHI Publication No. 231). University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.
Funderburk, B., Chaffin, M., Bard, E., Shanley, J., Bard, D., & Berliner, L. (2015). Comparing client outcomes for two evidence-based treatment consultation strategies. Journal of Clinical Child & Adolescent Psychology, 44(5), 730–741. https://doi.org/10.1080/15374416.2014.910790
Garnett, M., Bernard, K., Hoye, J., Zajac, L., & Dozier, M. (2020). Parental sensitivity mediates the sustained effect of Attachment and Biobehavioral Catch-up on cortisol in middle childhood: A randomized clinical trial. Psychoneuroendocrinology, 121, 104809. https://doi.org/10.1016/j.psyneuen.2020.104809
Hitzeman, C., Gonsalvez, C. J., Britt, E., & Moses, K. (2020). Clinical psychology trainees’ self versus supervisor assessments of practitioner competencies. Clinical Psychologist, 24(1), 18–29. https://doi.org/10.1111/cp.12183
Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42(2), 229–243. https://doi.org/10.1007/s10488-014-0548-2
Hogue, A., Porter, N., Bobek, M., MacLean, A., Bruynesteyn, L., Jensen-Doss, A., & Henderson, C. E. (2021). Online training of community therapists in observational coding of family therapy techniques: Reliability and accuracy. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-021-01152-4
Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2(1), 88–110. https://doi.org/10.1080/19345740802539325
Inman, A. G. (2006). Supervisor multicultural competence and its relation to supervisory process and outcome. Journal of Marital and Family Therapy, 32(1), 73–85. https://doi.org/10.1111/j.1752-0606.2006.tb01589.x
Institute of Medicine. (2015). Psychosocial interventions for mental and substance use disorders. National Academies Press. https://doi.org/10.17226/19013
Isenhart, C., Dieperink, E., Thuras, P., Fuller, B., Stull, L., Koets, N., & Lenox, R. (2014). Training and maintaining motivational interviewing skills in a clinical trial. Journal of Substance Use, 19(1–2), 164–170. https://doi.org/10.3109/14659891.2013.765514
Kazantzis, N., Whittington, C., Zelencich, L., Kyrios, M., Norton, P. J., & Hofmann, S. G. (2016). Quantity and quality of homework compliance: A meta-analysis of relations with outcome in cognitive behavior therapy. Behavior Therapy, 47(5), 755–772.
Kazdin, A. E., & Nock, M. K. (2003). Delineating mechanisms of change in child and adolescent therapy: Methodological issues and research recommendations. Journal of Child Psychology and Psychiatry, 44(8), 1116–1129. https://doi.org/10.1111/1469-7610.00195
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Ladany, N., Lehrman-Waterman, D., Molinaro, M., & Wolgast, B. (1999). Psychotherapy supervisor ethical practices: Adherence to guidelines, the supervisory working alliance, and supervisee satisfaction. The Counseling Psychologist, 27(3), 443–475. https://doi.org/10.1177/0011000099273008
Lind, T., Bernard, K., Yarger, H. A., & Dozier, M. (2020). Promoting compliance in children referred to child protective services: A randomized clinical trial. Child Development, 91(2), 563–576. https://doi.org/10.1111/cdev.13207
Lind, T., Raby, K. L., Caron, E., Roben, C. K., & Dozier, M. (2017). Enhancing executive functioning among toddlers in foster care with an attachment-based intervention. Development and Psychopathology, 29(2), 575–586. https://doi.org/10.1017/S0954579417000190
Liness, S., Beale, S., Lea, S., Byrne, S., Hirsch, C. R., & Clark, D. M. (2019). The sustained effects of CBT training on therapist competence and patient outcomes. Cognitive Therapy and Research, 43(3), 631–641.
Loades, M. E., & Myles, P. J. (2016). Does a therapist’s reflective ability predict the accuracy of their self-evaluation of competence in cognitive behavioural therapy? The Cognitive Behaviour Therapist. https://doi.org/10.1017/S1754470X16000027
Magill, M., Kiluk, B. D., McCrady, B. S., Tonigan, J. S., & Longabaugh, R. (2015). Active ingredients of treatment and client mechanisms of change in behavioral treatments for alcohol use disorders: Progress 10 years later. Alcoholism: Clinical and Experimental Research, 39(10), 1852–1862. https://doi.org/10.1111/acer.12848
Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19(2), 181–193. https://doi.org/10.1080/10503300802688460
Martino, S., Paris, M., Añez, L., Nich, C., Canning-Ball, M., Hunkele, K., Olmstead, T. A., & Carroll, K. M. (2016). The effectiveness and cost of clinical supervision for motivational interviewing: A randomized controlled trial. Journal of Substance Abuse Treatment, 68, 11–23. https://doi.org/10.1016/j.jsat.2016.04.005
Masia Warner, C., Brice, C., Esseling, P. G., Stewart, C. E., Mufson, L., & Herzig, K. (2013). Consultants’ perceptions of school counselors’ ability to implement an empirically-based intervention for adolescent social anxiety disorder. Administration and Policy in Mental Health and Mental Health Services Research, 40(6), 541–554. https://doi.org/10.1007/s10488-013-0498-0
McLeod, B. D., Cox, J. R., Jensen-Doss, A., Herschell, A., Ehrenreich-May, J., & Wood, J. J. (2018). Proposing a mechanistic model of clinician training and consultation. Clinical Psychology: Science and Practice, 25(3), e12260. https://doi.org/10.1111/cpsp.12260
McManus, F., Rakovshik, S., Kennerley, H., Fennell, M., & Westbrook, D. (2012). An investigation of the accuracy of therapists’ self-assessment of cognitive-behaviour therapy skills. British Journal of Clinical Psychology, 51(3), 292–306. https://doi.org/10.1111/j.2044-8260.2011.02028.x
Moyers, T. B., Manuel, J. K., Wilson, P. G., Hendrickson, S. M., Talcott, W., & Durand, P. (2008). A randomized trial investigating training in motivational interviewing for behavioral health providers. Behavioural and Cognitive Psychotherapy, 36(2), 149–162. https://doi.org/10.1017/S1352465807004055
Peavy, K. M., Guydish, J., Manuel, J. K., Campbell, B. K., Lisha, N., Le, T., Delucchi, K., & Garrett, S. (2014). Treatment adherence and competency ratings among therapists, supervisors, study-related raters and external raters in a clinical trial of a 12-step facilitation for stimulant users. Journal of Substance Abuse Treatment, 47(3), 222–228. https://doi.org/10.1016/j.jsat.2014.05.008
Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36, 24–34. https://doi.org/10.1007/s10488-008-0197-4
Raby, K. L., Freedman, E., Yarger, H. A., Lind, T., & Dozier, M. (2019). Enhancing the language development of toddlers in foster care by promoting foster parents’ sensitivity: Results from a randomized controlled trial. Developmental Science, 22(2), e12753. https://doi.org/10.1111/desc.12753
Raudenbush, S. W., & Bryk, A. S. (2002) Hierarchical linear models: Applications and data analysis methods. Sage
Reichelt, F. K., James, I. A., & Blackburn, I. M. (2003). Impact of training on rating competence in cognitive therapy. Journal of Behavior Therapy and Experimental Psychiatry, 34(2), 87–99. https://doi.org/10.1016/s0005-7916(03)00022-3
Schoenwald, S. K., Sheidow, A. J., & Letourneau, E. J. (2004). Toward effective quality assurance in evidence-based practice: Links between expert consultation, therapist fidelity, and child outcomes. Journal of Clinical Child and Adolescent Psychology, 33(1), 94–104. https://doi.org/10.1207/S15374424JCCP3301_10
Schwalbe, C. S., Oh, H. Y., & Zweben, A. (2014). Sustaining motivational interviewing: A meta-analysis of training studies. Addiction, 109(8), 1287–1294. https://doi.org/10.1111/add.12558
Serfaty, M., Shafran, R., Vickerstaff, V., & Aspden, T. (2020). A pragmatic approach to measuring adherence in treatment delivery in psychotherapy. Cognitive Behaviour Therapy, 49(5), 347–360. https://doi.org/10.1080/16506073.2020.1717594
Stice, E., Butryn, M. L., Rohde, P., Shaw, H., & Marti, C. N. (2013). An effectiveness trial of a new enhanced dissonance eating disorder prevention program among female college students. Behaviour Research and Therapy, 51(12), 862–871. https://doi.org/10.1016/j.brat.2013.10.003
Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012). Bridging research and practice: Models for dissemination and implementation research. American Journal of Preventive Medicine, 43(3), 337–350. https://doi.org/10.1016/j.amepre.2012.05.024
Weck, F., Kaufmann, Y. M., & Höfling, V. (2017). Competence feedback improves CBT competence in trainee therapists: A randomized controlled pilot study. Psychotherapy Research, 27(4), 501–509. https://doi.org/10.1080/10503307.2015.1132857
Wehby, J. H., Maggin, D. M., Moore Partin, T. C., & Robertson, R. (2012). The impact of working alliance, social validity, and teacher burnout on implementation fidelity of the good behavior game. School Mental Health, 4, 22–33. https://doi.org/10.1007/s12310-011-9067-4
Weisz, J. R., Kuppens, S., Eckshtain, D., Ugueto, A. M., Hawley, K. M., & Jensen-Doss, A. (2013). Performance of evidence-based youth psychotherapies compared with usual clinical care: A multilevel meta-analysis. JAMA Psychiatry, 70(7), 750–761. https://doi.org/10.1001/jamapsychiatry.2013.1176
Weisz, J. R., Kuppens, S., Ng, M. Y., Eckshtain, D., Ugueto, A. M., Vaughn-Coaxum, R., & Fordwood, S. R. (2017). What five decades of research tells us about the effects of youth psychological therapy: A multilevel meta-analysis and implications for science and practice. American Psychologist, 72(2), 79–117. https://doi.org/10.1037/a0040360
Acknowledgements
We would like to thank the parent coaches and consultants who participated in this study.
Funding
This study was supported by the National Institutes of Health under Grants R01 MH052135, R01 MH074374, and R01 MH084135.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Ethical Approval
This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Institutional Review Board of the University of Delaware.
Informed Consent
Informed consent was obtained from all clinicians and consultants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Caron, E., Dozier, M. Self-Coding of Fidelity as a Potential Active Ingredient of Consultation to Improve Clinicians’ Fidelity. Adm Policy Ment Health 49, 237–254 (2022). https://doi.org/10.1007/s10488-021-01160-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-021-01160-4