Advertisement

Systematic literature review and meta-analysis of the relationship between adherence, competence and outcome in psychotherapy for children and adolescents

  • Hannah CollyerEmail author
  • Ivan Eisler
  • Matt Woolgar
Open Access
Review
  • 197 Downloads

Abstract

The extent to which therapist adherence to guidelines and clinician skill or competence may play a role in the prediction of therapeutic outcomes remains inconclusive. This systematic literature review and meta-analysis considers whether adherence or competence predicts youth outcome in child and adolescent psychotherapy, and whether there are any identifiable factors which moderate the strength or direction of this relationship. A systematic literature search identified 35 studies in 52 papers. The studies contained 29 effect sizes for the relationship between adherence and outcome, while nine effect sizes were extracted for competence, and a further five effects measured a composite of adherence and competence constructs, referred to as fidelity in this report. The meta-analysis indicated a small but significant relationship between therapist adherence and outcome, although the small size of effect suggests that outcomes are likely to be more strongly associated with factors other than adherence. No significant relationship was identified between competence or composite fidelity and outcome. Although variance was observed in effect sizes, no significant moderation by client group, intervention type, or implementation measure informant was identified. Further study is needed to understand the specific circumstances under which adherence and outcome are related.

Keywords

Adherence Competence Fidelity Therapy Child Adolescent 

Introduction

Treatment fidelity is often used to refer to a range of processes involved in model delivery. The classification of this construct varies across studies, often including characteristics such as dosage and differentiation [1], as well as the two processes which are the focus of the present investigation, therapist adherence and competence. Therapist adherence refers to the accuracy with which the specified elements of an intervention model are implemented, while competence refers to how skilfully the intervention is delivered [2].

Proponents of manualised models or empirically supported treatments (ESTs) argue that variability in outcomes when transporting interventions from research settings could be attributed to variable therapist adherence or competence, rather than failures in the therapeutic model. This suggests that the key to transporting findings from well-controlled efficacy studies to everyday clinical settings is to ensure high levels of therapist adherence and competence in the delivery of the therapeutic model [3].

Critics, on the other hand [4], have argued that the assumptions of manualised interventions are not always well substantiated. For example, adherence is not consistently seen to be associated with outcome, and trials often fail to test the effectiveness of individual components within the therapy. It has also been argued that operationalizing the manual as being the treatment itself (at least implicitly but often explicitly) significantly reduces the fundamental role of clinical judgement [5]. Understanding the role of therapist adherence and competence in predicting treatment outcomes has important implications for our understanding of evidence-based therapeutic practice.

Characteristics of the therapeutic context can influence the therapist’s ability to deliver the intervention with adherence and competence. Factors which might influence practice include client complexity or motivation, or the level of training and supervision provided [6, 7, 8, 9].

There is some indication that the relationship between therapist adherence or competence and outcome is variable across intervention types and client groups. For example, there is considerable evidence that therapist adherence plays a role in predicting outcomes in multisystemic therapy (MST) for adolescent antisocial behaviour [10], but there is less consistent evidence of such a relationship in cognitive-behavioural interventions [3]. Further, adolescent substance use outcome has been associated with therapist competence in motivational interviewing (MI) [11], but was not found to be associated with competence in cognitive behavioural therapy and multidimensional family therapy (MDFT) [12]. These findings suggest that adherence and competence may play a more prominent role in some interventions than others.

However, even within the same intervention type, the relationship between implementation and outcome can be variable. This may relate to methodological differences across studies. Variation in approaches to measurement is seen across studies, which may affect the validity of ratings. For example, adherence rated by independent observers may be less subject to bias or demand characteristics than supervisor, therapist or client ratings [13]. Inconsistencies between studies may also be due to moderation by other variables. For instance, Barber et al. [14] found that therapist adherence was less strongly related to outcome when therapeutic alliance was strong. Similarly there is evidence that in the context of a poor therapeutic alliance, strong adherence is associated with worse outcome [15]. Further, Huppert et al. [9] found that therapist adherence predicted poorer outcome in clients with low motivation, but was not related to outcome in clients with high motivation. However, the role of moderators has not been extensively studied and, therefore, understanding of the characteristics which might affect the relationship between implementation and outcome remains limited.

Some research has also indicated a curvilinear relationship between therapist adherence and outcome [14]. This would suggest that rigidly high adherence may limit the opportunity for the therapist to respond to individual needs, while low adherence may lead the therapist to miss out many key elements of the therapy. Barber et al. [14] also propose a potential interaction between therapist adherence and competence, suggesting that where competence is high, moderate rather than high adherence may lead to the best outcomes. Although no such relationship was found, these authors suggest that overall high levels of competence reduced the chance of detecting this effect and that further study is warranted.

Syntheses of the evidence have attempted to make sense of the contradictory findings across the evidence base. A meta-analysis of individual psychotherapy, comprising mainly adult intervention studies, found that when controlling for therapeutic alliance, neither adherence nor competence predicted outcome particularly well, with the exception of a potential link between competence and outcome for certain interventions such as those for major depression [16]. These findings suggest that there is limited association between implementation and outcome for individual and primarily adult intervention. In contrast, a systematic review between adherence and outcome in child and adolescent mental health indicated that there was an association [17]. This suggests a more consistent relationship between implementation and outcome in child and adolescent intervention than with adults.

However, syntheses focussing on specific intervention approaches or client groups within child and adolescent psychotherapy show variation in the association between implementation and outcome. A recent systematic review of child and adolescent cognitive behavioural therapy concluded that the understanding of the relationships between adherence or competence and outcome remains inconclusive [18]. In contrast, a recent meta-analysis has indicated a relationship between integrity and outcome in intervention for juvenile antisocial behaviour [19]. These findings suggest, in line with inferences drawn based on individual studies, that implementation may be associated with outcome in some child and adolescent intervention approaches but not others.

Much of the evidence synthesis so far for child and adolescent psychotherapy relies on qualitative review or a narrow focus on specific intervention approaches or client groups. In light of this, the present meta-analysis aims to provide an updated, comprehensive quantitative synthesis across this complex evidence base. The primary research question aims to understand whether youth outcome in child and adolescent psychotherapy is predicted by (a) adherence, (b) competence and (c) composite fidelity. Composite fidelity can be variably defined in the literature, but for the purpose of this study is used to refer to a composite of adherence and competence.

A secondary research question considers characteristics which previous research has indicated may act as potential moderators of the strength of the relationship between implementation and outcome but which are not yet fully understood. This will test whether the size of any observed association between implementation and outcome is moderated by (a) intervention modality (i.e. the nature of the therapeutic approach) (b) clinical group (i.e. the diagnostic characteristics of the focus client group) or (c) adherence or competence informant (i.e. whether implementation is rated by clients, therapist, supervisor or an independent observer).

Method

This review was prospectively registered on PROSPERO International Prospective Register of Systematic Reviews (ID Number CRD42016046671). Procedures and reporting are in accordance with the recommendations of the PRISMA guidelines [20, 21].

Literature searches

Systematic literature searches were completed up to 21st July 2018 using PsychINFO, Embase and Medline databases. Searches and data extraction were carried out by the lead author.

Search terms

  • Fields: Abstract, Keyword, Subject Headings, Title.

  • Terms: (Fidelity or adherence or integrity or competence or implementation) and (treatment or therapy or psychotherapy or intervention or therapist or therapeutic or counselling) and (child or adolescent or parent or parenting or teenager or teenage or children or adolescents or youth).

Full details of search and exclusion terms are available in Online Resource 1. Duplicates were removed from the search results and the title and abstract of each article were screened, followed by screening of all remaining full text articles. Relevant articles referenced in the obtained studies were also considered for inclusion.

Inclusion and exclusion criteria

Included articles were those reporting a primary study of an outpatient psychosocial targeted intervention for mental health or behaviour, within a clinical population of children and adolescents up to age 21. Studies were included which reported statistical analysis of the relationship between a quantifiable measure of either therapist adherence, competence or composite fidelity at the therapist or client level and a measure of youth mental health or behavioural outcome. Interventions delivered by teachers, unqualified school professionals, caregivers or peers were excluded, as were universal interventions or those targeting health or educational outcomes. All included studies were reported in English language and published in a peer review journal.

Study quality

Included studies were critically appraised for study quality by the lead author, using criteria to assess power and risk of bias. Given that there are no consistently established guidelines for assessing study quality in non-experimental studies such as the observational and correlational designs included in the present analysis, the included studies were assessed according to eight pre-defined binary criteria, established based on recommendations from a number of similar previous reviews [22, 23, 24]. Studies which met the following criteria were rated as being of higher quality:
  • Reported power ≥ 80%, or sample size ≥ 53 per group [25, 26].

  • Reported participation rate ≥ 70% of eligible or approached sample [23].

  • Dropout at follow-up ≤ 30%, or missing data shown to not differ from those with complete data on any of the predictor variables, or showing that predictor–outcome relationships remained the same after adjusting for missing data [23].

  • Used a reliable diagnostic measure or rating scale, or clear selection criteria reported [24].

  • External or independent observer rated measure of adherence or competence, measured across multiple time points (unless a one-session intervention), and inter-rater agreement established in the study or use of coders trained to this level according to the following criteria: intraclass correlation coefficient (ICC) ≥ 0.60 [27] or Kappa ≥ 0.61 [28], or percent agreement ≥ 90% [29], or the score is based on agreement by multiple raters.

  • Used questionnaire outcome measures which demonstrate reliability and validity in the present or previous studies (internal consistency: α ≥ 0.70 [22], convergent validity: r = 0.6 [30] or interrater reliability: ICC ≥ 0.60 [27] or Kappa ≥ 0.61 [28]) or other type of outcome measure with low risk of bias (e.g., objective measure such as urine screening).

  • Controlled or adjusted for the influence of baseline symptom severity, or used a measure of change.

  • Reported adjustment or control for influence of potential confounders, or indication that potential confounders do not relate to outcome.

Meta-analysis

Data extraction and preparation

Effect sizes for included studies were classified according to the constructs of adherence and competence as defined in the introduction, while the term fidelity was used to classify studies reporting a composite of these two constructs. Where papers reported on the same or overlapping samples, a single effect size was used to ensure that each meta-analysis included only one independent effect size from each subject sample. The effect size selected for inclusion was that reported at the earliest outcome timepoint, or where different sample sizes were used, the largest sample. Where a range of outcomes were measured, effects were included only for those addressing the primary problem in each study (see Online Resource 2). To avoid inclusion of statistically dependent effect sizes, which can threaten validity [31], an average effect size was calculated where multiple primary outcomes were measured, where separate effect sizes were reported for multiple informants on the same measure, or where effects were reported for multiple adherence or competence components. Where multiple follow-up time points were reported, the earliest post intervention time point for which sufficient data were available was selected, most often a measurement directly post intervention.

The selected effect size, Pearson’s correlation coefficient (r), was directly reported in several studies. Results were standardised such that positive correlations were given to findings in the direction of our prediction, i.e. increased adherence or competence predicted therapeutic improvement. Where r was not directly reported, conversions were made from other effect sizes or from reported test statistics [32, 33, 34]. When sufficient statistical data were not available but effects were reported to be non-significant, the r value was estimated conservatively as zero. Remaining studies reporting significant effects but with insufficient data to compute effect sizes were excluded from the analysis.

Each study was coded for three multi-level categorical moderators: clinical diagnostic group [emotional disorders, child and adolescent behaviour problems, substance use, autism spectrum disorder (ASD)], intervention modality [cognitive behavioural therapy (CBT), family therapy, parenting, other youth non-CBT intervention (individual or group), other non-CBT intervention with youth and parent components (individual or group)] and adherence or competence informant (observer, therapist, client report, supervisor report, composite of client and therapist ratings).

Statistical analysis

Analysis was completed using SPSS version 25 [35] using syntax from Lipsey and Wilson [36] based on Hedges and Vevea [37] random effects meta-analysis. Estimated mean population effect and its significance was computed as an average of standardised Fisher’s z transformed effect sizes, weighted by inverse variance as a function of sample size (n − 3). A test of homogeneity established whether any variance in effect sizes is likely due to sampling error alone or systematic differences between studies. For significant effects, a sensitivity analysis tested how conclusions might be affected if studies at higher risk of bias were excluded from the analysis. Excluded studies were those scoring low quality in four or more of the eight areas against which they were rated.

A model analogous to analysis of variance examined whether moderator characteristics are associated with heterogeneity of effect sizes, by testing whether overall weighted effect size differs significantly between categories of each moderator. Moderator levels with only one available effect size could not be included in the moderation analysis.

Risk of publication bias was assessed using a rank correlation test of the strength or significance of the relationship between standardised effect size and associated variance [38]. The potential impact of any publication bias was estimated using a sensitivity analysis using syntax for SPSS and R [39] which computed adjusted estimates of effect size based on models estimating the impact of moderate or severe bias [40].

Results

The search returned 23,372 papers. 7815 duplicates were removed, 14,741 were excluded based on title and abstract and a further 765 were excluded based on fulltext (see Fig. 1). The remaining 51 papers met inclusion criteria, and one additional paper [41] was identified as a reference within one of the included studies. The final review, therefore, included 52 papers covering 35 separate studies. The complete list of included papers and their characteristics is presented in Table 1, with further detail available in Online Resource 2.
Fig. 1

Prisma flow chart of study identification

Table 1

Details of included Studies

Study

Client group

Intervention

Sample Size

Age range (mean, SD)

Primary outcome measure(s) used in meta-analysis

[50]

Families in crisis

Family crisis intervention program (FCIP)

183

0–18 (11.54, 5.17)

Parent and youth rated Strengths and Difficulties Questionnaire (SDQ)

[51]

Disruptive/withdrawn

Early risers “skills for success” program

262

Grade 1 and 2 (6.92, 0.93)

Teacher rated Behavioural Assessment System for Children-II (BASC-2)

[80]

Attention-deficit/hyperactivity disorder (ADHD)

Cognitive behavioural therapy (CBT) (solution-focused treatment)

36

12–17 (14.28, 1.19)

Dutch BRIEF (inventory of executive function) plan/organise scale

[52]

Behavioural problems

Family brief and intensive intervention (BII)

160

6–17 (14.7, 2.1)

Placement/services required

[71]a

Conduct Disorder

Incredible years parent group

86

36–59 months (46.1 months, 6)A

Parent rated Eyberg Child Behaviour Inventory (ECBI) intensity and problem scales

[81]

Risk of depressive disorder

Youth single session brief motivational interviewing (MI)

43

14–21 (17.44, 2.17)

Youth CES-D-10 (depression scale)

[43]

Substance abuse

Youth adolescent community reinforcement approach (A-CRA) (and parent and family sessions)

399

12–18 (N/A)

Alcohol and other drug use

[53]b

Substance abuse

Youth A-CRA (an parent and family sessions)

953

N/A (15.8, 1.4)

Global appraisal of individual needs (GAIN)—substance use remission status

[54]

Substance using youth offenders

Multisystemic therapy (MST)

40

12–17

Arrest record; self-report alcohol and cannabis use; Self-Report Delinquency scale (SRDS)

[55]

Depression symptoms

Youth group CBT

147

11–12 (N/A)

Youth report Clinical Depression Inventory (CDI)

[46]c

Anxiety disorder

Individual CBT (and parent sessions)

17

7–17 (11.12, 2.75)

Clinician anxiety disorders interview schedule (ADIS-C/P); parent and youth rated screen for child anxiety—related emotional disorders (SCARED)

[56]

Behavioural problems

Functional family therapy (FFT)

98

N/A (14.05, 1.93)

Parent and youth rated SDQ

[57]

Behavioural problems

FFT

42

N/A (14.22, 1.45)

Parent rated SDQ

[84]

Incarcerated youth with antisocial behaviour

Youth group CBT

89

N/A (15.54, 1.56)

Youth rated social skills; moral value and moral Judgement; cognitive distortion

[10]d

Violent and chronic youth offenders

MST

82

10–17 (15.22, N/A)A

Arrests and incarceration; SRDS; Parent rated Problem Behaviour Checklist (RPBC)

[58]

Substance using youth offenders

MST

58

12–17 (15.7, 1.0)A

Self-report alcohol/marijuana

[59]

Behavioural problems

FFT

59

9–16 (13 years 7 months)

Composite measure of parent-rated conduct problems, parent- and youth-rated alcohol and substance use frequency, and SRDS

[12]e

Substance use and behaviour problems

Family multidimensional family therapy (MDFT) and youth CBT

74 MDFT, 62 CBT

13–17 (15.5, 1.3)

Time line follow-back (TLFB) cannabis frequency

[60]

Substance use disorder

MST

41

13–17 (15 years 10 months, 14 months)

Self-report cannabis abstinence

[82]f

Externalising behaviour

Parent management training (individual)

331

04–12 (8.7, N/A)

Parent child behaviour checklist (CBCL) and parent daily report (PDR); teacher report form (TRF)

[78]g

Antisocial/problem behaviour

MST

4290

12–17 (15.62, 1.38)

Living at home; engaged in school or work; no new arrests

[47]

Anxiety disorder

Individual and group CBT (and parent sessions)

52

08–12 (10.22, 1.15)

Clinician ADIS-C/P; Youth MASC (anxiety scale); parent CBCL internalising scale

[63]h

Behaviour/conduct disorder

MST

973

12–17 (N/A)

Criminal behaviour at completion of treatment

[85]

Externalising behaviour

Parent Management Training Oregon (PMTO)

46

4–12 (7.85, 2.36)

Parent-rated CBCL and teacher-rated TRF (Dutch versions)

[11]

Cannabis use

Youth single session MI

75

16–19 (18, N/A)

Cannabis cessation

[62]

Exposure to interparental violence

Youth and parent trauma-focused psycho-education

100

06–12 (9.35, 1.55)

Parent report TSCYC; youth report TSCC (trauma symptom checklists)

[64]

Anxiety disorder

Youth individual CBT (and parent sessions)

279

07–17 (10.76, 2.79)

Clinician PARS (anxiety scale); youth MASC (anxiety scale); parent-rated CBCL anxiety/depression scale

[42]

Drug abuse

Brief strategic family therapy (BSFT)

246

12–17 (15.5, 1.3)

TLFB substance use

[8]

Cannabis use disorder

MDFT

Europe (212)

US (171)

13–18 (Europe: 16.3, 1.2)A (US: 15.6, 1.1)

TLFB substance use frequency and problem

[44]i

Serious antisocial behaviour

MST

1979

N/A (14.0, 2.35)

Parent rated CBCL externalising

[49]

Youth offenders

FFT

431

13–17 (N/A)

Recidivism

[45]

Social, emotional, and behavioural problems

Youth supportive-expressive group counselling

266

10–18 (N/A)

Self-report aggression

[83]j

Elevated problem behaviour

Family check-up

79

2 years 0 months-2 years 11 months (29.9 months, 3.2)

Parent-rated CBCL externalising

[65]k

Autism spectrum disorder

Youth and parent early intensive behavioural intervention (EIBI)

24

26-81 months (55.67 months, 17.63)

Observer-rated child challenging behaviour

[72]

Serious emotional disturbance

Youth and parent child psychiatric rehabilitation (CPSR)

79

N/A (10.31, 3.54)

Parent-rated youth counselling impact scale (YCIS)

AAge is reported for whole sample including control

Studies above were also reported in the following identified papers:a[73], b[74], c[75], d[76, 77], e[41, 67], f[86], g[78], h[66], i[7, 48, 68, 69, 70], j[87], k[79]

Characteristics of identified studies

The client groups in the included studies were children and adolescents with behaviour problems including offending (N = 18, 51%), substance abuse (N = 9, 26%), emotional disorders (N = 7, 20%) or autism spectrum disorder (ASD) (N = 1, 3%). Interventions were primarily family therapy (N = 16, 46%), or cognitive behavioural therapy (CBT) (N = 7, 20%), as well as parenting (N = 4, 11%), youth non-CBT intervention (N = 3, 9%) and non-CBT intervention with youth and parent components (N = 6, 17%). Detail on the specific interventions within each category is available in Online Resource 3.

Measures of therapist adherence, competence and composite fidelity differed in content and complexity. Some studies measured the frequency of implementation of certain core strategies [42], whilst others also incorporated an evaluation of thoroughness of their use [8]. There was also variation in the number and timing of assessments from which overall scores were based such that some interventions measured implementation measures at every session [43], some at regular intervals [44], or specific pre-planned sessions [45], whilst others took ratings from one or more randomly selected sessions over the course of therapy [46, 47].

The relationship between adherence and outcome was measured in 29 studies across 43 papers [7, 8, 10, 11, 12, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78]. Adherence was rated most frequently by observers (N = 13, 45%), as well as therapists (N = 4, 14%), clients (N = 4, 14%), supervisors (N = 4, 14%), or a composite of client and therapist ratings was used (N = 4, 14%). A significant relationship between adherence and at least one youth outcome was reported in 24 studies (83%), while five (17%) reported no relationship.

The relationship between competence and outcome was measured in nine studies across ten papers [11, 12, 46, 51, 53, 59, 64, 74, 79, 80]. Competence was rated mostly by observers (N = 7, 77%), and in two instances by supervisors. A significant relationship between competence and at least one youth outcome was reported in five studies (56%), while four (44%) found no relationship.

The relationship between composite fidelity and outcome was measured in five studies across seven papers [81, 82, 83, 84, 85, 86, 87]. Composite fidelity was rated mostly by observers (N = 4, 80%), and in one instance by supervisors. A significant relationship between composite fidelity and at least one youth outcome was reported in two studies (40%), while three (60%) found no relationship.

A small number of studies considered potential moderators of the strength or direction of the relationship. Sexton and Turner [49] found an interaction for youth risk, such that adherence more strongly predicted outcome in the presence of high peer risk. Schoenwald et al. [44] found an interaction between therapist adherence and organisational structure and climate, such that therapist job satisfaction predicted improvements in behaviour only when therapist adherence was low. Two studies measuring alliance found no interaction with the relationship between therapist adherence or fidelity and outcome [47, 82], whilst one study found the relationship between therapist competence and outcome became non-significant when controlling for alliance [80]. In the two studies in which it was investigated, an interaction between adherence and competence was not found in the prediction of outcomes [11, 12].

Study quality

Complete details of study quality ratings are available in Online Resource 4. 43% of studies (N = 15) reported a low rate of uptake of the intervention or research study by eligible cases, and 40% of studies (N = 14) reported risk of bias due to dropout or missing data from the study. 46% of studies (N = 16) also failed to report clear and valid selection criteria. Finally, low power or small sample size was identified in 34% of cases (N = 12).

Risk of bias in the reliability and validity of the adherence or competence measures was identified in 66% of studies (N = 23). For most studies, this risk of bias related to using a non-independent informant (i.e. therapist, client or supervisor) rather than an observer (17 studies). Where observer ratings were used, and inter-rater reliability was reported, the majority of studies met the inter-rater reliability thresholds, indicating that the reliability of observer ratings was rarely a concern. Inter-rater reliability scores for each study are presented in Online Resource 2. However, only 11% of studies (N = 4) reported outcome measures at risk of bias. Those that did use subjective ratings such as therapist reported criminal behaviour [63] or adapted forms of validated measures [84], rather than validated questionnaires.

Baseline symptom severity was controlled for in 83% of studies (N = 29), such that any relationship between implementation and outcome was independent of the influence baseline severity may have on both factors. However, 43% of studies (N = 15) failed to report consideration of any other confounding variables, and amongst those which did so there was considerable variation in controlled variables. These included demographic or treatment variables such as age or gender (N = 8), dosage, time in treatment or assessment interval (N = 5), parent marital status (N = 4), income (N = 3) and therapeutic alliance (N = 3).

Meta-analysis

Two adherence studies and one competence study with insufficient data to compute effect sizes were excluded from the analysis [71, 72, 79]. Two papers reported outcomes for two independent samples [8, 12].

The 29 adherence-outcome effect sizes ranged from − 0.070 to 0.444 (see Fig. 2), and a small but statistically significant relationship between therapist adherence and outcome was identified, r = 0.096 (95% CI = 0.058, 0.134), z = 4.938, p < 0.001 (see Table 2). Variance in effect sizes was significantly greater than would be expected by sampling error alone [Q (28) = 62.352, p < 0.001] and, therefore, likely affected by differences between studies, although no significant moderation effect was identified (see Table 3). However, consideration of individual effects for each moderator group suggested a small number of circumstances under which adherence was not significantly associated with outcome. These were youth non-CBT intervention (r = 0.006, 95% CI = − 0.145, 0.158, z = 0.082, p = 0.935), and where client informants were used to rate adherence (client informant r = 0.040, 95% CI = − 0.034, 0.113, z = 1.049, p = 0.294; client and therapist composite: r = 0.119, 95% CI = − 0.048, 0.280, z = 1.400, p = 0.162). All other moderator categories tested were significantly associated with outcome (clinical group categories: r = 0.071–0.127; intervention type categories: r = 0.089–0.169; informant categories: r = 0.105–0.148).
Fig. 2

Forest plot of adherence–outcome effect sizes

Table 2

Meta-Analysis

 

Distribution of effect size

Estimated population mean

Test of homogeneity

N

Min

Max

r

95% CI

Z

Sig.

Q

df

Sig.

Adherence

29

− 0.070

0.444

0.0964

0.0582

0.1342

4.9384

0.000

62.3516

28

0.000

Competence

9

0.000

0.173

0.0264

− 0.0199

0.0726

1.1188

0.263

2.5951

8

0.957

Composite fidelity

5

− 0.273

0.213

0.0615

− 0.0702

0.1911

0.9153

0.360

7.6997

4

0.103

Table 3

Moderation analysis

 

Clinical group

Intervention type

Informant

Q

df

Sig.

Q

df

Sig.

Q

df

Sig.

Adherence

1.4580

2

0.482

3.0602

3

0.382

3.5283

4

0.474

Competence

1.3663

2

0.505

0.7467

2

0.688

Composite fidelity

Moderator levels with only one available effect size could not be included in the moderation analysis

The nine competence-outcome effect sizes ranged from 0.000 to 0.173 (see Fig. 3); however, competence did not have a statistically significant association with outcome, r = 0.026 (95% CI = − 0.020, 0.073), z = 1.119, p = 0.263 (see Table 2). There was no significant variance in effect sizes [Q (7) = 2.595, p = 0.957] indicating the studies likely represent a common population mean. Although there were insufficient levels to test informant as a moderator, there was no significant moderation effect for clinical group or intervention modality (see Table 3).
Fig. 3

Forest plot of competence–outcome effect sizes

The five fidelity-outcome effect sizes ranged from − 0.273 to 0.213 (see Fig. 4); however, composite fidelity did not have a significant association with outcome, r = 0.06 (95% CI = − 0.070, 0.191), z = 0.9153, p = 0.360 (see Table 2). There was no significant variance in effect sizes (Q (4) = 7.700, p = 0.103), indicating the studies likely represent a common population mean, although there were insufficient levels to test any moderation analysis for this effect.
Fig. 4

Forest plot of composite fidelity–outcome effect sizes

Sensitivity analysis

Using a subsample of 22 adherence effect sizes with the lowest risk of methodological bias, the overall size of effect remained very similar to the effect seen in the main analysis when all effects were included (r = 0.097 (95% CI = 0.052, 0.141), z = 4.253, p < 0.001). Tests for heterogeneity also indicated that variance in effect sizes remained significantly greater than would be expected by sampling error alone [Q (21) = 44.579, p < 0.01], although the extent of observed variation was reduced.

Publication bias

The Begg and Mazundar [38] random effects rank correlation test (see Table 4) indicated no risk of publication bias, although the test may have lacked power to detect an effect with the relatively small sample size. Sensitivity analysis indicates that correction for moderate publication bias would reduce the strength for all effects, and correction for severe publication bias could reverse the direction (see Table 4).
Table 4

Publication bias

 

Random effects rank correlation test

Sensitivity analysis

Unadjusted estimate (r)

Adjusted estimate (r)

One-tailed

τ

Sig.

Moderate

Severe

Adherence

0.084

0.524

0.0966

0.0723

− 0.1922

Competence

0.167

0.532

0.0265

0.0146

− 0.0011

Composite fidelity

− 0.200

0.624

0.0829

0.0511

− 0.3553

Discussion

Three separate meta-analyses tested the relationship between therapist adherence, competence or composite fidelity and youth outcome in child and adolescent psychotherapy, and whether this relationship was moderated by client group, intervention modality or implementation measure informant.

A small but statistically significant relationship was identified between therapist adherence and youth outcome, suggesting that implementing the appropriate components of therapy plays a small role in child and adolescent therapeutic practice. This finding is in line with a previous qualitative review of child and adolescent mental health intervention and prevention [17], and with a meta-analysis of intervention for juvenile antisocial behaviour [19], although it appears inconsistent with the meta-analysis by Webb et al. [16]. This suggests, if taken at face value, that therapist adherence may be marginally more influential in child and family settings compared with adult individual therapy settings. However, the finding of a statistically significant association with outcome is tempered by the fact that the small effect suggests that adherence only accounts for just under one percent of variance in outcomes.

In line with the meta-analysis by Webb et al. [16], youth outcome was not significantly related to therapist competence, perhaps not surprising given the limited role that adherence was found to have. This of course does not mean that clinical competence is unrelated to outcome but simply that competence in implementing model specific skills may be less important than more general clinical skills. However, the non-significant findings may also result from limited variance in therapist competence such that competence is not often rated as low. Furthermore, the role of competence is not widely measured or reported across studies at present, limiting the extent to which this effect has been tested across settings. The present analysis also identified no relationship found between composite fidelity and outcome. This appears to be in line with the finding that therapist adherence and competence are differentially related to outcome, suggesting that it may be more informative to measure these as separate constructs. The findings relating to competence and fidelity, however, need to be viewed with considerable caution given the small number of studies in these analyses.

Although significant variance was detected in adherence effects, effect sizes were not moderated by intervention modality, clinical diagnostic group or adherence informant. This finding may suggest that the small but significant association between therapist adherence and outcome is present across diverse intervention types and client groups and is stable across adherence informants. The moderator categories where the adherence–outcome relationship was not apparent (i.e. youth non-CBT intervention, and client informant) may be an artefact of the small numbers of effect sizes within these subgroups and, therefore, reduced power to detect an effect. However, given observed variance in effects indicates that the mean effect size does not represent a single population, and the absence of moderator effects suggests that there may be other unmeasured or undetected explanations for the variance in adherence effects. Unmeasured moderators may include the time lag between the measurement of implementation and outcome, where a shorter time lag might be expected to be associated with a larger effect. However, given heterogeneity in measurement approaches observed, developing a consistent variable measuring this was out of scope for the current study. Other potential moderators of the relationship between adherence and outcome not measured in the present analysis may include client characteristics, such as likelihood to respond to intervention [88].

The finding of a statistically significant association between therapist adherence and child and adolescent psychotherapy outcomes is in line with the perspective of proponents of ESTs, as this suggests that maintaining a certain level of adherence may be an important component of implementing manualised interventions. On the other hand, the small size of effect is in line with the view by proponents of common factors in psychotherapy that specific therapy model factors contribute relatively little to treatment outcome [89], which is likely to be more strongly associated with a number of factors other than adherence such as client characteristics, e.g. baseline problem severity [90], or therapeutic processes such as therapeutic alliance [91].

There are a number of reasons why caution is needed before drawing conclusions about the relative contribution of model specific and common psychotherapy factors in child outcomes. First and foremost, the relative role of these factors was not the focus of the present study. In addition, most existing studies are not generally designed to test the role of adherence as their primary hypothesis. Further, for most included studies many of the intervention non-specific processes known to be associated with outcome, such as therapeutic alliance [16], were not controlled for and as such may confound any observed relationship between adherence and outcome [80]. In addition to this, common factors such as therapeutic alliance which are seen to be associated with outcome in wider research are also often observed to be small in size [92]. This suggests that common and specific factors may both play small but significant roles in contributing to outcomes. However, while small, mean effects of therapeutic alliance are still often slightly larger than the mean effect size observed in the present study between adherence and outcome.

The small effect seen in the present study may also relate to methodological bias in the included studies which may have led to an under or over-estimation of the relationship between adherence and outcome. In the absence of consistent guidelines there is variation in the methods, reliability and validity of implementation measures varies across studies. For example, adherence was often rated by therapist or clients rather than independent observers and where independent observers were used, inter-rater agreement was not always well established. Studies which measure average implementation or change over time may also provide more accurate evidence than those for which ratings are based on a single session. A sensitivity analysis showed that when excluding studies at highest risk of bias the overall association between adherence and outcome remained a similar size. This indicated that overall study quality was unlikely to be driving the observed effect, although heterogeneity was found to be reduced indicating that quality may contributing at least in part to the variance in effects. Given this, improved reliability and consistency in therapist adherence measures could still further strengthen the reliability of conclusions.

Small effects may also be due to minimal variance in therapist adherence reported in a number of studies, given that therapist adherence was often reported to be moderate to high, thereby reducing power to detect the effect of low adherence. A threshold level may also exist whereby therapist adherence becomes sufficient to achieve optimal outcomes but is no more effective as it increases past this point. For example, the review by Durlak and DuPre [93] suggested that once a sufficient threshold of around 60–80% implementation had been reached, further improvement did not necessarily lead to significantly better outcomes. Furthermore, there may be specific components of adherence in some interventions which are more strongly related to improved outcome than others. For example, in supportive-expressive group counselling, Shechtman and Leichtentritt [45] found that while therapist adherence to intervention components involving encouragement and self-disclosure improved anxiety and social competence, respectively, one component of adherence, challenge, had a negative impact on youth aggression and academic outcome. This would suggest that not all aspects of a given model may be linked to improved youth outcome and a composite measure may be masking effects.

Limitations

The non-significant moderator effects should be interpreted with caution given the heterogeneous nature of the moderator categories and small number of sample sizes in each. Up to ten effect sizes per moderation category has been indicated as an ideal target [32], which is well above the numbers of studies available for the present analyses. Given the small number of included studies, in order to achieve a reasonable number of studies in each category, the categories of intervention type into which studies were classified were relatively heterogeneous. For example there were insufficient studies to use separate categories for anxiety or depressive disorders, which were instead combined into a broad category of emotional disorder. Use of heterogeneous moderator categories was necessary to allow a moderator analysis to be conducted, but may have limited the validity of the moderator categories used in this analysis. Expansion of the evidence base to increase the number of studies and to enable more precise categorisations would improve the reliability and sensitivity of the test for moderation, allowing improved understanding of the reasons for the observed variation.

This meta-analysis was also unable to test for a number of other aspects of the relationship between therapist implementation and outcome, given the small number of studies in which they were measured. This includes the potential curvilinear relationship between adherence and outcome, such that moderate therapist adherence may be preferable to both very high and very low adherence. Furthermore, given that few studies of child and adolescent psychotherapy have considered the interaction between adherence and therapeutic alliance or competence, the present analysis was not able to consider the role of these moderators in the relationship between adherence and outcome.

Unpublished works such as dissertations were excluded from this meta-analysis on the basis of limited full text availability. Although findings indicated no risk of publication bias, the small sample may affect the reliability of this conclusion, and findings from the sensitivity analysis suggest that correction for publication bias could invalidate the present conclusions. In line with this, Webb et al. [16], who found no overall relationship of therapist adherence and competence with outcome, did include unpublished dissertations. Systematic measurement and reporting of implementation constructs in published research should be encouraged in order that this question can be more rigorously tested. Another limitation in the search approach was restricting the search to English language only, which could have excluded potentially relevant studies. A further limitation of the search approach was that the searches, data extraction and quality ratings were carried out by the lead author only. It was, therefore, not possible to test for inter-rater agreement on these judgements. Finally, conservatively estimating zero for non-significant effects with insufficient data to compute an effect size, to avoid excluding non-significant findings and over-estimating effects, may lead to under estimation of effects if these studies had been under powered to detect associations.

Conclusion

This study, aiming to understand whether treatment outcomes can be accounted for by therapist adherence to the treatment model or competence in delivery, extends the findings of existing reviews in the field of child and adolescent intervention through inclusion of a broader and more extensive range of studies and use of a more rigorous statistical analysis of findings. The findings indicate that there may be value added in using training and supervision as strategies to ensure the successful transportation and implementation of manualised intervention models across settings. However, the findings also indicate that measures to ensure adherence would only contribute to a small proportion of outcome variance and that there are, therefore, likely to be other intervention non-specific characteristics which may play an equally or more important role than therapist adherence in achieving therapeutic outcomes. These non-specific factors may include client motivation, severity of the presenting problem, the level of therapists’ experience and knowledge of the specific problem or therapeutic alliance. This may suggest the importance of considering both intervention-specific and non-specific factors to achieve outcome in child and adolescent psychotherapy.

Although observed heterogeneity indicates that the adherence–outcome relationship is not universal, this study was unable to identify the moderating factors explaining the observed variation in adherence–outcome effect sizes. As such, future research should aim to further consider the circumstances under which the relationship between adherence and outcome is observed in everyday clinical practice. More consistent reporting of adherence in intervention research, using robust observer-rated adherence measures, would ensure sufficient numbers of valid and reliable effects to enable robust tests for moderation through meta-analysis. Further, factors such as youth risk, therapeutic alliance and competence, which may potentially interact with adherence effects, should be more consistently measured and controlled for to enable an understanding of the adherence–outcome relation over and above the influence of these factors.

Ultimately, what is needed are studies that are specifically designed to evaluate the role of adherence and competence in everyday clinical practice where there is likely to be a greater variability both in the way the treatments are delivered and in factors that may moderate the impact of adherence and competence.

Notes

Compliance with ethical standards

Conflict of interest

On behalf of all the authors, the corresponding author states that there is no conflict of interest.

Ethical standards

The manuscript is based on secondary analysis of published data only, and does not contain primary analysis of any clinical study or patient data.

Supplementary material

787_2018_1265_MOESM1_ESM.pdf (117 kb)
Supplementary material 1 (pdf 117 kb)
787_2018_1265_MOESM2_ESM.pdf (231 kb)
Supplementary material 2 (pdf 230 kb)
787_2018_1265_MOESM3_ESM.pdf (179 kb)
Supplementary material 3 (pdf 178 kb)
787_2018_1265_MOESM4_ESM.pdf (172 kb)
Supplementary material 4 (pdf 171 kb)

References

  1. 1.
    Breitenstein SM, Gross D, Garvey CA et al (2010) Implementation fidelity in community-based interventions. Res Nurs Health 33:164–173PubMedPubMedCentralGoogle Scholar
  2. 2.
    Perepletchikova F, Kazdin AE (2005) Treatment integrity and therapeutic change: issues and research recommendations. Clin Psychol Sci Pract 12:365–383CrossRefGoogle Scholar
  3. 3.
    McHugh RK, Murray HW, Barlow DH (2009) Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventions. Behav Res Ther 47:946–953CrossRefGoogle Scholar
  4. 4.
    Westen D, Novotny CM, Thompson-Brenner H (2004) The empirical status of empirically supported psychotherapies: assumptions, findings, and reporting in controlled clinical trials. Psychol Bull 130:631–663.  https://doi.org/10.1037/0033-2909.130.4.631 CrossRefPubMedGoogle Scholar
  5. 5.
    Sackett DL, Rosenberg WMC, Gray JAM et al (1996) Evidence based medicine: what it is and what it isn’t. BMJ.  https://doi.org/10.1136/bmj.312.7023.71 CrossRefPubMedPubMedCentralGoogle Scholar
  6. 6.
    Schoenwald SK, Halliday-Boykins C, Henggeler SW (2003) Client-level predictors of adherence to MST in community service settings. Fam Process 42:345–359CrossRefGoogle Scholar
  7. 7.
    Halliday-Boykins CA, Schoenwald SK, Letourneau EJ (2005) Caregiver-therapist ethnic similarity predicts youth outcomes from an empirically based treatment. J Consult Clin Psychol 73:808–818.  https://doi.org/10.1037/0022-006X.73.5.808 CrossRefPubMedGoogle Scholar
  8. 8.
    Rowe C, Rigter H, Henderson C et al (2013) Implementation fidelity of multidimensional family therapy in an international trial. J Subst Abuse Treat 44:391–399.  https://doi.org/10.1016/j.jsat.2012.08.225 CrossRefPubMedGoogle Scholar
  9. 9.
    Huppert JD, Barlow DH, Gorman JM et al (2006) The interaction of motivation and therapist adherence predicts outcome in cognitive behavioral therapy for panic disorder: preliminary findings. Cogn Behav Pract 13:198–204CrossRefGoogle Scholar
  10. 10.
    Henggeler SW, Melton GB, Brondino MJ et al (1997) Multisystemic therapy with violent and chronic juvenile offenders and their families: the role of treatment fidelity in successful dissemination. J Consult Clin Psychol 65:821–833.  https://doi.org/10.1037/0022-006X.65.5.821 CrossRefPubMedGoogle Scholar
  11. 11.
    McCambridge J, Day M, Thomas BA, Strang J (2011) Fidelity to motivational interviewing and subsequent cannabis cessation among adolescents. Addict Behav 36:749–754.  https://doi.org/10.1016/j.addbeh.2011.03.002 CrossRefPubMedGoogle Scholar
  12. 12.
    Hogue A, Henderson CE, Dauber S et al (2008) Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. J Consult Clin Psychol 76:544–555.  https://doi.org/10.1037/0022-006X.76.4.544 CrossRefPubMedPubMedCentralGoogle Scholar
  13. 13.
    Peavy KM, Guydish J, Manuel JK et al (2014) Treatment adherence and competency ratings among therapists, supervisors, study-related raters and external raters in a clinical trial of a 12-step facilitation for stimulant users. J Subst Abus Treat 47:222–228CrossRefGoogle Scholar
  14. 14.
    Barber JP, Gallop R, Crits-Christoph P et al (2006) The role of therapist adherence, therapist competence, and alliance in predicting outcome of individual drug counseling: results from the National Institute Drug Abuse Collaborative Cocaine Treatment study. Psychother Res 16:229–240CrossRefGoogle Scholar
  15. 15.
    Castonguay LG (2011) Psychotherapy, psychopathology, research and practice: pathways of connections and integration. Psychother Res 21:125–140CrossRefGoogle Scholar
  16. 16.
    Webb CA, DeRubeis RJ, Barber JP (2010) Therapist adherence/competence and treatment outcome: a meta-analytic review. J Consult Clin Psychol 78:200CrossRefGoogle Scholar
  17. 17.
    Novins DK, Green AE, Legha RK, Aarons GA (2013) Dissemination and implementation of evidence-based practices for child and adolescent mental health: a systematic review. J Am Acad Child Adolesc Psychiatry 52:1009–1025CrossRefGoogle Scholar
  18. 18.
    Rapley HA, Loades ME (2018) A systematic review exploring therapist competence, adherence, and therapy outcomes in individual CBT for children and young people. Psychother Res.  https://doi.org/10.1080/10503307.2018.1464681
  19. 19.
    Goense P, Assink M, Stams GJ, Boendermaker L, Hoeve M (2016) Making “what works” work: a meta-analytic study of the effect of treatment integrity on outcomes of evidence-based interventions for juveniles with antisocial behavior. Aggress Violent Behav 31:106–115CrossRefGoogle Scholar
  20. 20.
    Liberati A, Altman DG, Tetzlaff J et al (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med 151:W-65–W-94CrossRefGoogle Scholar
  21. 21.
    Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 151:264–269CrossRefGoogle Scholar
  22. 22.
    Porter E, Chambless DL (2015) A systematic review of predictors and moderators of improvement in cognitive-behavioral therapy for panic disorder and agoraphobia. Clin Psychol Rev 42:179–192CrossRefGoogle Scholar
  23. 23.
    Priebe S, Richardson M, Cooney M et al (2011) Does the therapeutic relationship predict outcomes of psychiatric treatment in patients with psychosis? A systematic review. Psychother Psychosom 80:70–77CrossRefGoogle Scholar
  24. 24.
    Barnicot K, Katsakou C, Bhatti N et al (2012) Factors predicting the outcome of psychotherapy for borderline personality disorder: a systematic review. Clin Psychol Rev 32:400–412CrossRefGoogle Scholar
  25. 25.
    Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Elsevier, HillsdaleGoogle Scholar
  26. 26.
    Green SB (1991) How many subjects does it take to do a regression analysis. Multivar Behav Res 26:499–510CrossRefGoogle Scholar
  27. 27.
    Cicchetti DV (1994) Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 6:284CrossRefGoogle Scholar
  28. 28.
    McHugh ML (2012) Interrater reliability: the kappa statistic. Biochem medica 22:276–282CrossRefGoogle Scholar
  29. 29.
    Bajpai RC, Chaturvedi HK (2015) Evaluation of inter-rater agreement and inter-rater reliability for observational data: An overview of concepts and methods. J Indian Acad Appl Psychol 41:20Google Scholar
  30. 30.
    Swinscow TDV, Campbell MJ (2002) Statistics at square one. Wiley, LondonGoogle Scholar
  31. 31.
    Scammacca N, Roberts G, Stuebing KK (2014) Meta-analysis with complex research designs. Rev Educ Res 84:328–364.  https://doi.org/10.3102/0034654313500826 CrossRefPubMedGoogle Scholar
  32. 32.
    Borenstein M, Hedges LV, Higgins JPT, Rothstein HR (2009) Introduction to meta-analysis. Wiley, ChichesterCrossRefGoogle Scholar
  33. 33.
    Lenhard W, Lenhard A (2016) Calculation of effect sizes. Psychometrica. http://www.psychometrica.de/effect_size.html. Accessed 21 July 2018
  34. 34.
    Peterson RA, Brown SP (2005) On the use of beta coefficients in meta-analysis. J Appl Psychol 90:175CrossRefGoogle Scholar
  35. 35.
    IBM (2017) IBM SPSS Statistics for Windows, Version 25.0Google Scholar
  36. 36.
    Lipsey MW, Wilson DB (2001) Practical meta-analysis. Sage Publications, Thousand OaksGoogle Scholar
  37. 37.
    Hedges LV, Vevea JL (1998) Fixed-and random-effects models in meta-analysis. Psychol Methods 3:486CrossRefGoogle Scholar
  38. 38.
    Begg CB, Mazumdar M (1994) Operating characteristics of a rank correlation test for publication bias. Biometrics 50:1088–1101CrossRefGoogle Scholar
  39. 39.
    Field AP, Gillett R (2010) How to do a meta-analysis. Br J Math Stat Psychol 63:665–694CrossRefGoogle Scholar
  40. 40.
    Vevea JL, Woods CM (2005) Publication bias in research synthesis: sensitivity analysis using a priori weight functions. Psychol Methods 10:428CrossRefGoogle Scholar
  41. 41.
    Hogue A, Dauber S, Samuolis J, Liddle HA (2006) Treatment techniques and outcomes in multidimensional family therapy for adolescent behavior problems. J Fam Psychol 20:535CrossRefGoogle Scholar
  42. 42.
    Robbins MS, Feaster DJ, Horigian VE et al (2011) Therapist adherence in brief strategic family therapy for adolescent drug abusers. J Consult Clin Psychol 79:43–53.  https://doi.org/10.1037/a0022146 CrossRefPubMedPubMedCentralGoogle Scholar
  43. 43.
    Garner BR, Godley SH, Funk RR et al (2009) Exposure to adolescent community reinforcement approach treatment procedures as a mediator of the relationship between adolescent substance abuse treatment retention and outcome. J Subst Abuse Treat 36:252–264.  https://doi.org/10.1016/j.jsat.2008.06.007 CrossRefPubMedGoogle Scholar
  44. 44.
    Schoenwald SK, Sheidow AJ, Letourneau EJ, Liao JG (2003) Transportability of multisystemic therapy: evidence for multilevel influences. Ment Health Serv Res 5:223–239.  https://doi.org/10.1023/A:1026229102151 CrossRefPubMedGoogle Scholar
  45. 45.
    Shechtman Z, Leichtentritt J (2010) The association of process with outcomes in child group therapy. Psychother Res 20:8–21.  https://doi.org/10.1080/10503300902926562 CrossRefPubMedGoogle Scholar
  46. 46.
    Ginsburg GS, Becker KD, Drazdowski TK, Tein J-Y (2012) Treating anxiety disorders in inner city schools: results from a pilot randomized controlled trial comparing CBT and usual care. Child Youth Care Forum 41:1–19.  https://doi.org/10.1007/s10566-011-9156-4 CrossRefPubMedPubMedCentralGoogle Scholar
  47. 47.
    Liber JM, McLeod BD, Van Widenfelt BM et al (2010) Examining the relation between the therapeutic alliance, treatment adherence, and outcome of cognitive behavioral therapy for children with anxiety disorders. Behav Ther 41:172–186.  https://doi.org/10.1016/j.beth.2009.02.003 CrossRefPubMedGoogle Scholar
  48. 48.
    Schoenwald SK, Carter RE, Chapman JE, Sheidow AJ (2008) Therapist adherence and organizational effects on change in youth behavior problems one year after multisystemic therapy. Adm Policy Ment Heal Ment Heal Serv Res 35:379–394.  https://doi.org/10.1007/s10488-008-0181-z CrossRefGoogle Scholar
  49. 49.
    Sexton T, Turner CW (2010) The effectiveness of functional family therapy for youth with behavioral problems in a community practice setting. J Fam Psychol 24:339–348.  https://doi.org/10.1037/a0019406 CrossRefPubMedPubMedCentralGoogle Scholar
  50. 50.
    Al CM, Stams GJJ, Asscher JJ, van der Laan PH (2014) A programme evaluation of the Family Crisis Intervention Program (FCIP): relating programme characteristics to change. Child Fam Soc Work 19:225–236.  https://doi.org/10.1111/j.1365-2206.2012.00896.x CrossRefGoogle Scholar
  51. 51.
    Bloomquist ML, August GJ, Lee SS et al (2013) Going-to-scale with the early risers conduct problems prevention program: use of a comprehensive implementation support (CIS) system to optimize fidelity, participation and child outcomes. Eval Program Plann 38:19–27.  https://doi.org/10.1016/j.evalprogplan.2012.11.001 CrossRefPubMedGoogle Scholar
  52. 52.
    Dagenais C, Briere FN, Gratton G, Dupont D (2009) Brief and intensive family support program to prevent emergency placements: lessons learned from a process evaluation. Child Youth Serv Rev 31:594–600.  https://doi.org/10.1016/j.childyouth.2008.11.001 CrossRefGoogle Scholar
  53. 53.
    Garner BR, Godley SH, Dennis ML et al (2012) Using pay for performance to improve treatment implementation for adolescent substance use disorders: results from a cluster randomized trial. Arch Pediatr Adolesc Med 166:938–944.  https://doi.org/10.1001/archpediatrics.2012.802 CrossRefPubMedGoogle Scholar
  54. 54.
    Gillespie ML, Huey SJ, Cunningham PB (2017) Predictive validity of an observer-rated adherence protocol for multisystemic therapy with juvenile drug offenders. J Subst Abuse Treat 76:1–10.  https://doi.org/10.1016/j.jsat.2017.01.001 CrossRefPubMedGoogle Scholar
  55. 55.
    Gillham JE, Hamilton J, Freres DR et al (2006) Preventing depression among early adolescents in the primary care setting: a randomized controlled study of the penn resiliency program. J Abnorm Child Psychol 34:203–219.  https://doi.org/10.1007/s10802-005-9014-7 CrossRefPubMedGoogle Scholar
  56. 56.
    Graham C, Carr A, Rooney B et al (2014) Evaluation of functional family therapy in an Irish context. J Fam Ther 36:20–38.  https://doi.org/10.1111/1467-6427.12028 CrossRefGoogle Scholar
  57. 57.
    Hartnett D, Carr A, Sexton T (2016) The effectiveness of functional family therapy in reducing adolescent mental health risk and family adjustment difficulties in an Irish context. Fam Process 55:287–304.  https://doi.org/10.1111/famp.12195 CrossRefPubMedGoogle Scholar
  58. 58.
    Henggeler SW, Pickrel SG, Brondino MJ (1999) Multisystemic treatment of substance-abusing and dependent delinquents: outcomes, treatment fidelity, and transportability. Ment Health Serv Res 1:171–184CrossRefGoogle Scholar
  59. 59.
    Heywood C, Fergusson D (2016) A pilot study of functional family therapy in New Zealand. NZ J Psychol 45:12–22Google Scholar
  60. 60.
    Holth P, Torsheim T, Sheidow A et al (2011) Intensive quality assurance of therapist adherence to behavioral interventions for adolescent substance use problems. J Child Adolesc Subst Abuse 20:289–313.  https://doi.org/10.1080/1067828X.2011.581974 CrossRefPubMedPubMedCentralGoogle Scholar
  61. 61.
    Lange AM, van der Rijken RE, Busschbach JJ et al (2017) It’s not just the therapist: therapist and country-wide experience predict therapist adherence and adolescent outcome. Child Youth Care Forum 46:455–471.  https://doi.org/10.1007/s10566-016-9388-4 CrossRefPubMedPubMedCentralGoogle Scholar
  62. 62.
    Overbeek MM, de Schipper J, Lamers-Winkelman F, Schuengel C (2013) Effectiveness of specific factors in community-based intervention for child-witnesses of interparental violence: a randomized trial. Child Abuse Negl 37:1202–1214.  https://doi.org/10.1016/j.chiabu.2013.07.007 CrossRefPubMedGoogle Scholar
  63. 63.
    Lofholm CA, Eichas K, Sundell K (2014) The Swedish implementation of multisystemic therapy for adolescents: does treatment experience predict treatment adherence? J Clin Child Adolesc Psychol 43:643–655.  https://doi.org/10.1080/15374416.2014.883926 CrossRefPubMedGoogle Scholar
  64. 64.
    Podell JL, Kendall PC, Gosch EA et al (2013) Therapist factors and outcomes in CBT for anxiety in youth. Prof Psychol Res Pract 44:89–98.  https://doi.org/10.1037/a0031700 CrossRefGoogle Scholar
  65. 65.
    Strauss K, Vicari S, Valeri G et al (2012) Parent inclusion in early intensive behavioral intervention: the influence of parental stress, parent treatment fidelity and parent-mediated generalization of behavior targets on child outcomes. Res Dev Disabil 33:688–703.  https://doi.org/10.1016/j.ridd.2011.11.008 CrossRefPubMedGoogle Scholar
  66. 66.
    Sundell K, Hansson K, Lofholm CA et al (2008) The transportability of multisystemic therapy to Sweden: short-term results from a randomized trial of conduct-disordered youths. J Fam Psychol 22:550–560.  https://doi.org/10.1037/a0012790 CrossRefPubMedGoogle Scholar
  67. 67.
    Hogue A, Liddle HA, Dauber S, Samuolis J (2004) Linking session focus to treatment outcome in evidence-based treatments for adolescent substance abuse. Psychother Theory Res Pract Train 41:83–96.  https://doi.org/10.1037/0033-3204.41.2.83 CrossRefGoogle Scholar
  68. 68.
    Schoenwald SK, Chapman JE, Sheidow AJ, Carter RE (2009) Long-term youth criminal outcomes in MST transport: the impact of therapist adherence and organizational climate and structure. J Clin Child Adolesc Psychol 38:91–105.  https://doi.org/10.1080/15374410802575388 CrossRefPubMedPubMedCentralGoogle Scholar
  69. 69.
    Schoenwald SK, Sheidow AJ, Chapman JE (2009) Clinical supervision in treatment transport: effects on adherence and outcomes. J Consult Clin Psychol 77:410–421.  https://doi.org/10.1037/a0013788 CrossRefPubMedPubMedCentralGoogle Scholar
  70. 70.
    Chapman JE, Schoenwald SK (2011) Ethnic similarity, therapist adherence, and long-term multisystemic therapy outcomes. J Emot Behav Disord 19:3–16.  https://doi.org/10.1177/1063426610376773 CrossRefPubMedGoogle Scholar
  71. 71.
    Eames C, Daley D, Hutchings J et al (2009) Treatment fidelity as a predictor of behaviour change in parents attending group-based parent training. Child Care Health Dev 35:603–612.  https://doi.org/10.1111/j.1365-2214.2009.00975.x CrossRefPubMedGoogle Scholar
  72. 72.
    Williams NJ, Green P (2012) Reliability and validity of a treatment adherence measure for child psychiatric rehabilitation. Psychiatr Rehabil J 35:369–375.  https://doi.org/10.1037/h0094495 CrossRefPubMedGoogle Scholar
  73. 73.
    Eames C, Daley D, Hutchings J et al (2010) The impact of group leaders’ behaviour on parents acquisition of key parenting skills during parent training. Behav Res Ther 48:1221–1226.  https://doi.org/10.1016/j.brat.2010.07.011 CrossRefPubMedGoogle Scholar
  74. 74.
    Campos-Melady M, Smith JE, Meyers RJ et al (2017) The effect of therapists’ adherence and competence in delivering the adolescent community reinforcement approach on client outcomes. Psychol Addict Behav 31:117–129.  https://doi.org/10.1037/adb0000216 CrossRefPubMedGoogle Scholar
  75. 75.
    Becker EM, Becker KD, Ginsburg GS (2012) Modular cognitive behavioral therapy for youth with anxiety disorders: a closer look at the use of specific modules and their relation to treatment process and response. School Ment Health 4:243–253.  https://doi.org/10.1007/s12310-012-9080-2 CrossRefGoogle Scholar
  76. 76.
    Schoenwald SK, Henggeler SW, Brondino MJ, Rowland MD (2000) Multisystemic therapy: monitoring treatment fidelity. Fam Process 39:83–103.  https://doi.org/10.1111/j.1545-5300.2000.39109.x CrossRefPubMedGoogle Scholar
  77. 77.
    Huey SJ Jr, Henggeler SW, Brondino MJ, Pickrel SG (2000) Mechanisms of change in multisystemic therapy: reducing delinquent behavior through therapist adherence and improved family and peer functioning. J Consult Clin Psychol 68:451–467.  https://doi.org/10.1037/0022-006X.68.3.451 CrossRefPubMedGoogle Scholar
  78. 78.
    Lange AMC, van der Rijken REA, Delsing MJMH, et al (2018) Development of therapist adherence in relation to treatment outcomes of adolescents with behavioral problems. J Clin Child Adolesc Psychol.  https://doi.org/10.1080/15374416.2018.1477049
  79. 79.
    Strauss K, Benvenuto A, Battan B et al (2015) Promoting shared decision making to strengthen outcome of young children with autism spectrum disorders: the role of staff competence. Res Dev Disabil 38:48–63.  https://doi.org/10.1016/j.ridd.2014.11.016 CrossRefPubMedGoogle Scholar
  80. 80.
    Boyer B, MacKay KJ, McLeod BD, Van der Oord S (2018) Comparing alliance in two cognitive-behavioural therapies for adolescents with ADHD using a randomized controlled trial. Behav Ther 49(5):781–795CrossRefGoogle Scholar
  81. 81.
    Eisen JC, Marko-Holguin M, Fogel J et al (2013) Pilot study of implementation of an internet-based depression prevention intervention (CATCH-IT) for adolescents in 12 US primary care practices: Clinical and management/organizational behavioral perspectives. Prim Care Companion J Clin Psychiatry.  https://doi.org/10.4088/PCC.10m01065 CrossRefGoogle Scholar
  82. 82.
    Hukkelberg SS, Ogden T (2013) Working alliance and treatment fidelity as predictors of externalizing problem behaviors in parent management training. J Consult Clin Psychol 81:1010–1020.  https://doi.org/10.1037/a0033825 CrossRefPubMedGoogle Scholar
  83. 83.
    Smith JD, Dishion TJ, Shaw DS, Wilson MN (2013) Indirect effects of fidelity to the family check-up on changes in parenting and early childhood problem behaviors. J Consult Clin Psychol 81:962–974.  https://doi.org/10.1037/a0033950 CrossRefPubMedGoogle Scholar
  84. 84.
    Helmond P, Overbeek G, Brugman D (2012) Program integrity and effectiveness of a cognitive behavioral intervention for incarcerated youth on cognitive distortions, social skills, and moral development. Child Youth Serv Rev 34:1720–1728.  https://doi.org/10.1016/j.childyouth.2012.05.001 CrossRefGoogle Scholar
  85. 85.
    Maaskant AM, van Rooij FB, Overbeek GJ et al (2016) Parent training in foster families with children with behavior problems: follow-up results from a randomized controlled trial. Child Youth Serv Rev 70:84–94.  https://doi.org/10.1016/j.childyouth.2016.09.005 CrossRefGoogle Scholar
  86. 86.
    Forgatch MS, DeGarmo DS (2011) Sustaining fidelity following the nationwide PMTOTM implementation in Norway. Prev Sci 12:235–246.  https://doi.org/10.1007/s11121-011-0225-6 CrossRefPubMedPubMedCentralGoogle Scholar
  87. 87.
    Chiapa A, Smith JD, Kim H et al (2015) The trajectory of fidelity in a multiyear trial of the family check-up predicts change in child problem behavior. J Consult Clin Psychol 83:1006–1011.  https://doi.org/10.1037/ccp0000034 CrossRefPubMedPubMedCentralGoogle Scholar
  88. 88.
    DeRubeis RJ, Gelfand LA, German RE et al (2014) Understanding processes of change: how some patients reveal more than others—and some groups of therapists less—about what matters in psychotherapy. Psychother Res 24:419–428.  https://doi.org/10.1080/10503307.2013.838654 CrossRefPubMedGoogle Scholar
  89. 89.
    Laska KM, Gurman AS, Wampold BE (2014) Expanding the lens of evidence-based practice in psychotherapy: a common factors perspective. Psychotherapy 51:467CrossRefGoogle Scholar
  90. 90.
    Kazdin AE, Mazurick J, Siegel T (1994) Treatment outcome among children with externalizing disorder who terminate prematurely versus those who complete psychotherapy. J Am Acad Child Adolesc Psychiatry 33:549–557CrossRefGoogle Scholar
  91. 91.
    Kazdin AE, Durbin KA (2012) Predictors of child-therapist alliance in cognitive-behavioral treatment of children referred for oppositional and antisocial behavior. Psychother 49:202–217.  https://doi.org/10.1037/a0027933 CrossRefGoogle Scholar
  92. 92.
    Murphy R, Hutton P (2018) Practitioner review: therapist variability, patient-reported therapeutic alliance, and clinical outcomes in adolescents undergoing mental health treatment - a systematic review and meta-analysis. J Child Psychol Psychiatry 59:5–19.  https://doi.org/10.1111/jcpp.12767 CrossRefPubMedGoogle Scholar
  93. 93.
    Durlak JA, DuPre EP (2008) Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol 41:327–350CrossRefGoogle Scholar

Copyright information

© The Author(s) 2019

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Institute of Psychiatry, Psychology and NeuroscienceKing’s College LondonLondonUK

Personalised recommendations