INTRODUCTION

Technological innovations over the past decade have steadily reduced barriers to accessing healthcare1 both in the USA2 and internationally.3 Telemedicine holds the potential for patients to seek medical expertise more efficiently, reducing wait times and allowing specialists to direct their attention to individuals with the greatest health needs, regardless of geographic location.4 This is particularly true for the expertise of specialists, which may be unevenly distributed.5

Several mechanisms have arisen to enable increased access to specialist care, including e-consultations that allow specialists to consult remotely.6 However, with a growing physician and nursing shortage in settings ranging from the USA7 to large parts of sub-Saharan Africa,8 there is a fundamental need to equip front-line providers in rural areas with the specialized skills necessary to address community needs themselves. Such capacity-building is particularly relevant in the context of escalating health epidemics, such as the US opioid crisis or recent Ebola epidemic in West Africa, and for tackling increasingly common conditions like hepatitis C.

In 2016, the US Congress passed the ECHO (Expanding Capacity for Health Outcomes) Act, which aims to support and promote “technology-enabled collaborative learning and capacity building models.9” Several such models have been developed, the most ubiquitous being Project ECHO (Extension for Community Healthcare Outcomes).10 Project ECHO involves pairing front-line clinicians, typically located in underserved areas, with specialist mentors at academic medical centers, or “hubs”, using videoconference and a case-based mode of pedagogy. Launched in 2003 out of the University of New Mexico to increase access to hepatitis C treatment in parts of the rural southwest, the program now operates at more than over 100 academic and medical hubs across 48 states as well as multiple continents, and covers dozens of disease states and health conditions.11

While Project ECHO has been successful at expanding its scope and scale, there remains a paucity of evidence regarding the impact of ECHO and ECHO-like models (EELM) on provider- and patient-related outcomes. An investigation of the evidence is particularly warranted, given the extent of human and financial capital invested in this model: thousands of trainers and trainees, and millions of dollars in financial support. While an earlier review examined the impact of ECHO through the middle of 2015,12 this was prior to the ECHO Act and any experimental evidence, and did not extend beyond ECHO-affiliated programs. We present a systematic review of EELM that comprises peer-reviewed evidence of patient and provider outcomes between 2007 and 2018. We follow Cochrane Collaboration’s GRADE framework13 to examine the strength of evidence (SOE) and use this review as a basis for highlighting potential next steps and future directions.

METHODS

Data Sources and Searches

We reviewed academic literature in accordance with PRISMA guidelines,14 targeting publications that evaluated EELM (2007–2018). In consultation with public health experts in the field of telehealth, we established an operational definition for EELM as “a technology-enabled educational model, in which a mentor with specialized knowledge provides interactive and case-based guidance to a group of mentees for the purpose of strengthening their skills and knowledge to provide high-quality healthcare.” We delimited our search according to six inclusion criteria: (1) using a technology-enabling platform, (2) having a health-focused objective, (3) leveraging specialists to train generalists, (4) using interactive mentorship, (5) using case-based learning, and (6) implementing a hub-spoke framework rather than 1:1 learning.

We implemented a Boolean search procedure based on key words defined under three domains: (i) a technology-enabling component, (ii) involvement of health providers, and/or (iii) terms denoting resource or geographic barriers, which EELM often address. As a complementary strategy, we searched for ECHO-specific terminology linked by “or” statements. A detailed list of search terms can be found in Appendix Table 1 (online). A total of six databases were searched: PubMed, Embase, PsycINFO, Google Scholar, the Cochrane Central Register (CENTRAL), and Scopus. Google Scholar was limited to the first 100 returns. Using seminal articles, including a 2016 review by Zhou and colleagues,12 we also examined bibliographies.

Study Selection

We limited results to peer-reviewed articles reporting provider- or patient-related outcomes published in English between January 1, 2007 and December 1, 2018, including articles originating outside the USA. Returns were screened independently by two research team members for agreement with the six inclusion criteria. For situations in which agreement with criteria was unclear from the title and abstract, the full text was reviewed. Records that met inclusion criteria were flagged for full data abstraction (see Fig. 1). In the event a discrepancy arose, additional members of the research team were consulted.

Figure 1
figure 1

Study flow diagram.

Full size image

Data Extraction and Quality Assessment

Articles meeting inclusion criteria were independently entered by an investigator into a data abstraction form. A second investigator was then tasked with reviewing abstracted data to ensure accuracy and completeness. Summaries of abstracted data can be found in Tables 1 and 2.

Table 1 Included Studies Reporting Provider-Related Outcomes
Full size table
Table 2 Included Studies Reporting Patient-Related Outcomes
Full size table

Data Synthesis and Analysis

We selected key features from each study to review and summarize, based on a hierarchy of evidence, with highest quality evidence the focus of synthesis. Provider-related outcomes included participation, satisfaction, knowledge, self-efficacy, and behavior change, following a core competencies model for implementation science66 articulated by the National Implementation Research Network.67

Patient-related outcomes were grouped according to health condition and involved careful examination of health condition-specific outcomes as stated in the literature. These were classified as either process or outcome measures, according to US Agency for Healthcare Research and Quality68 definitions. Whenever possible, we report summary statistics—including means, standard deviations, odds ratios (ORs), and hazards ratios (HRs). We also report p values and 95% confidence intervals.

We rated SOE according to Cochrane GRADE criteria,13 following a two-step process. First, two research team members independently assigned a score to each article for the outcomes presented within, based on six GRADE characteristics: study design, risk of bias, inconsistency, indirectness, imprecision, and publication bias. The full research team collectively reviewed each score. Second, the research team deliberated weight of evidence across individual studies for each patient- and provider-related outcome. This systematic classification process is outlined in detail on the Cochrane website and involves, for example, evaluating the quantity of experimental versus observational evidence, and studying effect sizes and dose responses. SOE is assigned an ordinal score: very low (+), low (++), medium (+++), high (++++) (Table 3).

Table 3 Summary of SOE Scores for Patient- and Provider-Related Outcomes
Full size table

Role of the Funder

This investigation was supported by the Office of the Assistant Secretary for Planning and Evaluation (ASPE), within the US Department of Health and Human Services (HHS).69 In accordance with the ECHO Act of 2016, this investigation was commissioned as part of a Report to US Congress, released in 2019. Two staff members from ASPE are co-authors; the staff contributed to the selection of terms for the literature search and the criteria for study eligibility and provided edits of the final manuscript.

RESULTS

After implementation of search procedures, we reviewed 2970 records: 2965 from database searches, and an additional five from bibliographic reviews. There was an acceptable degree of inter-rater reliability: raters agreed 97.3% of the time, reflecting kappa coefficient of κ = 0.46. Following screening, 211 articles were identified for full-text review for eligibility. Of these, 52 met eligibility (see Fig. 1). Forty-three contained provider outcomes, 15 contained patient outcomes, and six contained both provider and patient outcomes.

The most common health topics addressed by EELM were hepatitis C, chronic pain management, and dementia and elderly care. Thirty-nine of 52 articles focused on EELM implemented in the USA, with Canada and Australia as the next most common countries. Year by year, there has been an overall increase in the number of published articles evaluating EELM (see Fig. 2).

Figure 2
figure 2

Publications by topic, year, and outcome classification.

Full size image

The format of sessions ranged from weekly to monthly, lasting 60 to 180 min per session, with wide variation in number of sessions conducted—in part because of the continuous nature of intervention protocols. Similarly, there was variability in the number of trainees and number of patients served by trainees. In some instances, these numbers were not reported. Below, we present a topical synthesis of the articles, organized by provider-related outcomes and patient-related outcomes; a full description of outcomes reported is found in the Online Appendix.

Provider-Related Outcomes

Between 2007 and 2018, 43 of 52 articles presented quantitative or qualitative evidence outlining provider-related outcomes of EELM. Studies most frequently measured outcomes in one of four areas: (i) provider satisfaction with quality and content of trainings (n = 17; 40%); (ii) provider knowledge acquired (n = 18; 42%); (iii) enhanced provider confidence or self-efficacy associated with care delivery (n = 18; 42%); and (iv) changes in self-reported provider behaviors associated with patient care (n = 7; 16%). In terms of study design, 23 of 43 (53%) involved a counterfactual—either within- (pre vs. post) or between-subjects. While only one of the studies included an element of randomization, three studies involved both within- and between-subject comparisons.

Provider Satisfaction

Assessment of provider satisfaction largely entailed administration of post-intervention structured surveys.19, 23,23,25, 27, 33, 34, 41, 44, 50, 70 The median response rate was low (under 50%); however, self-reports consistently indexed positive ratings, at both the item-level and survey-level. In several instances, satisfaction was framed in terms of participation benefits, such as “Because of [EELM], I have expanded my practice to include new skills.19” In addition to structured surveys, several authors conducted focus group discussions17, 20, 33, 54 and semi-structured interviews24, 28, 43, 53 to solicit feedback on aspects of EELM that worked well or less well, often with a focus on acceptability of the technology platform utilized. Here, responses were also generally positive.

Provider Knowledge

In one study,42 authors evaluated provider knowledge by merely asking participants after training to self-report whether they perceived their knowledge to improve. More often, studies implemented a pre-post design, in which providers were asked to self-assess their knowledge at baseline and again at endline, with significant changes observed.17, 36, 38, 39, 51, 52, 56, 71 In a subset of assessments, knowledge surveys were constructed by the authors and administered.15, 23, 26, 39, 41, 47 The authors found significant improvements in objectively measured content knowledge. By contrast, a cluster randomized controlled trial on chronic pain education did not show knowledge gains among ECHO participant clinics compared with non-participant clinics.26

Provider Confidence

Change in confidence and self-efficacy focused on whether providers reported greater confidence in ability to diagnose and/or treat patients following EELM participation.17, 23, 26, 30, 32, 34, 41, 48, 51, 52, 56, 71 Metrics along these lines were reported in most studies, ranging from self-reported changes following participation,32 to within-subjects change from baseline to endline,48 to between-subjects comparisons in perceived competence,30 including in one randomized controlled trial (RCT).26 In most instances, results were positive and significant; a notable exception came from the RCT on chronic pain management.26

Provider Behavior Change

Several studies administered surveys in which providers were asked to self-report behavior change as a result of participating in case presentations. For example, Komaromy and colleagues35 found 77% of participants reported that case discussion changed their patient care plan. Likewise, Catic and colleagues21 observed that recommendations for treatment were incorporated by case presenters 89% of the time. Qaddoumi and colleagues46 reported that 91% of case presenters followed recommendations. In other studies, providers were merely asked via survey whether EELM participation had or would alter care provision19, 43; on such occasions, providers responded positively.

Patient-Related Outcomes

Fifteen of 52 identified studies (29%) discussed patient-related outcomes, including changes in care processes and outcomes of care. Few studies examined cost of care.

Hepatitis C

Four studies reviewed hepatitis C outcomes. Arora and colleagues16 compared sustained virologic response (SVR) between patients at training versus trainee sites and found no difference (p > 0.05), indicating trainees (generalists) performed at a level comparable to trainers (specialists). Similarly, Mohsen and colleagues62 compared 100 patients of providers who participated in an EELM to 100 patients who received care in a tertiary liver clinic (TLC). Initiation of direct-acting antiviral therapy was similar between groups (EELM, 78%; TLC, 81%), as was completion of treatment (EELM, 89%; TLC, 86%) and—to a lesser extent—SVR (EELM, 87%; TLC, 96%). Statistical significance was not reported.

Beste and colleagues18 identified providers trained via EELM, and compared likelihood of patient treatment initiation among EELM participants and non-participants. The authors found treatment initiation was higher among trainees (hazard ratio [HR], 1.20; p < 0.01), but this effect was a result of increased initiations among only those patients presented in case discussions (HR, 3.30; p < 0.01). Ní Cheallaigh and colleagues43 conducted a series of semi-structured interviews with EELM trainees. Interviewees reported that patients attending their practice were beneficiaries of ECHO. For example, one trainee remarked, “Now, access to specialist clinics has improved. [The local specialist] has actually taken back some people he discharged. He’s also seen a couple of new people.43”

Chronic Liver Disease

We identified two studies on chronic liver disease. The first, by Glass and colleagues,29 found that EELM training allowed patients to access care an average of 9.6 days sooner and saved 250 miles of travel compared with those seeking in-clinic specialty care. A second study, by Su and colleagues,64 examined the effect of receiving a virtual consultation through the VA’s SCAN-ECHO program. Between 2011 and 2015, 513 veterans with chronic liver disease received a virtual consultation from a SCAN-ECHO provider, while 62,237 did not. After propensity score matching on characteristics predictive of receiving a visit, researchers found hazard ratio of all-cause mortality among those receiving a virtual consultation to be 0.54 (p = 0.003), compared with no visit.

Chronic Pain Management and Opioid Addiction

Four studies examined chronic pain management. Anderson and colleagues15 compared providers at community health centers who participated in EELM trainings with those who did not. They found, among those who participated, the percent of patients with an opioid prescription declined from 56.2 to 50.5% (p = 0.02), with no decline observed in the comparison group. Conversely, referrals for behavioral health and physical therapy increased (p < 0.001). Two other studies, by Katzman and colleagues61 and Frank and colleagues,59 examined prescription and referral rates, respectively. Katzman and colleagues61 inspected opioid prescription rates across 1382 clinics associated with the Army and Navy, 99 of which participated in an EELM. Compared with patients of providers who did not participate in EELM (n = 1,187,945), those with providers who did participate (n = 52,941) observed a much greater decline in prescriptions: from 23 to 9% (p < 0.001). Meanwhile, Frank and colleagues inspected likelihood of referral among patients presented as EELM cases versus those not presented as cases; cases were more likely to be referred to physical therapy (HR, 1.10; p < 0.05).59 A final study, by Carey and colleagues,58 performed a spatial reach analysis, concluding that patient travel distance to specialty pain care was associated with only slightly lower odds of access to an EELM-trained provider (OR, 0.98; p = 0.01), versus sizably lower odds of care receipt at a specialty care clinic (OR, 0.78; p < 0.001).

Geriatric Care

Three studies examined elderly care for those with mental health conditions, including dementia; one additional study examined transitional care. Catic and colleagues21 studied the effect of adhering to expert recommendations for residents with dementia, and found that providers who followed EELM recommendations were more likely to report “clinical improvement” among patients (74% vs. 20%; p = 0.03). Fisher and colleagues28 examined the relative change in care utilization and costs among elderly patients with mental health conditions, compared with elderly patients without such conditions, before versus after providers participated in EELM training. Among patients with mental health conditions, there was a reduction in emergency department costs: from $406 to $311 (p < 0.05); this reduction was not observed in the comparison group. Gordon and colleagues60 examined quality of care metrics among elderly patients at facilities with providers who were EELM-trained versus not. They observed non-significant differences on primary outcomes (restraint and antipsychotic medication use), but they did find lower rates of urinary tract infections (UTI) among patients seen at facilities with providers trained through EELM (OR for UTI, 0.77; p < 0.05).

Moore and colleagues63 examined transitional care among elderly adults. Among patients with providers at a skilled nursing facility who had participated in EELM training, the authors found shorter lengths of inpatient stay (p = 0.01), lower 30-day hospital re-admission rates (p = 0.03), and lower 30-day care costs (p < 0.001) compared with providers who had not participated. This difference was significant even after adjusting for baseline differences in case mix.

Diabetes Management

Watts and colleagues65 reported training two primary care physicians on diabetes management through EELM. Providers reported that—among patients with poorly controlled diabetes (i.e., all patients with HbA1c > 9%)—mean HbA1c levels decreased from 10.2% before training sessions to 8.4% after training (p < 0.001) 5 months later, a clinically significant difference.

Strength of Evidence

Provider-related outcomes have relied heavily on self-reports for providers who (i) self-select to participate in EELM, (ii) maintain participation in trainings over time, and (iii) complete feedback surveys. The one RCT examining provider-related outcomes concluded null results. Among studies that collected data before versus after EELM trainings, most offered no control group, raising the question of what would have happened in the absence of training, or if EELM trainings were substituted with a different set of learning tools.

Quality of patient-related outcomes varied widely. While mental health and substance use disorders have been the most frequently implemented EELM in the USA, we found no literature describing the impact of EELMs on patient outcomes associated with these conditions apart from among elderly adults. For conditions like osteoporosis, for which there were provider-related outcomes, we identified no articles assessing patient-related outcomes. For hepatitis C, chronic pain management, dementia care, and diabetes, there was at least one article published in which a counterfactual was incorporated. Two studies, one by Anderson and colleagues15 and one by Katzman and colleagues,61 employed quasi-experimental approaches. In a majority of instances, authors identified statistically significant results in favor of EELM. With the exceptions of virologic suppression in the context of hepatitis C and A1C levels in the context of diabetes, reported outcomes were process measures rather than outcome measures.

DISCUSSION

We identified 52 studies between 2007 and 2018 that reported provider- and/or patient-related outcomes from EELM of medical tele-education. Based on our analysis, the empirical evidence for EELM’s impact on patient and provider outcomes is low.

Regarding provider-related outcomes, 43 articles have been published in the past 11 years. Over three-quarters provide no between-subjects comparison group, raising a question as to what would be observed in the absence of intervention or under an alternative intervention. While measures like provider satisfaction and self-efficacy are inherently subjective and susceptible to social desirability bias, such biases could be addressed by the inclusion of an active control condition or a third-party evaluator. In 14 instances, no baseline data were recorded, meaning that change in outcomes due to the intervention was unobservable. For measures like provider knowledge, which can be more objectively measured through formal testing, over half of studies relied on subjective self-reports. The response rate was also as low as 7% of participants, suggesting self-selection bias.37 The one recorded cluster RCT, by Eaton and colleagues,26 did not find a benefit of EELM in terms of provider knowledge or perceived competence.

Arguably, the most important provider-related outcomes are behavior changes. To this end, three studies examined the effect of provider presentations on provider behavior. The authors found that providers who presented cases altered care 77–91% of the time.21, 35, 46 Beste and colleagues18 also found that presenting cases resulted in increased initiation of patient care for hepatitis C. However, increased treatment initiation was only observed among those cases presented during trainings, and not among other patients treated by the same provider. This raises a question of whether EELM are truly building capacity to handle cases without assistance and constitutes an important area for further investigation.

With respect to patient-related outcomes, we identified 15 EELM studies published over the past 11 years. Eleven of these included a comparison group; however, none involved an element of randomization. For all but two measures, outcomes examined were process measures—for example, frequency of prescriptions or number of referrals. While process measures are likely associated with direct patient outcomes, the inferences drawn from these are indirect. However, for three conditions—hepatitis C, dementia care, and chronic pain management—studies showed improvements in different processes and outcomes of care, which suggests that EELM may be beneficial in those conditions.

In terms of direct patient outcome measures, three studies examined the rate of sustained virologic response among individuals treated for hepatitis C. While two articles found that SVR was similar among patients who sought care from EELM trainees and experts16, 62—an indication of project success—another found that SVR did not differ between those seeing providers who received EELM training versus did not receive EELM training.18 In a separate study looking at the effect of EELM on diabetes care,65 the authors found positive results that training led to reductions in patient mean HbA1c values within practitioner panels, a result not found in a comparison group. While results here show promise, the study was limited by sample size, with only 2 EELM providers and 2 control providers.

While existing evidence suggests the potential impact of EELM in improving patient outcomes, our SOE assessment underscores the need for more rigorous evaluation to substantiate the model. One of the main findings is that the quality of evidence for the effectiveness of EELM is generally rated as “low” or “very low” based on the GRADE system. However, it is important to note that this finding is by no means limited to EELM; in fact, many models of service delivery are supported by limited evidence, for example, the widescale initiative of Housing First, which provides rapid housing to improve client housing security and health. While it is a widely used model, only limited evidence exists for its impact on long-term health outcomes, despite four randomized controlled trials.72 Nevertheless, it is appropriate to continue to strive for higher quality evidence.

A few study limitations should be noted. First, while search criteria were meant to be broad, it is possible articles were overlooked, particularly if they did not contain key words in the Appendix Table online. Second, we were unable to include works in progress, though we identified several. Similarly, due to a limited subset of studies that were not ECHO affiliated, we were not equipped to make ECHO versus non-ECHO EELM comparisons, though this may be an interesting avenue to pursue as non-ECHO EELM proliferate.

In summary, we identified 52 articles over the past 10 years which outline provider- and patient-related outcomes from EELM implementations. One of these comprised a randomized controlled trial with non-significant findings, while a plurality were cross-sectional surveys with high risk of bias. Given the capacity-building orientation of EELM, it would be important for studies to include longer periods of follow-up to assess maintenance, as well as to compare the costs and outcomes of EELM to alternative forms of continuing medical education that rely on technology. As noted, models like EELM that are novel within the healthcare delivery landscape are liable to incrementally establish an evidence base. In this respect, our findings are not surprising. Rather, the assessment is meant to provide an inventory of the existing literature, which may establish a benchmark for EELM moving forward.