The Hospital Readmissions Reduction Program created in the Affordable Care Act imposes financial penalties on hospitals with higher readmission rates for heart failure, acute myocardial infarction, or pneumonia. As one of the few remediable factors associated with readmission rates are transitions of care,1 there is increasing interest in assessing and improving transitions of care. The National Quality Forum (www.qualityforum.org) endorsed a 3-item Care Transitions Measure (CTM-3) for evaluation of hospital performance in care transitions, and this was incorporated into the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) patient satisfaction survey.2

Scores on the CTM-3 have been reported to be inversely correlated with risk of emergency department (ED) visits or readmissions within 30 days.3,4,5 However, the studies demonstrating this were small (200 and 192 patients with COPD, stroke, hip fracture, heart failure, or diabetes)3, 4 or highly selected (2963 patients with PCI or CABG from a single center).5 Moreover, all 3 of these studies could only capture ED visits or readmissions within the same hospital system—a substantial limitation since up to one-quarter of readmissions in North America are at hospitals other than the original one and these readmissions exhibit poorer outcomes.6,7,8,9,10 Thus, it is unknown whether the CTM-3 exhibits the same relationship to ED visits or readmissions in more diverse patient populations and with complete capture of subsequent healthcare resource use. Furthermore, although multiple studies have examined the psychometric properties of patient experience questionnaires such as the HCAHPS and the NHS Inpatient Survey (which contain questions analogous to those in the CTM-3 in the “Leaving Hospital” subsection), few studies have examined whether specific elements of patient experience questionnaires actually measure what they are intended to.11 As recently pointed out by Atreya and Pack, transitional care programs that improve CTM-3 scores do not necessarily reduce recidivism rates to the ED or the hospital suggesting that “there is substantial work yet to be done with the CTM-3 before it is ready for prime time use”.12

Despite these caveats, several health plans (including Medicare under the Value-Based Purchasing program) have created pay-for-performance initiatives built around hospital performance on patient experience of care measures such as CTM-3 scores.13 Thus, there is an urgent need to establish whether CTM-3 scores really are a valid proxy for quality of transitional care.14 To that end, we designed this study to examine CTM-3 scores in a random sample of all patients discharged from acute care hospitals in a universal access, publicly funded healthcare system where all interactions with the healthcare system are recorded (permitting 100% capture of subsequent ED visits or readmissions to any hospital in the entire region of study).

METHODS

Participants

We included all adults discharged from any of the 113 acute care hospitals in Alberta between April 2011 and March 2016 who completed the HCAHPS survey within 6 weeks of discharge. Alberta Health Services (AHS) is the single healthcare organization for all acute care hospitals and EDs in Alberta. AHS uses trained interviewers to conduct telephone surveys between 3 and 6 weeks after discharge on a random sample of 10% of all patients 18 years or older discharged to home, long-term care residence, or hospice after an inpatient stay of at least 24 h. Patients discharged from psychiatric units or with psychiatric consultations, and any hospitalizations associated with still-births, dilation and curettage procedures, or linked to a newborn with length of stay greater than 6 days are excluded. For the purposes of this study, we excluded any obstetric or gynecologic hospitalizations to focus on medical and surgical discharges and randomly selected one hospitalization for any patients with multiple HCAHPS-eligible hospitalizations. The patient flow chart is in Figure 1.

Figure 1
figure 1

Cohort selection flow chart.

Full size image

Survey Instrument

Although AHS administers a 51-item modified version of the HCAHPS, it does include all 3 questions in the CTM-3: “during the hospital stay, staff took my preferences and those of my family or caregiver into account in deciding what my healthcare needs would be when I left”, “when I left the hospital I had a good understanding of the things I was responsible for in managing my health”, and “when I left the hospital, I clearly understood the purpose for taking each of my medications”. There are 4 response options for each question ranging from strongly disagree to strongly agree (and a “not applicable” option for the third question). The telephone interviewers use a standard script and data is captured using computer-assisted telephone interview software (Voxco; Montreal, Canada). We converted the CTM-3 results to a 100-point scale in two ways. In the first, we calculated the average CTM-3 score for each respondent after assigning points to each of their answers (from 1 for strongly disagree to 4 for strongly agree) and averaging across the 3 questions (https://caretransitions.org/wp-content/uploads/2015/08/CTM-3-SCORING.pdf). In the second, we calculated the proportion of respondents who strongly agreed with all 3 questions (https://caretransitions.org/wp-content/uploads/2015/08/CTM3Specs0807.pdf).

Data Linkage

We linked HCAHPS survey results with the Discharge Abstract Database (which records whether the hospitalization was elective or unplanned, admission date, discharge date, most responsible diagnosis, and up to 25 other diagnoses for all acute care hospitalizations in Alberta) and the National Ambulatory Care Reporting System (which captures all patient visits to any of the 104 EDs in Alberta with coding for up to 10 conditions) using each patient’s unique personal health number. We used the index hospitalization and all hospitalizations/ED visits in the prior 12 months to establish comorbidity profiles for each patient using ICD-10 codes and case definitions previously validated in these databases.15 We classified patients as being rural or urban residents based on their primary residence postal code in the Alberta Health Registry.

Outcome Measures

We examined all-cause ED visits and all-cause non-elective readmissions at 30 days, 3 months, and 12 months after index discharge date. The primary outcome was all-cause ED visit or non-elective readmission at 3 months after discharge.

Statistical Analyses

We compared baseline characteristics and outcomes between patients who did vs. did not complete the CTM-3 questions, broken down by whether they were discharged after a medical hospitalization or a surgical hospitalization, by standardized effect sizes (Cohen’s d of 0.2, 0.5, and 0.8 represent small, medium, and large effect sizes, respectively). We compared outcome rates between non-respondents and respondents. We examined the relationship between the CTM-3 score and each of the outcomes of interest. In multivariate analyses, we created a series of logistic regression models adjusted for baseline characteristics (age, sex, urban/rural residence, and any other variables in Table 1 with differences of greater than 0.1 based on the standard difference between respondents and non-respondents [except teaching hospital status since it was collinear with urban residence], using a backward stepwise selection method) and LACE (length of stay, acuity, comorbidity, and emergency department use) score16 to test whether CTM-3 scores were independently associated with our outcomes of interest. In a sensitivity analysis, we excluded any patients with events in the first 42 days to eliminate potential confounding from early events (i.e., before the patient completed the HCAHPS survey) influencing CTM-3 scores and examined the relationship between CTM-3 scores and readmissions/ED visits between 43 and 365 days after discharge using a series of logistic regression models.

Table 1 Characteristics of Patients
Full size table

Ethics

As all analyzed data was de-identified and there was no contact between study personnel and patients, a waiver of consent was granted by the University of Alberta ethics board due to the retrospective nature of this study (Pro00081121). Of note, respondents to the HCAHPS are informed at the time of the telephone survey that their anonymized data could be used for quality assurance and/or research purposes.

RESULTS

In the 5 years (2011–2016) we studied, 48,384 (8.5%) of all 569,648 adult Albertans with medical or surgical hospital discharges were contacted for the HCAHPS survey and the response rate was 39.3% (45.2% for patients discharged from medical wards and 32.6% for patients discharged from surgical wards). The 19,007 patients who completed the CTM-3 were more likely to be female, rural residents, and from smaller non-teaching hospitals than the 29,377 patients who did not respond but age, length of stay (LOS) for the index hospitalization, Charlson score, LACE score, admitting service, and prior health resource use were very similar in both groups (Table 1). Amongst CTM-3 respondents, patients discharged from medical units were older (62.2 vs. 55.3 years, p < 0.001), had longer LOS (6.8 vs. 4.6 days, p < 0.0001), higher Charlson scores (2.3 vs. 1.3, p < 0.0001), and greater LACE scores (11.4 vs. 7.7, p < 0.0001) than those discharged from surgical units (Table 1).

Amongst medical discharges, the proportions of patients with all-cause ED visits were similar between CTM-3 respondents and non-respondents: 56.9% vs. 57.1% at 30 days, 77.5% vs. 77.2% at 3 months, and 92.7% vs. 92.1% at 12 months (Table 2). However, the all-cause readmission rates were significantly higher for CTM-3 respondents after medical discharges: 8.8% vs. 7.9% at 30 days, 17.6% vs. 16.0% at 3 months, and 36.5% vs. 33.9% at 12 months (Table 2). The opposite pattern was seen for surgical discharges with fewer CTM-3 respondents (compared with non-respondents) having all-cause ED visits (61.5% vs. 64.1% at 30 days, 77.6% vs. 81.8% at 3 months, and 89.0% vs. 91.4% at 12 months, Table 2) or readmissions (4.1% vs. 4.7% at 30 days, 6.8% vs. 7.6% at 3 months, and 14.2% vs. 14.5% at 12 months, Table 2). Similar patterns were seen when expressed as rates per 100 person years (Online Appendix, eTable 1) but all of the standardized differences were very small (all less than 0.09, and most less than 0.04). Our sensitivity analysis excluding patients with events that happened within the first 42 days demonstrated the same pattern as described above: after medical discharges, CTM-3 respondents exhibited higher event rates than non-respondents but after surgical discharges, CTM-3 respondents exhibited lower event rates than non-respondents (Online Appendix, eTable 2).

Table 2 Post-discharge Outcomes
Full size table

CTM-3 scores were significantly lower (all p < 0.01) for females, older patients, those discharged from medical vs. surgical units, those hospitalized in teaching hospitals, patients who did not return to independent living after discharge, and those with longer LOS, higher Charlson scores, higher LACE scores, or prior ED visits or hospitalizations (Table 3). It is important to note that all of these factors are known to influence post-discharge outcomes, and thus these factors confound the relationship between CTM-3 scores and outcomes after discharge. Even without adjusting for these confounders, there was no significant association between CTM-3 scores and outcomes at 30 days (mean score 77.5 in those who subsequently had an ED visit or readmission vs. 77.9 in those who did not, p = 0.13, aOR 0.99, 95% CI 0.986–0.004) (Table 4). The multivariate model to predict 30-day readmissions using age and LACE score alone had a c-statistic of 0.65; adding CTM-3 scores to that model only changed the c-statistic to 0.655 which was not statistically significantly different. CTM-3 scores were significantly lower (both p < 0.01, Table 4) in patients who subsequently had non-elective readmission/ED visit at 3 months (77.5 vs. 78.5) and 12 months (77.6 vs. 79.5). However, the magnitude of outcome differences was small even for large changes in the CTM-3 score: for every 10 point decrease in the CTM-3 score, the risk of readmission/ED visit was 2.6% higher (aOR 1.026, 95% CI 1.01–1.05) at 3 months and 4.0% higher (aOR 1.04, 95% CI 1.01–1.08) at 12 months. After excluding events that happened within the first 42 days, we found that CTM-3 scores were still significantly associated with readmissions/ED visits between 43 and 365 days after discharge but the magnitude of the association again remained small: for every 10 point decrease in the CTM-3 score, the risk of readmission/ED visit at 12 months was only 4.4% higher (aOR 1.04, 95% CI 1.01–1.08).

Table 3 Mean CTM-3 Scores by Baseline and Hospital Characteristics
Full size table
Table 4 Mean CTM-3 Scores by Outcomes
Full size table

DISCUSSION

In order for a performance measure to be a good quality indicator, it must be feasible to measure in a timely fashion, comparable across settings (i.e., risk adjustment is possible to permit fair comparisons), valid (i.e., truly discriminates quality for the measure it is the supposed proxy for), reliable (i.e., reproducible in different hands), precise (i.e., clear, well-specified definitions with wide variation between patients or settings and sensitive to change), and actionable.17,18,19

We believe our findings suggest that the CTM-3 score is not a suitable performance measure for the quality of transitional care for several reasons. First, while the CTM-3 score is feasible to measure and reproducible (since it consists of 3 standardized survey questions), and was shown to improve after implementation of a care transition program in a randomized trial, that same trial demonstrated that the improvement in CTM-3 scores was not associated with a change in 30-day readmission or ED visit rates.20 We believe that identifying problems in preparedness to transition back to self-care 4 to 6 weeks after discharge is too late and outside the actionable window; while rates of early recidivism (within 1–2 weeks) seem more likely to reflect the quality of transitional care, this is before CTM-3 scores are often collected. Second, we demonstrated that CTM-3 scores are influenced by patient and hospital factors at baseline, none of which are readily modifiable, and to the best of our knowledge (and others),12 there are no models to risk adjust CTM-3 scores for these confounders at this time. Indeed, 2 prior studies in the USA also reported that age, gender, and baseline health status influenced CTM-3 scores.5,14 Third, the fact that CTM-3 scores were not associated with 30-day post-discharge outcomes, and were only weakly associated with 3- and 12-month outcomes, raises questions about whether the CTM-3 truly is a valid proxy for the quality of transitional care or whether other confounders influence late outcomes more than the transitional care 3–12 months earlier. Mixon and colleagues also recently reported that the CTM-3 score was not significantly associated with 30-day or 3-month death/readmission in 1239 patients with cardiovascular diagnoses in Tennessee—the c-statistic for the CTM-3 score was only 0.52 for 30-day outcomes in their study.21 While an examination of the Medicare Hospital Compare database reported statistically significant inverse relations between CTM-3 scores and 30-day readmission rates after hospitalizations for heart failure, pneumonia, or acute myocardial infarction, all of the correlations were weak (with correlation coefficients between − 0.20 and − 0.30).22

We found that patients who received the HCAHPS survey differed from those who were not surveyed in several baseline factors—this was not surprising since AHS selected medical and surgical patients and excluded psychiatric patients and certain obstetric cases from the HCAHPS surveys. We also found that CTM-3 respondents were different from non-respondents in several baseline factors and had different outcome rates. This is again not surprising since response bias is common in any survey—more satisfied patients are more likely to respond. Regardless, it means that we may have underestimated the magnitude of the inverse association between scores and event rates in our study, although the fact that the differences in post-discharge outcome rates were so small argues against a substantive effect.

Although we were able to examine CTM-3 scores for a wide spectrum of patients discharged from general medical or surgical units in an entire Canadian province, and were able to identify and capture all non-elective readmissions or ED visits anywhere in the province, there are some limitations to our work. First, the CTM-3 does not capture data on socioeconomic status or other social determinants of health which influence readmission risk and ED usage. Second, in examining the association between CTM-3 scores and post-discharge outcomes, we were not able to adjust for post-discharge outpatient visits or continuity of care, both of which are known to influence rates of readmission or ED visits.23,24,25 Third, the CTM-3 does not assess patient understanding of medication instructions, yet medication adherence is known to be a key driver of post-discharge outcomes.26 A fourth limitation, which is common to all studies examining the relationship between CTM-3 scores and post-discharge events, is that patients with early readmissions/ED visits may complete the HCAHPS survey after the second event, thereby biasing their answers. We do not have data on when each respondent in this cohort completed the CTM-3 survey but prior work in Alberta revealed that the vast majority of surveys were completed within 4 weeks after hospital discharge.27 For this reason, we looked beyond the 30-day post-discharge period to consider event rates at 3 and 12 months and in a sensitivity analysis, we excluded events within the first 6 weeks of discharge. Fifth, although selection and response biases are always a concern in survey studies, earlier work has demonstrated that those patients AHS chooses to administer the HCAHPS to, and those who respond, are representative of the broader Alberta medical/surgical inpatient population.27 Furthermore, the comorbidity profiles and resource utilization patterns of our cohort are consistent with reports of patients discharged from medical and surgical units in other Canadian provinces and Boston.28 Finally, any survey study, particularly one conducted by telephone,29 is at risk for social desirability response bias, which could inflate CTM-3 scores and thereby potentially reduce any apparent associations between CTM-3 scores and outcomes.

In conclusion, there is increasing enthusiasm for the use of patient experience of care measures to judge hospital quality and drive pay for performance programs, and there is evidence that some of these measures do correlate with important outcomes during and after hospitalization such as mortality, all-cause readmissions/ED visits, or other patient safety indicators.30,31 However, we do not think the CTM-3 score meets the evidentiary threshold for a performance measure of the quality of transitional care since it is influenced by baseline patient and hospital factors, is not associated with 30-day post-discharge outcomes, and is only weakly associated with 3- and 12-month outcomes.