Background

There is broad recognition of the importance of evidence in determining clinical decision-making [1]. For evidence-based healthcare, decision-makers (e.g., patients, clinicians, guideline developers) increasingly rely on systematic reviews (“reviews”) [1]. Reviews identify primary studies, such as clinical trials and observational studies, that have addressed the research question of interest. This research question typically defines the population, interventions, and comparators; these defined aspects in turn help delineate the primary studies eligible for the review.

Reviews may or may not include quantitative syntheses of data across studies (“meta-analyses”). When appropriately conducted, meta-analyses provide decision-makers with summary estimates (e.g., relative risks) and accompanying estimates of uncertainty (e.g., 95% confidence intervals) that convey information about treatment effectiveness or safety succinctly [2]. Often, however, meta-analyses cannot be conducted because the studies address somewhat different clinical questions, assess different outcomes than the systematic reviewer (“reviewer”) had pre-specified, are methodologically heterogeneous, or are poorly-reported (e.g., inadequate information about results). In these circumstances, a study may be eligible for the review, but may not contribute to a meta-analysis [3]. When a review includes multiple studies, but these studies cannot be included in the meta-analysis, both doers (i.e., reviewers) and users of reviews (i.e., decision-makers) can be frustrated. Decision-makers want to know how treatments compare quantitatively; they may not be able to get reliable information about this when only some included studies contribute data to the meta-analysis or when no meta-analysis is possible [4].

Outcomes are measures or events used to assess the effectiveness and/or safety of clinical interventions [5]. A frequent reason for non-conduct of meta-analyses is that the studies assess different outcomes or assess the same outcomes, but do so differently. These scenarios can occur even among high-quality studies.

Although outcomes are fundamental to reviews of interventions, outcomes are typically not considered when determining the eligibility of a primary study in such reviews [6]. This is because outcomes inform meta-analyses, not whether the primary study is eligible for the review. Consistent with guidance in the Cochrane Handbook for Systematic Reviews of Interventions [6], we believe that studies that address the population, interventions, and comparators of interest should be included and cataloged in systematic reviews even if they do not report outcomes of interest. Outcome choice in a review is crucial because: (1) outcomes serve as yardsticks for basing conclusions about treatments; and (2) which outcomes are chosen and how they are defined can impact how many meta-analyses can be done and how many studies can be included in them [7,8,9,10,11].

Outcomes may be assessed differently in different studies because an “outcome” (a seemingly monolithic entity) actually comprises five elements: domain, e.g., visual acuity; specific measurement, e.g., Snellen chart; specific metric, e.g., ≥3 lines of vision lost; method of aggregation, e.g., proportion; and time-points, e.g., 6 months [9, 12]. Another example of the application of this five-element framework to clearly specify a particular data point of interest related to the outcome of “anxiety” is mean (method of aggregation) change (specific metric) in anxiety (domain) measured through the Hamilton Anxiety Rating Scale (specific measurement) from baseline to 1 year (time-point) [9, 12].

We previously demonstrated, through case studies in the fields of eyes and vision [11] and HIV/AIDS [10], that reviewers and clinical trialists addressing the same research question often examine different outcomes. In addition, inconsistency in outcome reporting across eligible studies prevents incorporation of all eligible studies into meta-analyses. For instance, a 2017 Cochrane systematic review comparing non-steroidal anti-inflammatory drugs (NSAIDs) with corticosteroids for inflammation after cataract surgery [13] included 48 trials, none of which reported data for the review’s pre-specified primary outcome, “proportion of patients with intraocular inflammation at 1 week after surgery.”

To document the extent and determinants of this problem, we embarked on the current case study in the field of eyes and vision. Our objectives were to examine the extent to which Cochrane reviews in eyes and vision conducted meta-analyses for the main outcome domain and the reasons why some otherwise eligible studies were not incorporated into meta-analyses.

Methods

Reviews examined

We examined all completed systematic reviews published by Cochrane Eyes and Vision in the Cochrane Database of Systematic Reviews as of August 11, 2017. We excluded reviews that were still in the protocol stage.

Data extraction

We developed a data extraction form in the Systematic Review Data Repository (SRDR), an open-source platform for extracting and archiving data [14, 15]. Using a pilot-tested form, two individuals (from among SM, HK, BTS, and IJS) independently extracted data, resolving discrepancies through discussion. We extracted the following data: year published, population (i.e., eye function/region affected), and types of interventions and comparators. We extracted the numbers of primary, secondary, and other, i.e., non-primary and non-secondary, outcome domains. We also extracted the number of studies included in the review and in ≥1 meta-analysis for any, any primary, any secondary, and any other domain.

“Main” outcome domains

We categorized one domain from each review as its “main” outcome domain (Table 1). For reviews that named only one primary outcome domain, we categorized it as the main outcome domain; for reviews that named more than one primary outcome domain (or named more than one secondary outcome domain), we categorized the primary outcome domain (or secondary outcome domain) with the highest number of included studies as the main outcome domain. For reviews that did not name any primary or secondary outcome domains, we categorized the “other”, i.e., nonprimary and non-secondary, outcome domain with the highest number of included studies as the main outcome domain.

Table 1 Algorithm for categorizing the “main” outcome domain for each systematic review
Full size table

For each main outcome domain, we extracted the other four elements specified: specific measurement, specific metric, method of aggregation, and time-points. For the main outcome domain, we also extracted the numbers of studies that reported measuring it, reported any data, reported any meta-analyzable data, and were incorporated into ≥ 1 meta-analysis. We considered data for a given outcome from a given study to be “meta-analyzable” if the study reported adequate information so that it could be incorporated into a meta-analysis. For categorical outcomes, meta-analyzable meant that either of these conditions were met: (1) total number of participants and number of participants with the outcome were reported for each study arm; and (2) the between-group treatment effect (e.g., relative risk) and an uncertainty estimate (e.g., 95% confidence interval) were reported. For continuous and time-to-event outcomes, meta-analyzable meant that either of these conditions were met: (1) mean and uncertainty estimates were reported for each study arm; and (2) the between-group treatment effect (e.g., mean difference) and an uncertainty estimate were reported.

Results

Reviews examined

We identified 175 completed systematic reviews published by Cochrane Eyes and Vision in the Cochrane Database of Systematic Reviews (Table 2). The reviews were published between January 1, 2005 and August 11, 2017 (median = 2014). The most common populations were patients with retinal/choroidal disease (35 reviews; 20%) and visual impairment/low vision (33 reviews; 19%). The most common types of interventions/comparators were drugs (74 reviews; 42%) and surgeries (67 reviews; 38%).

Table 2 Characteristics of systematic reviews examined
Full size table

Incorporation of studies into meta-analyses for any outcome domain

The 175 included reviews examined a median of 6 total outcome domains, including a median of 1 primary outcome domain, 4 secondary outcome domains, and 1 other outcome domain.

The 175 reviews included a median of 3 studies (IQR 1–9); 125 reviews (71%) included ≥2 studies. For these 125 reviews, Fig. 2 plots the percentage of studies incorporated into a meta-analysis for any outcome domain (blue line) and for the main outcome domain (red bars). Among these reviews, 44/125 reviews (35%) incorporated every included study into ≥1 meta-analysis (for any outcome domain). Conversely, 33/125 reviews (26%) did not incorporate any study into any meta-analysis for any outcome, i.e., they did not conduct any meta-analysis. The remaining 48/125 reviews (38%) incorporated only a subset of their studies into ≥1 meta-analysis. These 48 reviews included a median of 12.5 studies (IQR 6–22), and the meta-analyses in these reviews incorporated a median of 6.5 studies (IQR 4–13).

Among the 125 reviews that could have conducted a meta-analysis, i.e., those including ≥2 studies, the median proportion of studies incorporated into ≥1 meta-analysis for any outcome was 74% (IQR 0–100%). Among the 92 reviews that conducted a meta-analysis, the median proportion of studies incorporated into ≥1 meta-analysis for any outcome was 93% (IQR 64–100%).

Characteristics of main outcome domains

Almost all reviews (172/175 reviews; 98%) named ≥1 primary outcome domain (Table 1). Three in four reviews (131/175 reviews; 75%) each named exactly one primary outcome domain, which we categorized as their main outcome domain. The most frequent main outcome domains across the 175 reviews were visual acuity (31%) and intraocular pressure (6%) (Table 3). Thirty-eight outcome domains were main outcome domains in just one review each. The main outcome was categorical in 70% and continuous in 29% of reviews. Most main outcome domains (98%) were efficacy outcomes, i.e., not safety outcomes.

Table 3 Characteristics of main outcome domains in all 175 systematic reviews examined
Full size table

Incorporation of studies into meta-analyses for the main outcome domain

Among the 125 reviews including ≥2 studies, only 18 reviews (14%) incorporated all their studies into a meta-analysis for the main outcome domain. Conversely, 51/125 reviews (41%) did not incorporate any study into the meta-analysis for the main outcome domain, i.e., they did not conduct any meta-analysis for the main outcome domain. The remaining 56/125 reviews (45%) incorporated only a subset of their studies into the meta-analysis for the main outcome domain. These 56 reviews included a median of 12 studies each, and the meta-analyses for the main outcome domain in these reviews incorporated a median of 4 studies each.

Among the 125 reviews that could have conducted a meta-analysis, i.e., those including ≥2 studies, the median proportion of studies incorporated into ≥1 meta-analysis for the main outcome domain was 28% (IQR 0–71%). Among the 74 reviews that conducted meta-analyses for the main outcome domain, the median proportion of studies incorporated was 67% (IQR 39–91%).

Meta-analysis conduct for the main outcome domain

Figure 1 illustrates a cascading effect of loss of information as regards the main outcome domain in the 175 reviews. Thirty-five reviews (20%) included no studies, i.e., were empty reviews, and 15 (9%) included one study each (Fig. 1). Of the 125 reviews including ≥2 studies, i.e., those in which a meta-analysis could theoretically be done for the main outcome if ≥2 studies reported meta-analyzable data, only 74 reviews (59%) conducted a meta-analysis for the main outcome.

Fig. 1
figure 1

Conduct of meta-analyses for the main outcome domain

Full size image

Reasons for non-conduct of meta-analyses for the main outcome domain

Among the 125 reviews including ≥2 studies, 51 reviews (41%) did not conduct a meta-analysis for the main outcome domain. For 21/51 reviews (41%), fewer than two studies measured the review’s main outcome (Table 4). When ≥2 studies reported meta-analyzable data, there were numerous reasons why reviewers did not conduct a meta-analysis, most frequently due to inconsistency in outcome elements among the included studies. Specifically, data could not be meta-analyzed because the specific measurements used (16/51 reviews; 31%) and time-points examined (9/51 reviews; 18%) were inconsistent among studies.

Table 4 Reasons for non-conduct of a meta-analysis for the systematic review’s main outcome even when ≥2 studies were included in the systematic review (N = 51 of 125 reviews that included ≥2 studies)
Full size table

Figure 2 demonstrates that the loss of information for the main outcome domain (red bars) was similar in pattern to the loss of information when considering any outcome domain (blue line).

Fig. 2
figure 2

Percentage of studies included in the review that were incorporated into a meta-analysis for any outcome (blue line) and for the review’s main outcome (red bars)

Notes: This Figure excludes the 50 systematic reviews in whom a meta-analysis was not possible: 35 systematic reviews that each included 0 studies (i.e., “empty reviews”) and 15 systematic reviews included that each included only 1 study. When the blue line is non-0 but the red bars are 0, it implies that the systematic review did not conduct a meta-analysis for the main outcome, but did so for ≥1 of the remaining outcomes

Full size image

Discussion

Through a case study of all Cochrane reviews in the field of eyes and vision, the current work demonstrates three major areas that need improvement.

First, primary studies addressing similar research questions should align their outcomes better. Studies often could not be incorporated into meta-analyses because the outcomes were not aligned, either because the domains or ≥1 of the other four outcome elements did not overlap. Among the reviews including ≥2 studies, only 59 and 74% could conduct a meta-analysis for the main outcome and for any outcome, respectively. In other words, even when reviews included ≥2 studies, 41 and 26% of reviews missed opportunities to conduct a meta-analysis to succinctly convey information regarding the main outcome and any outcome, respectively.

Second, reviews and primary studies should align their outcomes better. When looking at reviews that could have conducted a meta-analysis, i.e., those including ≥2 studies, the median percentages of included studies incorporated into the meta-analysis for the main outcome and for any outcome were 28 and 74%, respectively. This suggests that, approximately 7 in 10 studies that reviewers include are not incorporated into the meta-analysis for the main outcome, and 1 in 4 studies are not incorporated into the meta-analysis for any outcome. In previous work, we demonstrated poor overlap between outcomes in clinical trials and reviews, and possible differences in the types of outcomes they examine [10, 11]. For HIV/AIDS, we demonstrated that reviewers examined more long-term clinical outcomes and patient-centered outcomes than did clinical trialists. Such differences may arise because: (1) reviews may more directly inform clinical practice guidelines, and (2) reviewers may be less affected by common constraints faced by clinical trialists, e.g., costs and sample size [10].

Our findings beg the question of who should prioritize outcomes for measurement and reporting in research. It has aptly been stated that achieving consensus in outcome use across research “cannot be left to serendipity.” [16] One deliberate and fundamental aspect of the solution to the problem of outcome inconsistency is the development of “core outcome sets.” A core outcome set is a minimum set of outcomes that should be measured and reported in all clinical trials addressing a given condition [17]. Core outcome sets are increasingly common in various health fields; a 2018 systematic review identified 307 core outcomes sets [18]. However, outcome inconsistency remains widespread; 40% of recent (2019) published Cochrane reviews explicitly noted this problem [19].

We [10, 11] and others [20] have argued that, as stakeholders in a given field, systematic reviewers should both participate in the development of and adopt core outcome sets for that field. By broadening the participation in outcome prioritization efforts, this could potentially help ensure that the outcomes that are measured and reported in research are widely relevant and important. Two aspects of core outcome sets are worthy of clarification. First, core outcome sets do not stifle innovation; they are simply meant to represent a minimum set of outcomes that should be reported. Once a core outcome set exists for a given topic, clinical trialists working in that topic area should explicitly specify the intention to measure and report the outcomes in the set. Second, core outcome sets are not static; they can and should be updated as the field advances and new knowledge emerges.

The third major area in need of improvement that our study demonstrates is the reporting of outcomes in primary studies. Results data from primary studies were often not meta-analyzable even when outcomes might have been aligned. In addition, outcome domains were frequently not reported in primary studies or ≥1 of the outcome elements were frequently missing or inadequately reported (e.g., “worsening of disease” without clarification of how “worsening” was defined). It is possible that the studies measured these outcomes, but did not report measuring them or reported them inadequately. If such selective reporting, either non-reporting or inadequate reporting, of outcomes in the included studies occurred as a function of the direction of the outcome’s results, it would be suggestive of outcome reporting bias [21]. In this case study, we relied on the reviewers’ reporting of the extent to which the primary studies reported the outcomes. Because we did not examine the reports of the primary studies (or their protocols), we are unable to comment definitively on whether non-reporting of the outcomes indicates outcome reporting bias. However, outcome reporting bias in primary studies has been documented to be a widespread problem across reviews [22,23,24,25,26], and, as such, is a likely explanation for some outcomes not being reported.

Implications

For the evidence-based medicine paradigm to work, decision-makers must be able to rely on systematic reviews, which in turn rely on the results of primary studies. For results of primary studies to be actionable, there (1) needs to be alignment in outcomes considered important to both primary study researchers and reviewers, and (2) those outcomes need to be reported completely. Important discussions need to be had regarding who should choose outcomes for the field and how such choices should be made. We, in conjunction with others, suggest that these discussions should include, at the least, clinicians, patients, clinical trialists, systematic reviewers, regulators, and other decision-makers [27].

We have demonstrated that the choice of outcomes for systematic reviews may have led to loss of information through non-incorporation of results from included studies into meta-analyses. The most substantial drops in the percentage of reviews conducting meta-analyses for the main outcome domain appeared to be due to inadequate numbers of studies reporting the outcome and, when there were adequate numbers of studies for a meta-analysis (i.e., ≥2 studies), differences in the specific measurements and time-points used.

Our findings also demonstrate that even when focusing on reviews that conducted meta-analyses for their main outcome domain, only about 2 in 3 studies were incorporated into those meta-analyses. As such, non-incorporation of included studies into meta-analyses represents two main problems. First, it represents missed opportunities for using research to inform decision-making through evidence synthesis. This contributes considerably towards research waste [28,29,30]. Second, non-incorporation of included studies into meta-analyses represents a failed obligation on the part of the researchers (both trialists and reviewers) [31]. As a community of researchers, both parties have a solemn obligation to research participants to ensure that their participation will lead to a useful contribution to science; failing to agree upon outcomes that should be collected and adequately reported likely violates this obligation.

Other solutions

Core outcome sets are integral to solving the problems this study illustrates. Other parts of the solution are worth discussing. We agree with existing recommendations against studies being excluded from systematic reviews solely on the basis of the lack of relevant outcome data [3]. Thankfully, such recommendations have been associated with a reduction in the number of reviews excluding studies solely on the basis of outcome data [32]. As the current study demonstrates, the review team’s choice of outcomes may not align with that of the primary studies. This may be particularly true for eyes and vision, a field with few core outcome sets [4, 18]. We also encourage reviewers to report an outcome matrix [23, 24], a transparent and simple way to indicate all fully-reported, partially-reported, or non-reported outcomes in each included study.

Large numbers of empty reviews and reviews including only one study

Twenty-percent of the reviews we examined were empty and 9% included only one study each. While such reviews are useful in driving primary research, the possible reasons for the paucity of studies in them are worth exploring. One possibility is that these represent topics that primary researchers have not yet studied. Another is that only observational studies addressing these topics may exist; Cochrane reviews typically include only randomized trials. It also is possible that these topics reflect the priorities of Cochrane Eyes and Vision and the authors of these reviews, rather than of the field at-large.

Limitations

Our study has certain limitations. First, we focused on Cochrane reviews within one field. Loss of information due to the choice of review outcomes could be a bigger, similar, or smaller problem in non-Cochrane systematic reviews in eyes and vision or systematic reviews in other fields. Second, we analyzed in-depth the extent of incorporation of included studies into meta-analyses only for the main outcome domain. Meta-analyses of other primary, secondary, and other outcome domains may have incorporated higher percentages of included studies. However, Fig. 2 suggests that this is likely not the case. It is possible that our algorithm for categorizing the “main” outcome for each review could have impacted our findings. But, in reviews where more than one outcome domain could have served as the main outcome, we categorized as the main outcome the outcome that the highest number of included studies had reported. Our results thus represent the best-case scenario. Third, most outcome domains (98%) were efficacy outcomes. Selective outcome reporting has also been reported to be a problem for safety outcomes [33]. Fourth, we relied on the reviews to determine whether or not each included study did the following for the main outcome domain: reported measuring it, reported any results for it, and reported meta-analyzable data for it. Related to this, we did not examine the appropriateness or feasibility of the reviewers’ being able to conduct meta-analyses when the included studies reported data in a format different from what the reviewers were interested. As such, our results document what was actually done in the reviews.

Conclusions

This case study of all Cochrane systematic reviews addressing an entire field (eyes and vision) demonstrates that only 59 and 74% of the reviews including ≥2 studies could conduct a meta-analysis for the main outcome and for any outcome, respectively. In evidence-based healthcare, such loss of information represents missed opportunities and a failed obligation by researchers to research participants to ensure that their participation will lead to a useful contribution to science. Core outcome sets and improved outcome reporting can help solve some of these problems.