Advertisement

Internal and Emergency Medicine

, Volume 5, Issue 2, pp 151–155 | Cite as

Selective outcome reporting: telling and detecting true lies. The state of the science

  • Ana Macura
  • Iosief Abraha
  • Jamie Kirkham
  • Gian Franco Gensini
  • Lorenzo Moja
  • Alfonso IorioEmail author
CE - Cochrane's Corner

Introduction

Osteoarthritis is a common form of joint disease in the elderly and may cause severe pain and disability. Besides nonsteroidal anti-inflammatory drugs, the mainstay of management for osteoarthritic pain, transcutaneous electrostimulation (TENS), ultrasound and opioids have been advocated as viable treatment options [1, 2, 3]. However, the evidence for their effectiveness and safety is contradictory. The Cochrane database of systematic reviews offers a series of recently published reviews dealing with osteoarthritis, clarifying the therapeutic role of these interventions. In critically appraising what we already know using the GRADE Working Group grades of evidence, Rutjes et al. rate the quality of the originally published reports for key clinical core outcomes: pain, function and the number of patients experiencing an adverse event. In many of these instances, the overall quality of evidence is rated as low because, among other reasons, only a small proportion of included studies report a specific outcome (i.e., 8 out of 18 studies report the number of patients withdrawn because of adverse events) [3]. In these cases, the authors warn readers against the likely risk of selective outcome reporting bias.

What is selective outcome reporting and how can it lead to bias? Is selective outcome reporting related to publication bias?

In this paper, we briefly summarize the threats to meta-analyses due to selective outcome reporting and discuss recent discoveries in this field.

Publication bias

Randomised clinical trials (RCTs) are considered superior to observational studies in obtaining a precise and statistically unbiased estimate of the effects of an intervention. Unfortunately, although RCTs are desirable experiments in searching for moderate treatment effects, they are still prone to subtle form of bias. The family of publication-related biases plays a major role, and can lead to unrealistic estimates of drug effectiveness or alter the risk–benefit ratio. Publication bias originates in a prejudiced peer-review attitude: many reviewers are predisposed against recommending the publication of studies reporting non-significant findings [4]. Such studies tend to take longer to find their way into the published literature, or remain unpublished. The consequence is that it is difficult to find and include them in a meta-analysis when compared with studies producing statistically significant results [5]. If a study is not published on the basis of its results (publication bias), the omission of negative unpublished trials can lead to an over-inflation of intervention effects [6]. Chan and Altman [7] conclude in a milestone paper, by stating “The medical literature represents a selective and biased subset of study outcomes”. Pushing the concept by Chan and Altman to the extreme, the effect size observed in a meta-analysis of different studies could be on average, an accurate estimate of the extent of net publication bias operating in a specific field.

The awareness of publication bias prompted the use of early registration of trial protocols, and as a response to the 1997 FDA Modernisation Act [8], the US National Library of Medicine established the web-based registry http://clinicaltrials.gov in 2000 [9]. In 2004, the International Committee of Medical Journal Editors announced that any clinical trial aimed at publication in major scientific journals must be registered by September 2005 in a public clinical trials registry before participant enrolment [10]. This resulted in a large increase in the number of trials registered within http://clinicaltrials.gov, which in 2007 also mandatorily requires the reporting of the trial start date, and primary and secondary outcome results within 2 years of trial completion.

Notwithstanding, evidence suggests that registration does not guarantee publication of clinical trials in a timely manner in the scientific literature, and that often the quality of the information provided during the registration process is poor [11]. In particular, studies with statistically significant results or large sample sizes are likely to be published than those with nonsignificant results, and are published earlier. Moreover, the nonpublication of studies is often due to failure to submit rather than rejection by journals [12].

Selective reporting bias

Selective reporting bias in a study is defined to be the selection of a subset of analyses to be reported. When the selection process occurs in relation to outcomes, we refer to it as selective outcome reporting [13]. However, selective reporting may also occur in relation to subgroup analyses [14] per protocol rather than intention to treat analyses [15], as well as other analyses [16]. Selection may be driven to avoid redundancy of similar outcomes (i.e., two health related quality of life scales measuring the same criterion) or futility of some outcomes. Researchers, reviewers and editors are all involved in selecting the most interesting and attention-worthy outcomes (or to save precious journal space). In this case, selective reporting is less problematic. When outcome selection is driven by the significance or effect size, we refer to this as selective outcome reporting bias. In other words, in the presence of selective outcome reporting bias, published results are prone to the ‘statistically significant’ cliché: new statistically significant outcomes are introduced at the time of publication; statistically significant secondary outcomes are upgraded to primary end points; and nonsignificant primary outcomes are possibly omitted from reports [17].

Empirical evidence of outcome reporting bias

The existence of outcome reporting bias has been widely suspected for years although until recently, little was known about the prevalence and impact this has had on systematic reviews. The availability of trial protocols made it evident that often the published reports do not correspond to the registered study protocol. The findings of the principal studies that compared outcomes between protocols and publications are summarised in Table 1 (freely adapted from Table 5 of Dwan et al. [18]). In comparing trial publications to protocols, Dwan’s systematic review finds that 40–62% of studies have at least one primary outcome that was changed, introduced, or omitted, and that outcomes that are statistically significant have higher odds of being fully reported (range of odds ratios: 2.2–4.7).
Table 1

Relevance of outcome reporting bias in major published studies in the field

Study

Number of protocols/publications

Primary outcome stated in protocol is the same as in publication

Primary outcome stated in protocol is downgraded to secondary in publication

Primary outcome stated in protocol is omitted from the publication

A non-primary outcome in the protocol is changed to primary in the publication

A new primary outcome that was not stated in the protocol (as primary or secondary) is included in the publication

Hahn, 2002 [30]

56/–

40% (6/15) stated which outcome variables were of primary interest and 4 of these (67%) showed consistency in the reports

17 (1/6)

17 (1/6)

Chan, 2004a [31]

519/553

67% (32/48)

23% (11/48)

13% (6/48)

9% (4/45)

18% (8/45)

Chan 2004ba [21]

102/122

47% (37/76)

34% (26/76)

26% (20/76)

19% (12/63)

17% (11/63)

Ghershi, 2006b

74%

31%

10%

Von Elm 2008 [32],

451/375

26% (24/92)

11% (11/101)

Rising, 2008 [19]

164/128

86.6% (155/179)

22.9% (41/179)

10% (18/179)

Al Marzouki, 2008 [33]

64/50

78.1% (57/73)

2.7% (2/73)

6.8% (5/73)

10.9% (8/73)

Mathieu, 2009 [34]

–/323

66.7% (98/147)

4% (6/147)

10.2% (15/147)

5.4% (8/147)

15% (22/147)

–The study did not investigate this item

aGives results for trials with discrepancies for ≥1 primary outcome

bGhersi D (2006) Issues in the design, conduct and reporting of clinical trials that impact on the quality of decision making. Thesis (Ph. D). School of Public Health, Faculty of Medicine, University of Sydney

Rising et al. [19] illustrate the problems of selective outcome reporting bias with the example of the trials submitted to the Food and Drug Administration (FDA) in approved New Drug Applications (NDAs). The publication rate of efficacy trials submitted to the FDA and the trial characteristics as submitted to the FDA were compared with trials and characteristics published in peer-reviewed journals. Forty-one primary outcomes from the NDAs were omitted from the papers; the papers included 15 additional outcomes that favoured the test drug and 2 other neutral or unknown additional outcomes. There were 43 outcomes in the NDAs that did not favour the NDA drug, and of these, 20 were not included in the papers. Thus, the papers included more outcomes favouring the test drug than did the NDAs. These findings indicate that there are discrepancies between the data and the conclusions in NDAs and those published in medical journals, which tend to lead to more favourable presentation to practitioners of the NDA drugs.

A prevalence and impact of outcome reporting bias in a large unselected cohort of Cochrane reviews have been investigated in the ORBIT (Outcome Reporting Bias In Trials) project. This study finds that one-fifth of the statistically significant meta-analyses of the review primary outcomes are not robust to outcome reporting bias, and one-quarter will have overestimated the treatment effect by 20% or more [20].

These studies on selective outcome reporting help to explain the preponderance of favourable results observed in the medical literature. Three additional considerations can provide further explanation.

First, the number of outcomes reported in protocols is remarkably high, and it is generally not possible to reliably discern the primary from the secondary outcomes. Although not all the cited studies reported all the measured outcomes, Chan et al. [21] find that the median number of efficacy outcome measures per trial to be 20 (10–90th centile: 5–63) in one study that included 122 trials, and 11 (10–90th centiles: 5–63) in another study that included 519 trials [7]. The specification of a high number of outcomes can increase the potential for data dredging, where potentially significant associations are stumbled upon during data analysis. Often only those that are found to be significant are reported.

Second, in clinical trials, the primary outcome is the most important measure of efficacy, on which the sample size is calculated. The assumptions for this calculation are based on previously observed data or published results. When the primary outcome is replaced by a secondary one, these assumptions may differ, and analyses performed on an inadequate sample size may lead to erroneous results, usually favourable. Indeed, sample size and statistical methods in published trials are often clearly discrepant with respect to the pre-specified protocol [22]. The addition and removal of outcomes together with the sample size recalculation carry a high risk of bias. In fact, the prevalence of favourable results from the cohort of trials examined in the reported surveys is very high, and may reflect no true differences [18].

Third, the reporting of favourable preliminary results to the treatment under investigation has been under some scrutiny. As a result, scientists have been advised to critically appraise any favourable results presented at Meetings and Congresses or in the abstracts of published studies [23]. This high prevalence of favourable results can be related to outcome reporting bias of secondary or subgroup analysis, and it is often not followed or confirmed by peer-reviewed full paper publications.

It clearly appears that even in the era of evidence-based medicine, most of the knowledge we balance our clinical practice upon is at high risk of being, in reality, a true lie. True because any scientific trust is an exercise in trust [24]. True because this knowledge passed a reviewer critical appraisal, and received a favourable assessment.

A lie because even the most successful and appreciated studies are simply the ones that may suffer the worst net bias [25]. A lie because even the meta-analytic process is not always robust enough to detect selective outcome reporting bias [20].

Conclusions and policy implications for researchers and reviewers

Given the variability in protocol registration of clinical trials and in the quality of information provided, editors, peer reviewers, and readers of medical journals must carefully scrutinise trial registration records as a first step in the critical appraisal and interpretation of the results [26]. Any discrepancy in the trial protocol should be reported in the published article so that the clinician can evaluate the potential for bias. Specifically to outcome reporting bias, outcome definitions should not be vague to avoid possible tampering. The standardisation of outcomes in specific clinical areas, if implemented, will reduce the potential for bias [27, 28].

Those who slip in allowing outcome reporting bias to occur should be seriously noted by the scientific community. The adoption of reporting guidelines and quality assessment tools such as those promoted by the EQUATOR Network, The Cochrane Collaboration and the GRADE working Group [29] may improve the conduct and reporting of trials. All new Cochrane reviews––like the ones about osteoarthritis [3]––will include a risk of bias assessment step to ascertain the likeliness of outcome reporting bias, following the guidelines presented in the Cochrane Handbook (Chap 8.13) [16]. Finally policy makers should defend open access to research. Formal legislation for granting public timely access to the protocols approved by research ethics committees and basic trial results, regardless of their potential publication in medical journals, should be encouraged and protected. All these efforts together will increase the trust in science.

Notes

Conflict of interest

None.

References

  1. 1.
    Nuesch E, AW Rutjes, E Husni, V Welch, P Juni (2009) Oral or transdermal opioids for osteoarthritis of the knee or hip. Cochrane Database Syst Rev CD003115Google Scholar
  2. 2.
    Rutjes AW, E Nuesch, R Sterchi, P Juni (2010) Therapeutic ultrasound for osteoarthritis of the knee or hip. Cochrane Database Syst Rev CD003132Google Scholar
  3. 3.
    Rutjes AW, E Nuesch, R Sterchi et al. (2009) Transcutaneous electrostimulation for osteoarthritis of the knee. Cochrane Database Syst Rev CD002823Google Scholar
  4. 4.
    Greenwald AG (1975) Consequences of prejudice against the null hypothesis. Psychol Bull 82:1–20CrossRefGoogle Scholar
  5. 5.
    Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr (1987) Publication bias and clinical trials. Control Clin Trials 8:343–353CrossRefPubMedGoogle Scholar
  6. 6.
    Dickersin K (2005) Publication bias: recognizing the problem, understanding its origins and scope, and preventing harm. In: Rothstein H, Sutton A, Borenstein M (eds) Publication bias in meta-analysis: prevention, assessment, and adjustments. Wiley, London, pp 11–33Google Scholar
  7. 7.
    Chan AW, Altman DG (2005) Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ 330:753CrossRefPubMedGoogle Scholar
  8. 8.
    US Government. Food and Drug Administration Amendments Act of 2007. Public Law 110:85 (September 27, 2007)Google Scholar
  9. 9.
    US National Institutes of Health. ClinicalTrials.gov [online]. http://clinicaltrial.gov/(2009)
  10. 10.
    DeAngelis CD, Drazen JM, Frizelle FA et al (2004) Clinical trial registration: a statement from the International Committee of Medical Journal Editors. JAMA 292:1363–1364CrossRefPubMedGoogle Scholar
  11. 11.
    Zarin DA, Tse T, Ide NC (2005) Trial Registration at ClinicalTrials.gov between May and October 2005. N Engl J Med 353:2779–2787CrossRefPubMedGoogle Scholar
  12. 12.
    Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA (2006) Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust 184:621–626PubMedGoogle Scholar
  13. 13.
    Williamson PR, Gamble C (2005) Identification and impact of outcome selection bias in meta-analysis. Stat Med 24:1547–1561CrossRefPubMedGoogle Scholar
  14. 14.
    Hahn S, Williamson PR, Hutton JL, Garner P, Flynn EV (2000) Assessing the potential for bias in meta-analysis due to selective reporting of subgroup analyses within studies. Stat Med 19:3325–3336CrossRefPubMedGoogle Scholar
  15. 15.
    Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R (2008) Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 358:252–260CrossRefPubMedGoogle Scholar
  16. 16.
    Higgins J (2008) Cochrane handbook for systematic reviews of interventions. Version 5.0.1Google Scholar
  17. 17.
    Moja L (2010) Clinical trials: trial registration cannot alone transform scientific conduct. Nat Rev Urol 7:7–8CrossRefPubMedGoogle Scholar
  18. 18.
    Dwan K, Altman DG, Arnaiz JA et al (2008) Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 3:e3081CrossRefPubMedGoogle Scholar
  19. 19.
    Rising K, Bacchetti P, Bero L (2008) Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med 5:e217CrossRefPubMedGoogle Scholar
  20. 20.
    Kirkham J, Dwan K, Altman DG et al (2010) The impact of outcome reporting bias on systematic reviews. BMJ 340:c356CrossRefGoogle Scholar
  21. 21.
    Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG (2004) Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 291:2457–2465CrossRefPubMedGoogle Scholar
  22. 22.
    Chan AW, Hrobjartsson A, Jorgensen KJ, Gotzsche PC, Altman DG (2008) Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols. BMJ 337:a2299CrossRefPubMedGoogle Scholar
  23. 23.
    Gotzsche PC (2006) Believability of relative risks and odds ratios in abstracts: cross-sectional study. BMJ 333:231–234CrossRefPubMedGoogle Scholar
  24. 24.
    Chapin S (1994) A social history of truth: civility and science in the seventeenth-century England. University of Chicago PressGoogle Scholar
  25. 25.
    Ioannidis JP (2005) Why most published research findings are false. PLoS Med 2:e124CrossRefPubMedGoogle Scholar
  26. 26.
    Moja LP, Moschetti I, Nurbhai M et al (2009) Compliance of clinical trial registries with the World Health Organization minimum data set: a survey. Trials 10:56CrossRefPubMedGoogle Scholar
  27. 27.
    Sinha I, Jones L, Smyth RL, Williamson PR (2008) A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children. PLoS Med 5:e96CrossRefPubMedGoogle Scholar
  28. 28.
    Clarke M (2008) Standardising outcomes in paediatric clinical trials. PLoS Med 5:e102CrossRefGoogle Scholar
  29. 29.
    Schunemann HJ, Oxman AD, Brozek J et al (2008) GRADE: assessing the quality of evidence for diagnostic recommendations. Evid Based Med 13:162–163CrossRefPubMedGoogle Scholar
  30. 30.
    Hahn S, Williamson PR, Hutton JL (2002) Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract 8:353–359CrossRefPubMedGoogle Scholar
  31. 31.
    Chan AW, Krleza-Jeric K, Schmid I, Altman DG (2004) Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 171:735–740PubMedGoogle Scholar
  32. 32.
    Von EE, Rollin A, Blumle A et al (2008) Publication and non-publication of clinical trials: longitudinal study of applications submitted to a research ethics committee. Swiss Med Wkly 138:197–203Google Scholar
  33. 33.
    Al-Marzouki S, Roberts I, Evans S, Marshall T (2008) Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet. Lancet 372:201CrossRefPubMedGoogle Scholar
  34. 34.
    Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P (2009) Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302:977–984CrossRefPubMedGoogle Scholar

Copyright information

© SIMI 2010

Authors and Affiliations

  • Ana Macura
    • 1
  • Iosief Abraha
    • 2
  • Jamie Kirkham
    • 3
  • Gian Franco Gensini
    • 4
  • Lorenzo Moja
    • 5
  • Alfonso Iorio
    • 1
    Email author
  1. 1.Sezione di Medicina Interna e Vascolare, Dipartimento di Medicina InternaUniversità di PerugiaPerugiaItaly
  2. 2.Regional Health Authority of UmbriaPerugiaItaly
  3. 3.Centre for Medical Statistics and Health EvaluationUniversity of LiverpoolLiverpoolUK
  4. 4.Department of Critical Care Medicine and SurgeryUniversity of Florence and Azienda Ospedaliero-Universitaria CareggiFlorenceItaly
  5. 5.Italian Cochrane CentreMario Negri Institute for Pharmacological ResearchMilanItaly

Personalised recommendations