Advertisement

Journal of General Internal Medicine

, Volume 33, Issue 3, pp 247–248 | Cite as

A Cross-Sectional Analysis of Spin in Randomized Controlled Trials

  • Alexandra Woodbridge
  • Ann Abraham
  • Rosa Ahn
  • Susan Saba
  • Deborah Korenstein
  • Erin Madden
  • Salomeh KeyhaniEmail author
Concise Research Reports

INTRODUCTION

Randomized controlled trials (RCTs) are the most reliable form of evidence for evaluating drug safety and efficacy. Because clinicians rely on RCTs to inform clinical practice, accurate representation of clinical trial results is important to patient health and safety. Spin, defined as reporting that distorts results or misleads the reader,1 threatens accurate evidence interpretation and application by clinicians. Given that many clinicians obtain information from the study abstract only, spin in the abstract is concerning.2 We examined the prevalence of spin among abstracts in a random sample of trials focused on the efficacy of drugs and examined the association of spin with study characteristics.

METHODS

We searched Medline for all English-language RCTs examining drug efficacy published in 2013. Our search identified 2851 potential studies. After screening titles and abstracts, 1101 potential studies remained. We randomly selected 646 of this sample, of which 190 studies were focused on drug efficacy and met inclusion criteria. For each study, we extracted information on outcome, funding source, financial ties to the manufacturer, and design characteristics (Table 1). The methods are described in full in a prior publication.3
Table 1

Prevalence of Spin by Study Characteristics (N = 59)

 

All studies

(n = 59)

N

Spin present

(n = 25)

N (%)

Spin absent

(n = 34)

N (%)

p-value

Funding source

   

0.89

 Industry involvement

36

15 (42)

21 (58)

 

 No industry involvement

23

10 (43)

13 (57)

 

Financial ties of PI present

   

0.08

 Yes

29

9 (31)

20 (69)

 

 No

30

16 (53)

14 (47)

 

Any relationship with manufacturer (funding and/or financial ties)

 Yes

41

17 (42)

24 (58)

0.83

 No

18

8 (44)

10 (56)

 

Sample size

   

0.01*

 Q1 (24–109)

15

10 (67)

5 (33)

 

 Q2 (110–270)

15

5 (33)

10 (67)

 

 Q3 (271–410)

14

8 (57)

6 (43)

 

 Q4 (411–13,229)

15

2 (13)

13 (87)

 

First author affiliation by continent

   

0.17

 Europe

22

10 (45)

12 (55)

 

 North America

27

8 (30)

19 (70)

 

 Asia

7

5 (71)

2 (29)

 

 Other

3

2 (67)

1 (33)

 

First author affiliation by country

   

0.35

 USA

23

8 (35)

15 (65)

 

 Other

36

17 (47)

19 (53)

 

Specialty

   

0.63

 Cardiology

9

3 (33)

6 (67)

 

 Oncology

7

4 (57)

3 (43)

 

 Other

43

18 (42)

25 (58)

 

Trial registration

   

0.69

 Yes

53

22 (42)

31(58)

 

 No

6

3 (50)

3 (50)

 

RCT type

   

0.93

 Phase 2

21

9 (43)

12 (57)

 

 Phase 3

26

10 (38)

16 (62)

 

 Phase 4

4

2 (50)

2 (50)

 

 Other

8

4 (50)

4 (50)

 

Type of analysis

   

0.24

 Superiority

58

24 (41)

34 (59)

 

 Non-inferiority

1

1 (100)

0 (0)

 

Comparator

   

0.59

 Placebo

49

20 (41)

29 (59)

 

 Active

10

5 (50)

5 (50)

 

Outcome measure

   

0.20

 Clinical

45

17 (38)

28 (62)

 

 Surrogate

14

8 (57)

6 (43)

 

Blinding

   

0.40

 Double-blinded

46

18 (39)

28 (61)

 

 Open-label

12

6 (50)

6 (50)

 

 Single-blinded

1

1 (100)

0 (0)

 

*Kruskal–Wallis p-value of 0.03 based on medians for studies with spin (201) vs. those without (352)

PI principal investigator

Identification of RCT Outcome

Trained abstractors used the results section of each study to determine whether the study reported positive or negative primary efficacy outcome. For superiority studies, if the drug of interest was statistically superior to the control (e.g., p-value < 0.05), the study outcome was defined as positive. For non-inferiority studies, if the drug of interest and the control had no significant difference, the study outcome was defined as positive. Study outcomes were assessed independently and in duplicate. Any disagreement on study outcome was resolved by discussion among the research team.

Outcome Measure

The main outcome variable considered was the presence or absence of spin in the abstract of the RCT. We considered spin present if the abstract outcome was positive or mixed and the study reported a negative primary efficacy outcome in the results section of the manuscript.1 Two clinician reviewers (SK, DK) evaluated all abstract conclusion sections, rating each conclusion as positive (in favor of study drug), negative (neutral or in favor of control), or mixed. A mixed rating meant that it was unclear in the abstract whether the study drug or the control was favored (i.e., if a subgroup analysis was emphasized over the primary outcome). The two clinicians remained blinded to the study outcome during this discussion.

Analysis

We report the prevalence of spin in a sample of RCTs. We examined the association between spin and study characteristics using a two-sided, 0.05-level χ2 test of significance. Statistical analysis was performed using SAS version 9 statistical software (SAS Institute Inc., Cary, NC).

RESULTS

Of the 190 RCTs identified, 59 had a negative primary outcome in the results. These 59 studies were evaluated for the presence of spin. Among the 59 studies, clinician reviewers rated 8 abstracts as having a positive outcome and 17 as having a mixed outcome, for a total of 25 (42%) abstracts with spin. Study characteristics were largely similar across studies with and without spin (Table 1). Overall, studies with spin had smaller samples (median: 201) than studies without spin (median: 352; p = 0.03). There was no relationship between any financial tie to the manufacturer and presence of spin in the abstract (p = 0.83).

DISCUSSION

Nearly half of abstracts of RCTs focused on drug efficacy that report negative results contain spin. We did not find an association between spin and financial ties to industry, but our study may be underpowered to detect this association. Many clinicians do not read beyond the abstract, and many readers of the literature may not have the skill to critically analyze a trial themselves to combat spin or other bias in the report.4, 5, 6 Given the widespread reliance on the abstract, the peer review process needs to be improved to reduce spin in abstracts. A simple prompt asking reviewers to comment on the presentation of the results of the study with a specific question about spin may help focus reviewer attention on this issue. Editors can also review for spin in the editorial decision process. These simple steps that allow more scrutiny of the abstract and provide feedback to authors may reduce, if not eliminate, spin in the literature.

Notes

Funding

This project was not directly supported by any research funds. Dr. Keyhani is funded by grants from the NIH (grants RO1 HL116522-01A1, RO1 HL114563-01A1) and VA HSR&D (1IP1HX001994). Dr. Korenstein’s work on this paper was supported by a Cancer Center Support Grant from the National Cancer Institute to Memorial Sloan Kettering Cancer Center (award number P30 CA008748).

Compliance with Ethical Standards

Prior Presentations

SGIM meeting April 21, 2017.

Conflict of Interest

All authors declare that they have no conflict of interest.

Ethical approval

Not needed.

Data sharing

Data set available from corresponding author on request.

Transparency

The manuscript’s guarantor (SK) affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is noncommercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

References

  1. 1.
    Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and Interpretation of Randomized Controlled Trials With Statistically Nonsignificant Results for Primary Outcomes. JAMA. 2010; 303(20): 2058–64. doi:  https://doi.org/10.1001/jama.2010.651 CrossRefPubMedGoogle Scholar
  2. 2.
    Saint S, Christakis DA, Saha S, et al. Journal reading habits of internists. J Gen Intern Med. 2000;15(12):881–4. doi: https://doi.org/10.1046/j.1525-1497.2000.00202.x CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Ahn R, Woodbridge A, Abraham A, et al. Financial ties of principal investigators and randomized controlled trial outcomes: cross sectional study. BMJ. 2017;356:i6770. doi:  https://doi.org/10.1136/bmj.i6770.CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Caverly TJ, Matlock DD, Prochazka AV, Lucas BP, Hayward RA. Interpreting Clinical Trial Outcomes for Optimal Patient Care: A Survey of Clinicians and Trainees. J Grad Med Educ. 2016;8(1):57–62. doi:  https://doi.org/10.4300/JGME-D-15-00137.1.CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Johnson TV, Abbasi A, Schoenberg ED, et al. Numeracy among trainees: are we preparing physicians for evidence-based medicine? J Surg Educ. 2014;71(2):211–5. doi:  https://doi.org/10.1016/j.jsurg.2013.07.013.CrossRefPubMedGoogle Scholar
  6. 6.
    Korenstein D. Blinding Them With Science? Evidence-Based Medicine as a Barrier to Health Care Value. J Grad Med Educ 2016;8(1):106–8. doi:  https://doi.org/10.4300/JGME-D-15-00570.1.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Society of General Internal Medicine (outside the USA) 2018

Authors and Affiliations

  • Alexandra Woodbridge
    • 1
  • Ann Abraham
    • 2
  • Rosa Ahn
    • 3
  • Susan Saba
    • 4
  • Deborah Korenstein
    • 5
  • Erin Madden
    • 2
  • Salomeh Keyhani
    • 2
    • 6
    Email author
  1. 1.Tulane University School of MedicineNew OrleansUSA
  2. 2.San Francisco VA Medical CenterSan FranciscoUSA
  3. 3.Oregon Health & Science UniversityPortlandUSA
  4. 4.Stanford University School of MedicinePalo AltoUSA
  5. 5.Memorial Sloan Kettering Cancer CenterNew YorkUSA
  6. 6.University of California at San FranciscoSan FranciscoUSA

Personalised recommendations