European Radiology

, Volume 29, Issue 4, pp 1657–1664 | Cite as

Citation bias in imaging research: are studies with higher diagnostic accuracy estimates cited more often?

  • Robert A. Frank
  • Anahita Dehmoobad Sharifabadi
  • Jean-Paul Salameh
  • Trevor A. McGrath
  • Noémie Kraaijpoel
  • Wilfred Dang
  • Nicole Li
  • Isabelle D. Gauthier
  • Mark Z. Wu
  • Patrick M. Bossuyt
  • Deborah Levine
  • Matthew D. F. McInnesEmail author



To assess the risk of citation bias in imaging diagnostic accuracy research by evaluating whether studies with higher accuracy estimates are cited more frequently than those with lower accuracy estimates.


We searched Medline for diagnostic accuracy meta-analyses published in imaging journals from January 2005 to April 2016. Primary studies from the meta-analyses were screened; those assessing the diagnostic accuracy of an imaging test and reporting sensitivity and specificity were eligible for inclusion. Studies not indexed in Web of Science, duplicates, and inaccessible articles were excluded. Topic (modality/subspecialty), study design, sample size, journal impact factor, publication date, times cited, sensitivity, and specificity were extracted for each study. Negative binomial regression was performed to evaluate the association of citation rate (times cited per month since publication) with Youden’s index (sensitivity + specificity -1), highest sensitivity, and highest specificity, controlling for the potential confounding effects of modality, subspecialty, impact factor, study design, sample size, and source meta-analysis.


There were 1016 primary studies included. A positive association between Youden’s index and citation rate was present, with a regression coefficient of 0.33 (p = 0.016). The regression coefficient for sensitivity was 0.41 (p = 0.034), and for specificity, 0.32 (p = 0.15).


A positive association exists between diagnostic accuracy estimates and citation rates, indicating that there is evidence of citation bias in imaging diagnostic accuracy literature. Overestimation of imaging test accuracy may contribute to patient harm from incorrect interpretation of test results.

Key Points

• Studies with higher accuracy estimates may be cited more frequently than those with lower accuracy estimates.

This citation bias could lead clinicians, reviews, and clinical practice guidelines to overestimate the accuracy of imaging tests, contributing to patient harm from incorrect interpretation of test results.


Diagnostic test Routine Publication bias Bibliometrics Sensitivity and specificity 



Conventional radiography


Computed tomography


Diagnostic test accuracy






Magnetic resonance imaging


Nuclear medicine/other


Positron emission tomography









This study has received funding by the University of Ottawa, Department of Radiology Research Stipend Program.

Compliance with ethical standards


The scientific guarantor of this publication is Matthew McInnes.

Conflict of interest

The authors declare that they have no conflict of interest.

Statistics and biometry

One of the authors has significant statistical expertise (Drs McInnes and Bossuyt).

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was not required because this study is an evaluation of published literature.


• Retrospective

• Cross sectional study

• Multicenter study

Supplementary material

330_2018_5801_MOESM1_ESM.pdf (148 kb)
ESM 1 (PDF 148 kb)
330_2018_5801_MOESM2_ESM.docx (5.6 mb)
ESM 2 (DOCX 5683 kb)
330_2018_5801_MOESM3_ESM.xlsx (34 kb)
ESM 3 (XLSX 33 kb)


  1. 1.
    Duyx B, Urlings MJE, Swaen GMH, Bouter LM, Zeegers MP (2017) Scientific citations favor positive results: a systematic review and meta-analysis. J Clin Epidemiol 88:92–101CrossRefGoogle Scholar
  2. 2.
    Jannot AS, Agoritsas T, Gayet-Ageron A, Perneger TV (2013) Citation bias favoring statistically significant studies was present in medical research. J Clin Epidemiol 66:296–301CrossRefGoogle Scholar
  3. 3.
    Greenberg SA (2009) How citation distortions create unfounded authority: analysis of a citation network. BMJ 339:b2680CrossRefGoogle Scholar
  4. 4.
    Seglen PO (1997) Citations and journal impact factors: questionable indicators of research quality. Allergy 52:1050–1056CrossRefGoogle Scholar
  5. 5.
    Steneck NH (2006) Fostering integrity in research: definitions, current knowledge, and future directions. Sci Eng Ethics 12:53–74CrossRefGoogle Scholar
  6. 6.
    Hutchison BG, Oxman AD, Lloyd S (1995) Comprehensiveness and bias in reporting clinical trials. Study of reviews of pneumococcal vaccine effectiveness. Can Fam Physician 41:1356–1360Google Scholar
  7. 7.
    Ravnskov U (1992) Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ 305:15–19CrossRefGoogle Scholar
  8. 8.
    Misemer BS, Platts-Mills TF, Jones CW (2016) Citation bias favoring positive clinical trials of thrombolytics for acute ischemic stroke: a cross-sectional analysis. Trials 17:473CrossRefGoogle Scholar
  9. 9.
    Kjaergard LL, Gluud C (2002) Citation bias of hepato-biliary randomized clinical trials. J Clin Epidemiol 55:407–410CrossRefGoogle Scholar
  10. 10.
    Korevaar DA, Cohen JF, Spijker R et al (2016) Reported estimates of diagnostic accuracy in ophthalmology conference abstracts were not associated with full-text publication. J Clin Epidemiol 79:96–103CrossRefGoogle Scholar
  11. 11.
    Korevaar DA, van Es N, Zwinderman AH, Cohen JF, Bossuyt PM (2016) Time to publication among completed diagnostic accuracy studies: associated with reported accuracy estimates. BMC Med Res Methodol 16:68CrossRefGoogle Scholar
  12. 12.
    Sharifabadi AD, Korevaar DA, McGrath TM, et al (2018) Reporting bias in imaging: higher accuracy is linked to faster publication. Eur Radiol 28:3632−3639Google Scholar
  13. 13.
    Frank RA, McInnes MDF (2018) Citation bias in imaging research: are studies with higher diagnostic accuracy estimates cited more often? Study protocol. Retrieved from Scholar
  14. 14.
    Frank RA, McInnes MDF, Levine D et al (2017) Are study and journal characteristics reliable indicators of ‘truth’ in imaging research? Radiology 287:215–223CrossRefGoogle Scholar
  15. 15.
    McGrath TA, McInnes MD, Korevaar DA, Bossuyt PM (2016) Meta-analyses of diagnostic accuracy in imaging journals: analysis of pooling techniques and their effect on summary estimates of diagnostic accuracy. Radiology 281:78–85CrossRefGoogle Scholar
  16. 16.
    van Enst WA, Ochodo E, Scholten RJ, Hooft L, Leeflang MM (2014) Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study. BMC Med Res Methodol 14:70CrossRefGoogle Scholar
  17. 17.
    Devillé WL, Bezemer PD, Bouter LM (2000) Publications on diagnostic test evaluation in family medicine journals: an optimal search strategy. J Clin Epidemiol 53:65–69CrossRefGoogle Scholar
  18. 18.
    Reuters T Science citation index expanded-radiology, nuclear medicine & medical imaging-journal list. Available via: Accessed 15 Apr 2015
  19. 19.
    R Core Team (2017) R: A language and environment forstatistical computing. R Foundation for Statistical Computing, V., Austria. Available via Accessed 1 Aug 2018
  20. 20.
    Nieminen P, Carpenter J, Rucker G, Schumacher M (2006) The relationship between quality of research and citation frequency. BMC Med Res Methodol 6:42CrossRefGoogle Scholar
  21. 21.
    Kulkarni AV, Busse JW, Shams I (2007) Characteristics associated with citation rate of the medical literature. PLoS One 2:403CrossRefGoogle Scholar
  22. 22.
    Sawin VI, Robinson KA (2016) Biased and inadequate citation of prior research in reports of cardiovascular trials is a continuing source of waste in research. J Clin Epidemiol 69:174–178CrossRefGoogle Scholar
  23. 23.
    Sheth U, Simunovic N, Tornetta P 3rd, Einhorn TA, Bhandari M (2011) Poor citation of prior evidence in hip fracture trials. J Bone Joint Surg Am 93:2079–2086Google Scholar
  24. 24.
    Etter JF, Stapleton J (2009) Citations to trials of nicotine replacement therapy were biased toward positive results and high-impact-factor journals. J Clin Epidemiol 62:831–837CrossRefGoogle Scholar
  25. 25.
    Nieminen P, Rucker G, Miettunen J, Carpenter J, Schumacher M (2007) Statistically significant papers in psychiatry were cited more often than others. J Clin Epidemiol 60:939–946CrossRefGoogle Scholar
  26. 26.
    Saha S, Saint A, Christakis DA (2003) Impact factor: a valid measure of journal quality? J Med Libr Assoc 9:42–46Google Scholar
  27. 27.
    Dilauro M, McInnes MD, Korevaar DA et al (2016) Is there an association between STARD statement adherence and citation rate? Radiology 280:62–67Google Scholar
  28. 28.
    Choi YJ, Chung MS, Koo HJ, Park JE, Yoon HM, Park SH (2016) Does the reporting quality of diagnostic test accuracy studies, as defined by STARD 2015, affect citation? Korean J Radiol 17:706–714CrossRefGoogle Scholar
  29. 29.
    McInnes MDF, Moher D, Thombs BD et al (2018) Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: the PRISMA-DTA statement. JAMA 319:388–396CrossRefGoogle Scholar
  30. 30.
    McGrath TA, McInnes MDF, van Es N, Leeflang MMG, Korevaar DA, Bossuyt PMM (2017) Overinterpretation of research findings: evidence of “spin” in systematic reviews of diagnostic accuracy studies. Clin Chem 63:1353–1362CrossRefGoogle Scholar
  31. 31.
    Ochodo EA, de Haan MC, Reitsma JB, Hooft L, Bossuyt PM, Leeflang MM (2013) Overinterpretation and misreporting of diagnostic accuracy studies: evidence of “spin”. Radiology 267:581–588CrossRefGoogle Scholar

Copyright information

© European Society of Radiology 2018

Authors and Affiliations

  • Robert A. Frank
    • 1
    • 2
  • Anahita Dehmoobad Sharifabadi
    • 1
    • 2
  • Jean-Paul Salameh
    • 2
    • 3
  • Trevor A. McGrath
    • 1
    • 2
  • Noémie Kraaijpoel
    • 4
    • 5
  • Wilfred Dang
    • 1
    • 2
  • Nicole Li
    • 1
    • 2
  • Isabelle D. Gauthier
    • 1
    • 2
  • Mark Z. Wu
    • 1
    • 2
  • Patrick M. Bossuyt
    • 4
  • Deborah Levine
    • 6
  • Matthew D. F. McInnes
    • 2
    • 3
    Email author
  1. 1.Faculty of MedicineUniversity of OttawaOttawaCanada
  2. 2.Department of RadiologyUniversity of OttawaOttawaCanada
  3. 3.Clinical Epidemiology ProgramOttawa Hospital Research InstituteOttawaCanada
  4. 4.Clinical Epidemiology and Biostatistics and Bioinformatics, Amsterdam Public Health Research Institute, Academic Medical CenterUniversity of AmsterdamAmsterdamNetherlands
  5. 5.Department of Vascular MedicineAcademic Medical CenterAmsterdamthe Netherlands
  6. 6.Department of RadiologyBeth Israel Deaconess Medical CenterBostonUSA

Personalised recommendations