Abstract
Citation frequencies and journal impact factors (JIFs) are being used more and more to assess the quality of research and allocate research resources. If these bibliometric indicators are not an adequate predictor of research quality, there could be severe negative consequences for research. To analyse to which extent citation frequencies and journal impact factors correlate with the methodological quality of clinical research articles included in an SBU systematic review of antibiotic prophylaxis in surgery. All 212 eligible original articles were extracted from the SBU systematic review “Antibiotic Prophylaxis in Surgery” and categorized according to their methodological rigourness as high, moderate or low quality articles. Median of citation frequencies and JIFs were compared between the methodological quality groups using Kruskal–Wallis non-parametric test. An in-depth study of low-quality studies with higher citation frequencies/JIFs was also conducted. No significant differences were found in median citation frequencies (p = 0.453) or JIFs (p = 0.185) between the three quality groups. Studies that had high citation frequencies/JIFs but were assessed as low-quality lacked control groups, had high dropout rates or low internal validity. This study of antibiotic prophylaxis in surgery does not support the hypothesis that bibliometric indicators are a valid instrument for assessing methodological quality in clinical trials. This is a worrying observation, since bibliometric indicators have a major influence on research funding. However, further studies in other areas are needed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akre, O., Barone-Adesi, F., Pettersson, A., Pearce, N., Merletti, F., & Richiardi, L. (2011). Differences in citation rates by country of origin for papers published in top-ranked medical journals: do they reflect inequalities in access to publication? Journal of Epidemiology and Community Health, 65(2), 119–123.
Andersen, J., Belmont, J., & Cho, C. T. (2006). Journal impact factor in the era of expanding literature. Journal of Microbiology, Immunology, and Infection, 39(6), 436–443.
Banta, D., & Jonsson, E. (2009). History of HTA: Introduction. International Journal of Technology Assessment in Health Care, 25(Suppl 1), 1–6.
Barbui, C., Cipriani, A., Malvini, L., & Tansella, M. (2006). Validity of the impact factor of journals as a measure of randomized controlled trial quality. Journal of Clinical Psychiatry, 67(1), 37–40.
Bhandari, M., Busse, J., Devereaux, P. J., Montori, V. M., Swiontkowski, M., Tornetta Iii, P., et al. (2007). Factors associated with citation rates in the orthopedic literature. Canadian Journal of Surgery, 50(2), 119–123.
Callaham, M., Wears, R. L., & Weber, E. (2002). Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. Journal of the American Medical Association, 287(21), 2847–2850.
Cho, J. E., Cho, C. T., & Belmont, J. M. (1998). Learning to assess the value of infectious disease journals. Journal of Microbiology, Immunology, and Infection, 31, 1–4.
du Prel, J. B., Rohrig, B., & Blettner, M. (2009). Critical appraisal of scientific articles: part 1 of a series on evaluation of scientific publications. Deutsches Ärzteblatt International, 106(7), 100–105.
Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295(1), 90–93.
Grange, R. I. (1999). National bias in citations in urology journals: parochialism or availability? British Journal of Urology International, 84(6), 601–603.
Guyatt, G. H., Sackett, D. L., & Cook, D. J. (1993). Users’ guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. Journal of the American Medical Association, 270(21), 2598–2601.
Guyatt, G. H., Sackett, D. L., & Cook, D. J. (1994). Users’ guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. Journal of the American Medical Association, 271(1), 59–63.
Kuroki, L. M., Allsworth, J. E., & Peipert, J. F. (2009). Methodology and analytic techniques used in clinical research: associations with journal impact factor. Obstetrics and Gynecology, 114(4), 877–884.
Lee, K. P., Schotland, M., Bacchetti, P., & Bero, L. A. (2002). Association of journal quality indicators with methodological quality of clinical research articles. Journal of the American Medical Association, 287(21), 2805–2808.
Lundberg, J., Brommels, M., Skar, J., & Tomson, G. (2008). Measuring the validity of early health technology assessment: bibliometrics as a tool to indicate its scientific basis. International Journal of Technology Assessment in Health Care, 24(1), 70–75.
Nieminen, P., Carpenter, J., Rucker, G., & Schumacher, M. (2006). The relationship between quality of research and citation frequency. BMC Medical Research Methodology, 6, 42.
SBU. (2010). Antibiotic prophylaxis for surgical procedures. A systematic review. Stockholm: Statens beredning för medicinsk utvärdering (SBU); SBU-rapport nr 200. ISBN 978-91-85413-36–2.
West, R., & McIlwaine, A. (2002). What do citation counts count for in the field of addiction? An empirical evaluation of citation counts and their link with peer ratings of quality. Addiction, 97(5), 501–504.
Conflict of Interest
All authors declare no conflict of interest.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Akcan, D., Axelsson, S., Bergh, C. et al. Methodological quality in clinical trials and bibliometric indicators: no evidence of correlations. Scientometrics 96, 297–303 (2013). https://doi.org/10.1007/s11192-013-0949-0
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-013-0949-0