Advertisement

Scientometrics

, Volume 96, Issue 1, pp 297–303 | Cite as

Methodological quality in clinical trials and bibliometric indicators: no evidence of correlations

  • Derya Akcan
  • Susanna Axelsson
  • Christina Bergh
  • Thomas Davidson
  • Måns Rosén
Article

Abstract

Citation frequencies and journal impact factors (JIFs) are being used more and more to assess the quality of research and allocate research resources. If these bibliometric indicators are not an adequate predictor of research quality, there could be severe negative consequences for research. To analyse to which extent citation frequencies and journal impact factors correlate with the methodological quality of clinical research articles included in an SBU systematic review of antibiotic prophylaxis in surgery. All 212 eligible original articles were extracted from the SBU systematic review “Antibiotic Prophylaxis in Surgery” and categorized according to their methodological rigourness as high, moderate or low quality articles. Median of citation frequencies and JIFs were compared between the methodological quality groups using Kruskal–Wallis non-parametric test. An in-depth study of low-quality studies with higher citation frequencies/JIFs was also conducted. No significant differences were found in median citation frequencies (p = 0.453) or JIFs (p = 0.185) between the three quality groups. Studies that had high citation frequencies/JIFs but were assessed as low-quality lacked control groups, had high dropout rates or low internal validity. This study of antibiotic prophylaxis in surgery does not support the hypothesis that bibliometric indicators are a valid instrument for assessing methodological quality in clinical trials. This is a worrying observation, since bibliometric indicators have a major influence on research funding. However, further studies in other areas are needed.

Keywords

Bibliometric analysis Citations Impact factor Study quality HTA 

Notes

Conflict of Interest

All authors declare no conflict of interest.

References

  1. Akre, O., Barone-Adesi, F., Pettersson, A., Pearce, N., Merletti, F., & Richiardi, L. (2011). Differences in citation rates by country of origin for papers published in top-ranked medical journals: do they reflect inequalities in access to publication? Journal of Epidemiology and Community Health, 65(2), 119–123.CrossRefGoogle Scholar
  2. Andersen, J., Belmont, J., & Cho, C. T. (2006). Journal impact factor in the era of expanding literature. Journal of Microbiology, Immunology, and Infection, 39(6), 436–443.Google Scholar
  3. Banta, D., & Jonsson, E. (2009). History of HTA: Introduction. International Journal of Technology Assessment in Health Care, 25(Suppl 1), 1–6.CrossRefGoogle Scholar
  4. Barbui, C., Cipriani, A., Malvini, L., & Tansella, M. (2006). Validity of the impact factor of journals as a measure of randomized controlled trial quality. Journal of Clinical Psychiatry, 67(1), 37–40.CrossRefGoogle Scholar
  5. Bhandari, M., Busse, J., Devereaux, P. J., Montori, V. M., Swiontkowski, M., Tornetta Iii, P., et al. (2007). Factors associated with citation rates in the orthopedic literature. Canadian Journal of Surgery, 50(2), 119–123.Google Scholar
  6. Callaham, M., Wears, R. L., & Weber, E. (2002). Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. Journal of the American Medical Association, 287(21), 2847–2850.CrossRefGoogle Scholar
  7. Cho, J. E., Cho, C. T., & Belmont, J. M. (1998). Learning to assess the value of infectious disease journals. Journal of Microbiology, Immunology, and Infection, 31, 1–4.Google Scholar
  8. du Prel, J. B., Rohrig, B., & Blettner, M. (2009). Critical appraisal of scientific articles: part 1 of a series on evaluation of scientific publications. Deutsches Ärzteblatt International, 106(7), 100–105.Google Scholar
  9. Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295(1), 90–93.CrossRefGoogle Scholar
  10. Grange, R. I. (1999). National bias in citations in urology journals: parochialism or availability? British Journal of Urology International, 84(6), 601–603.CrossRefGoogle Scholar
  11. Guyatt, G. H., Sackett, D. L., & Cook, D. J. (1993). Users’ guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. Journal of the American Medical Association, 270(21), 2598–2601.CrossRefGoogle Scholar
  12. Guyatt, G. H., Sackett, D. L., & Cook, D. J. (1994). Users’ guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. Journal of the American Medical Association, 271(1), 59–63.CrossRefGoogle Scholar
  13. Kuroki, L. M., Allsworth, J. E., & Peipert, J. F. (2009). Methodology and analytic techniques used in clinical research: associations with journal impact factor. Obstetrics and Gynecology, 114(4), 877–884.CrossRefGoogle Scholar
  14. Lee, K. P., Schotland, M., Bacchetti, P., & Bero, L. A. (2002). Association of journal quality indicators with methodological quality of clinical research articles. Journal of the American Medical Association, 287(21), 2805–2808.CrossRefGoogle Scholar
  15. Lundberg, J., Brommels, M., Skar, J., & Tomson, G. (2008). Measuring the validity of early health technology assessment: bibliometrics as a tool to indicate its scientific basis. International Journal of Technology Assessment in Health Care, 24(1), 70–75.CrossRefGoogle Scholar
  16. Nieminen, P., Carpenter, J., Rucker, G., & Schumacher, M. (2006). The relationship between quality of research and citation frequency. BMC Medical Research Methodology, 6, 42.CrossRefGoogle Scholar
  17. SBU. (2010). Antibiotic prophylaxis for surgical procedures. A systematic review. Stockholm: Statens beredning för medicinsk utvärdering (SBU); SBU-rapport nr 200. ISBN 978-91-85413-36–2.Google Scholar
  18. West, R., & McIlwaine, A. (2002). What do citation counts count for in the field of addiction? An empirical evaluation of citation counts and their link with peer ratings of quality. Addiction, 97(5), 501–504.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2013

Authors and Affiliations

  • Derya Akcan
    • 1
  • Susanna Axelsson
    • 1
  • Christina Bergh
    • 2
    • 5
  • Thomas Davidson
    • 1
    • 6
  • Måns Rosén
    • 3
    • 4
  1. 1.Swedish Council on Health Technology Assessment (SBU)StockholmSweden
  2. 2.Department of Obstetrics and Gynaecology, Institute of Clinical Sciences, Sahlgrenska AcademyGothenburg UniversityGothenburgSweden
  3. 3.Swedish Council on Health Technology Assessment (SBU)StockholmSweden
  4. 4.Department of Learning, InformaticsManagement and Ethics (LIME), Karolinska InstitutetStockholmSweden
  5. 5.Reproductive MedicineSahlgrenska University HospitalGothenburgSweden
  6. 6.Center for Medical Technology Assessment (CMT)Linköping UniversityLinköpingSweden

Personalised recommendations