The use and misuse of journal metrics and other citation indicators

Varia - Scientometrics


This article reviews the nature and use of the journal impact factor and other common bibliometric measures for assessing research in the sciences and social sciences based on data compiled by Thomson Reuters. Journal impact factors are frequently misused to assess the influence of individual papers and authors, but such uses were never intended. Thomson Reuters also employs other measures of journal influence, which are contrasted with the impact factor. Finally, the author comments on the proper use of citation data in general, often as a supplement to peer review. This review may help government policymakers, university administrators, and individual researchers become better acquainted with the potential benefits and limitations of bibliometrics in the evaluation of research.


impact factor bibliometric indicators peer review citation analysis 


  1. Banks MA, Dellavalle R (2008) Emerging alternatives to the impact factor. OCLC Systems & Services 24(3). Available via:
  2. Bar-Ilan J (2008) Informetrics at the beginning of the 21st century: a review. J Informetrics 2: 1-52CrossRefGoogle Scholar
  3. Bensman SJ (2007) Garfield and the impact factor. Annu Rev Inf Sci Technol 41: 93-155CrossRefGoogle Scholar
  4. Bensman SJ (2007) The impact factor, total citations, and better citation mouse traps: a commentary. J Am Soc Inf Sci Technol 58: 1904-1908CrossRefGoogle Scholar
  5. Bergstrom CT (2007) Eigenfactor: measuring the value and prestige of scholarly journals. C&RL News 68:5. Available via: , also see:
  6. Bollen J, Rodriguez MA, Vande Sompel H (2006) Journal status. Scientometrics 69: 669-687CrossRefGoogle Scholar
  7. Borgman C, Furner J (2002) Scholarly communication and bibliometrics. Annu Rev Inf Sci Technol 36: 3-72Google Scholar
  8. Bornmann L, Mutz R, Neuhaus C et al (2008) Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics Sci. Environmental Politics 8: 93-102CrossRefGoogle Scholar
  9. Braun T, Glanzel W, Schubert A (2005) A Hirsch-type index for journals. Scientist 19:8Google Scholar
  10. Brin S, Page L (1998) The anatomy of a large-scale hypertextual web search engine. Comput Networks ISDN Systems 30: 107-117CrossRefGoogle Scholar
  11. Butler L (2008) Using a balanced approach to bibliometrics: quantitative performance measures in the Australian Research Quality Framework. Ethics Sci Environmental Politics 8: 83-92CrossRefGoogle Scholar
  12. Egghe L, Rousseau R (2002) A general framework for relative impact factors. Can J Inf Library Sci 21: 29-38Google Scholar
  13. Egghe L (2005) Continuous, weighted Lorenz theory and applications to the study of fractional relative impact factors. Inf Processing Management 41: 1330-1359CrossRefGoogle Scholar
  14. Egghe L (2006) Theory and practice of the g-index. Scientometrics 69: 131-152CrossRefGoogle Scholar
  15. Falagas ME, Kouranos VD, Arencibia-Jorge R et al (2008) Comparison of SCImago journal rank indicator with journal impact factor. FASEB J 22: 2623-2628CrossRefPubMedGoogle Scholar
  16. Garfield E (1955) Citation indexes for science. A new dimension in documentation through association of ideas. Science 122: 108-111Google Scholar
  17. Garfield E (1972) Citation analysis as a tool in journal evaluation. Science 178: 471-479CrossRefPubMedGoogle Scholar
  18. Garfield E (1976) Significant journals of science. Nature 264: 609-615CrossRefPubMedGoogle Scholar
  19. Garfield E (1979) Citation Indexing: Its Theory and Application in Science, Technology, and Humanities. John Wiley & Sons, New YorkGoogle Scholar
  20. Garfield E (1996) How can impact factors be improved?. BMJ 313: 411-413PubMedGoogle Scholar
  21. Garfield E (1999) Journal impact factor: a brief review. CMAJ 161: 979-980PubMedGoogle Scholar
  22. Garfield E (2006) The history and meaning of the journal impact factor. JAMA 295: 90-103CrossRefPubMedGoogle Scholar
  23. Garfield E, Sher IH (1963) New factors in the evaluation of scientific literature through citation indexing. Am Documentation 14: 195-201CrossRefGoogle Scholar
  24. Glanzel W (2008) Seven myths in bibliometrics: about facts and fiction in quantitative science studies. Proceedings of the WIS 2008, Berlin. Fourth International Conference on Webometrics, Informetrics and Scientometrics & Ninth COLLNET Meeting. Available via:
  25. Glanzel W, Moed HF (2002) Journal impact measures in bibliometric research. Scientometrics 53: 171-193CrossRefGoogle Scholar
  26. Harnad S (2008) Validating research performance metrics against peer rankings. Ethics Sci. Environmental Politics 8: 103-107CrossRefGoogle Scholar
  27. Harzing A-WK, van der Wal R (2008) Google Scholar as a new source for citation analysis. Ethics Sci. Environmental Politics 8:61-73. Available via:
  28. Hirsch JE (2005) An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A 102: 16569-16572CrossRefPubMedGoogle Scholar
  29. Hirsch JE (2007) Does the h-index have predictive power?. Proc Natl Acad Sci U S A 104: 19193-19198CrossRefPubMedGoogle Scholar
  30. Jasco P (2005) As we may search: Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Curr Sci 89: 1537-1547Google Scholar
  31. Joint Committee on Quantitative Assessment of Research (2008) Citation Statistics. A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS), by Robert Adler, John Ewing (Chair), and Peter Taylor. Available via:
  32. Ketcham CM, Crawford JM (2007) The impact of review articles. Lab Invest 87: 1174-1185CrossRefPubMedGoogle Scholar
  33. Krauze T (1977) The sociology of science in Poland. In: D. A. Pendlebury: Citation indicators use and misuse Merton RK, Gaston J (eds) The Sociology of Science in Europe. Southern Illinois University Press, Carbondale, Illinois, pp. 193-223Google Scholar
  34. Leydesdorff L (2008) Caveats in the use of citation indicators in research and journal evaluations. J Am Soc Inf Sci Technol 59: 278-287CrossRefGoogle Scholar
  35. Markpin T, Boonradsamee B, Ruksinsut K et al (2008) Article-count impact factor of materials science journals in SCI database. Scientometrics 75: 251-261CrossRefGoogle Scholar
  36. Meho LI, Yang K (2007) A new era in citation and bibliometric analyses: Web of Science. Scopus, and Google Scholar. J Am Soc Inf Sci Technol 58: 2105-2125CrossRefGoogle Scholar
  37. Moed HF (2002) The impact-factors debate: the ISI’s uses and limits. Nature 415: 731-732PubMedGoogle Scholar
  38. Moed HF (2005) Citation analysis of scientific journals and journal impact measures. Curr Sci 89: 1990-1996Google Scholar
  39. Moed HF (2005) Citation analysis in research evaluation. Springer, DordrechtGoogle Scholar
  40. Moed HF, Glanzel W, Schmoch U (eds) (2004) Handbook of quantitative science and technology research. The use of publication and patent statistics in studies of S&T systems. Kluwer Academic Publishers, DordrechtGoogle Scholar
  41. Moed HF, van Leeuwen TN (1995) Improving the accuracy of Institute for Scientific Information’s journal impact factors. J Am Soc Inf Sci 46: 461-467CrossRefGoogle Scholar
  42. Moed HF, van Leeuwen TN (1996) Impact factors can mislead. Nature 381: 186CrossRefPubMedGoogle Scholar
  43. Moed HF, van Leeuwen TN, Reedijk J (1996) A critical analysis of the journal impact factors of Angewandte Chemie and Journal of the American Chemical Society: inaccuracies in published impact factors based on overall citation counts only. Scientometrics 37: 105-116CrossRefGoogle Scholar
  44. Moed HF, van Leeuwen TN, Reedijk J (1999) Towards appropriate indicators of journal impact. Scientometrics 46: 575-589CrossRefGoogle Scholar
  45. Monastersky R (2005) The number that’s devouring science. The Chronicle of Higher Education 52: A12Google Scholar
  46. Narin F (1976) Evaluative bibliometrics: the use of publication and citation analysis in the evaluation of scientific activity. Computer Horizons Inc., Cherry Hill, New JerseyGoogle Scholar
  47. Nicolaisen J (2007) Citation analysis. Annu Rev Inf Sci Technol 41: 609-641CrossRefGoogle Scholar
  48. Ossowska M, Ossowski S (1936) The science of science. Organon 1: 1-12Google Scholar
  49. Pinski G, Narin F (1976) Citation influence for journal aggregates of scientific publications: theory, with application to the literature of physics. Inf Processing Management 12: 297-312CrossRefGoogle Scholar
  50. Pudovkin AI, Garfield E (2004) Rank-normalized impact factor: a way to compare journal performance across subject categories. ASIST 2004: Proceedings of the 67th ASIS&T Annual Meeting 41:507-515Google Scholar
  51. Rousseau R (2002) Journal evaluation: technical and practical issues. Library Trends 50: 418-439Google Scholar
  52. Rousseau R (2005) Median and percentile impact factors. Scientometrics 63: 431-441CrossRefGoogle Scholar
  53. Seglen PO (1997) Why the impact factor of journals should not be used for evaluating research. BMJ 314: 498-502PubMedGoogle Scholar
  54. Sombatsompop N, Markpin T (2005) Making an equality of ISI impact factors for different subject fields. J Am Soc Inf Sci Technol 56: 676-783CrossRefGoogle Scholar
  55. Sombatsompop N, Markpin T, Premkamolnetr N (2004) A modified method for calculating the impact factors of the journals in ISI Journal Citation Reports: polymer science category in 1997-2001. Scientometrics 60: 217-235CrossRefGoogle Scholar
  56. Sombatsompop N, Markpin T, Yochai W et al (2005) An evaluation of research performance for different subject categories using impact factor point average (IFPA) index: Thailand case study. Scientometrics 65: 293-205CrossRefGoogle Scholar
  57. Stringer MJ, Sales-Pardo M, Nunes Amaral LA (2008) Effectiveness of journal ranking schemes as a tool for locating information. PLoS One 3:e1683. Available via:
  58. Vaidya JS (2005) V-index: a fairer index to quantify an individual’s research output capacity. BMJ Rapid Response. Available via:
  59. van Leeuwen TN, Moed HF (2002) Development and application of journal impact measures in the Dutch science system. Scientometrics 53: 249-266CrossRefGoogle Scholar
  60. van Leeuwen TN, Moed HF (2005) Characteristics of journal impact factors: the effects of uncitedness and citation distribution on the understanding of journal impact factors. Scientometrics 63: 357-371CrossRefGoogle Scholar
  61. van Leeuwen TN, Moed HF, Reedijk J (1999) Critical comments on Institute for Scientific Information impact factors: a sample of inorganic molecular chemistry journals. J Inf Sci 25: 489-498CrossRefGoogle Scholar
  62. Vinkler P (2004) Characterization of the impact of sets of scientific papers: the Garfield (impact) factor. J Am Soc Inf Sci Technol 55: 431-435CrossRefGoogle Scholar
  63. Wouters P (1999) The Citation Culture. University of Amsterdam, Amsterdam, pp. 82-93Google Scholar
  64. Zitt M, Bassecoulard E (2008) Challenges for scientometric indicators: data demining, knowledge-flow measurements and diversity issues. Ethics Sci. Environmental Politics 8: 49-50CrossRefGoogle Scholar
  65. Zitt M, Small H (2008) Modifying the journal impact factor by fractional citation weighting: the audience factor. J Am Soc Inf Sci Technol 59: 1856-1860CrossRefGoogle Scholar

Copyright information

© L. Hirszfeld Institute of Immunology and Experimental Therapy, Wroclaw, Poland 2009

Authors and Affiliations

  1. 1.Thomson ReutersPhiladelphiaUSA

Personalised recommendations