Measuring the Impact of Scientific Research

  • Pali U. K. De SilvaEmail author
  • Candace K. Vance
Part of the Fascinating Life Sciences book series (FLS)


The body of scientific knowledge grows with incremental additions. Assessing the scientific quality and the impact of these contributions is necessary because future scientific research is based on previous knowledge. As the key literature consulted and influenced their work should be cited when researchers publish findings, measures based on citations metrics became the most widely accepted impact assessment tools, and citation analysis is considered an objective means to evaluate scholarly publications . Historical developments, strengths, and limitations in citation-based assessment tools, use of impact factor in measuring the scientific quality of scholarly journals, and use, misuse, and manipulation of the journal impact factor are examined in this chapter. The discussion also includes citation indexes and related issues, and other journal ranking systems. Assessing the performance of individual scientists using citation metrics , the Hirsch index , and many variations proposed to correct its deficiencies are discussed. Although citation metrics can be considered the best tools yet implemented to assess the quality and influence of scientific research, the importance of understanding their strengths, limitations, and implications when using them is stressed.


Scientific scholarly impact Citation analysis Bibliometrics Journal impact factor Hirsch index Self citation 


  1. Abbott, A., Cyranoski, D., Jones, N., Maher, B., Schiermeier, Q., & Van Noorden, R. (2010). Metrics: Do metrics matter? Nature News, 465(7300), 860–862.CrossRefGoogle Scholar
  2. Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2009). h-index: A review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3(4), 273–289. doi: 10.1016/j.joi.2009.04.001.CrossRefGoogle Scholar
  3. Banks, M. G. (2006). An extension of the Hirsch index: Indexing scientific topics and compounds. Scientometrics, 69(1), 161–168.CrossRefGoogle Scholar
  4. Basken, P. (2016). Better than impact factor? NIH team claims key advance in ranking journal articles. The Chronicle of Higher Education. Retrieved from
  5. Bensman, S. J. (2007). Garfield and the impact factor. Annual Review of Information Science and Technology, 41(1), 93–155.CrossRefGoogle Scholar
  6. Bergstrom, C. T., West, J. D., & Wiseman, M. A. (2008). The Eigenfactor™ metrics. The Journal of Neuroscience, 28(45), 11433–11434.CrossRefPubMedGoogle Scholar
  7. Bollen, J., Rodriquez, M. A., & Van de Sompel, H. (2006). Journal status. Scientometrics, 69(3), 669–687.CrossRefGoogle Scholar
  8. Bornmann, L., & Marx, W. (2014). How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics, 98(1), 487–509.CrossRefGoogle Scholar
  9. Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H.-D. (2011). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5(3), 346–359.CrossRefGoogle Scholar
  10. Bucur, O., Almasan, A., Zubarev, R., Friedman, M., Nicolson, G. L., Sumazin, P., … Kunej, T. (2015). An updated h-index measures both the primary and total scientific output of a researcher. Discoveries, 3(3).Google Scholar
  11. Burrell, Q. L. (2007). Hirsch’s h-index: A stochastic model. Journal of Informetrics, 1(1), 16–25.CrossRefGoogle Scholar
  12. Cantín, M., Muñoz, M., Roa, I., CantÍN, M., MuÑOz, M., & Roa, I. (2015). Comparison between impact factor, Eigenfactor score, and SCImago journal rank indicator in anatomy and morphology journals. International Journal of Morphology, 33(3), 1183–1188.CrossRefGoogle Scholar
  13. Chew, M., Villanueva, E. V., & Van Der Weyden, M. B. (2007). Life and times of the impact factor: retrospective analysis of trends for seven medical journals (1994–2005) and their Editors’ views. Journal of the Royal Society of Medicine, 100(3), 142–150. doi: 10.1258/jrsm.100.3.142.CrossRefPubMedPubMedCentralGoogle Scholar
  14. Durieux, V., & Gevenois, P. A. (2010). Bibliometric indicators: Quality measurements of scientific publication 1. Radiology, 255(2), 342–351.CrossRefPubMedGoogle Scholar
  15. Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69(1), 131–152.CrossRefGoogle Scholar
  16. Egghe, L. (2008). Mathematical theory of the h-and g-index in case of fractional counting of authorship. Journal of the American Society for Information Science and Technology, 59(10), 1608–1616.CrossRefGoogle Scholar
  17. Egghe, L., & Rao, I. K. (2008). Study of different h-indices for groups of authors. Journal of the American Society for Information Science and Technology, 59(8), 1276–1281.CrossRefGoogle Scholar
  18. Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., & Karageorgopoulos, D. E. (2008). Comparison of SCImago journal rank indicator with journal impact factor. The FASEB Journal, 22(8), 2623–2628.CrossRefPubMedGoogle Scholar
  19. Fassoulaki, A., Paraskeva, A., Papilas, K., & Karabinis, G. (2000). Self-citations in six anaesthesia journals and their significance in determining the impact factor. British Journal of Anaesthesia, 84(2), 266–269.CrossRefPubMedGoogle Scholar
  20. Garfield, E. (1955). Citation indexes for science—new dimension in documentation through association of ideas. Science, 122(3159), 108–111. doi: 10.1126/science.122.3159.108.CrossRefPubMedGoogle Scholar
  21. Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science (New York, NY), 178(4060), 471.CrossRefGoogle Scholar
  22. Garfield, E. (1976). Significant journals of science. Nature, 264, 609–615.CrossRefPubMedGoogle Scholar
  23. Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161(8), 979–980.PubMedPubMedCentralGoogle Scholar
  24. Garfield, E., & Merton, R. K. (1979). Citation indexing: Its theory and application in science, technology, and humanities (Vol. 8). New York: Wiley.Google Scholar
  25. Garfield, E., & Sher, I. H. (1963). New factors in the evaluation of scientific literature through citation indexing. American documentation, 14(3), 195–201.CrossRefGoogle Scholar
  26. Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193.CrossRefGoogle Scholar
  27. González-Pereira, B., Guerrero-Bote, V. P., & Moya-Anegón, F. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of informetrics, 4(3), 379–391.CrossRefGoogle Scholar
  28. Hakkalamani, S., Rawal, A., Hennessy, M. S., & Parkinson, R. W. (2006). The impact factor of seven orthopaedic journals factors influencing it. Journal of Bone & Joint Surgery, British, 88(2), 159–162.CrossRefGoogle Scholar
  29. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572. doi: 10.1073/pnas.0507655102.CrossRefPubMedPubMedCentralGoogle Scholar
  30. Hutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M. (2016). Relative Citation Ratio (RCR): A new metric that uses citation rates to measure influence at the article level. bioRxiv, 029629.Google Scholar
  31. Imperial, J., & Rodríguez-Navarro, A. (2007). Usefulness of Hirsch’s h-index to evaluate scientific research in Spain. Scientometrics, 71(2), 271–282.CrossRefGoogle Scholar
  32. Jin, B., Liang, L., Rousseau, R., & Egghe, L. (2007). The R-and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52(6), 855–863.CrossRefGoogle Scholar
  33. King, J. (1987). A review of bibliometric and other science indicators and their role in research evaluation. Journal of information science, 13(5), 261–276.CrossRefGoogle Scholar
  34. Kirchhof, B., Bornfeld, N., & Grehn, F. (2006). The delicate topic of the impact factor. Graefe’s Archive for Clinical and Experimental Ophthalmology, 245(7), 925–927.CrossRefGoogle Scholar
  35. Kostoff, R. N. (2002). Citation analysis of research performer quality. Scientometrics, 53(1), 49–71.CrossRefGoogle Scholar
  36. Krell, F.-T. (2010). Should editors influence journal impact factors? Learned Publishing, 23(1), 59–62.CrossRefGoogle Scholar
  37. Kurmis, A. P. (2003). Current concepts review—understanding the limitations of the journal impact factor. Journal of Bone and Joint Surgery-American, 85A(12), 2449–2454.CrossRefGoogle Scholar
  38. Leydesdorff, L., & Opthof, T. (2010). Normalization at the field level: Fractional counting of citations. Journal of Informetrics, 4(4), 644–646.CrossRefGoogle Scholar
  39. MacRoberts, M. H., & MacRoberts, B. F. (1989). Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40(5), 342–349.CrossRefGoogle Scholar
  40. MacRoberts, M. H., & MacRoberts, B. R. (2010). Problems of citation analysis: A study of uncited and seldom-cited influences. Journal of the American Society for Information Science and Technology, 61(1), 1–12.CrossRefGoogle Scholar
  41. Mavrogenis, A. F., Ruggieri, P., & Papagelopoulos, P. J. (2010). Self-citation in publishing (editorial). Clinical Orthopaedics and Related Research, 468(10), 2803–2807.CrossRefPubMedPubMedCentralGoogle Scholar
  42. McVeigh, M. E. (2004). Journal self-citation in the Journal Citation Reports®-Science Edition (2002): A Citation Study from The Thomson Corporation. Retrieved from
  43. Miguel, A., & Martí-Bonmatí, L. (2002). Self-citation: Comparison between Radiología, European radiology and radiology for 1997–1998. European Radiology, 12(1), 248–252.CrossRefPubMedGoogle Scholar
  44. Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.CrossRefGoogle Scholar
  45. Molinari, A., & Molinari, J.-F. (2008). Mathematical aspects of a new criterion for ranking scientific institutions based on the h-index. Scientometrics, 75(2), 339–356.CrossRefGoogle Scholar
  46. Pendlebury, D. A. (2009). The use and misuse of journal metrics and other citation indicators. Archivum immunologiae et therapiae experimentalis, 57(1), 1–11.CrossRefPubMedGoogle Scholar
  47. PLoS Medicine E. (2006). The impact factor game. PLoS Med, 3(6), e291.CrossRefGoogle Scholar
  48. Ramin, S., & Shirazi, A. S. (2012). Comparison between Impact factor, SCImago journal rank indicator and Eigenfactor score of nuclear medicine journals. Nuclear Medicine Review, 15(2), 132–136.PubMedGoogle Scholar
  49. Romanovsky, A. A. (2012). Revised h index for biomedical research. Cell Cycle, 11(22), 4118–4121.CrossRefPubMedPubMedCentralGoogle Scholar
  50. Saha, S., Saint, S., & Christakis, D. A. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91(1), 42.PubMedPubMedCentralGoogle Scholar
  51. Sahel, J.-A. (2011). Quality versus quantity: assessing individual research performance. Science translational medicine, 3(84), 84cm13–84cm13.Google Scholar
  52. Schiermeie, Q. (2008). Self-publishing editor set to retire. Nature, 456(432).Google Scholar
  53. Schreiber, M. (2007). Self-citation corrections for the Hirsch index. EPL (Europhysics Letters), 78(3), 30002.CrossRefGoogle Scholar
  54. Schreiber, M. (2008). A modification of the h-index: The hm-index accounts for multi-authored manuscripts. Journal of Informetrics, 2(3), 211–216.CrossRefGoogle Scholar
  55. Van Eck, N. J., Waltman, L., van Raan, A. F. J., Klautz, R. J. M., & Peul, W. C. (2013). Citation analysis may severely underestimate the impact of clinical research as compared to basic research. PloS One, 8(4), e62395. Google Scholar
  56. van Leeuwen, T. (2012). Discussing some basic critique on journal impact factors: Revision of earlier comments. Scientometrics, 92(2), 443–455.CrossRefPubMedPubMedCentralGoogle Scholar
  57. Van Leeuwen, T. N., Moed, H. F., & Reedijk, J. (1999). Critical comments on institute for scientific information impact factors: A sample of inorganic molecular chemistry journals. Journal of Information Science, 25(6), 489–498.CrossRefGoogle Scholar
  58. Vanclay, J. K. (2009). Bias in the journal impact factor. Scientometrics, 78(1), 3–12.CrossRefGoogle Scholar
  59. Vanclay, J. K. (2012). Impact factor: Outdated artefact or stepping-stone to journal certification? Scientometrics, 92(2), 211–238. doi: 10.1007/s11192-011-0561-0.CrossRefGoogle Scholar
  60. Vinkler, P. (2007). Eminence of scientists in the light of the h-index and other scientometric indicators. Journal of Information Science, 33(4), 481–491.CrossRefGoogle Scholar
  61. Waltman, L., & van Eck, N. J. (2013). Source normalized indicators of citation impact: An overview of different approaches and an empirical comparison. Scientometrics, 96(3), 699–716.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Murray State UniversityMurrayUSA

Personalised recommendations