Skip to main content
SpringerLink
Log in

The Use of Bibliometrics for Assessing Research: Possibilities, Limitations and Adverse Effects

  • Chapter
  • First Online:
Incentives and Performance

Abstract

Researchers are used to being evaluated: publications, hiring, tenure and funding decisions are all based on the evaluation of research. Traditionally, this evaluation relied on judgement of peers but, in the light of limited resources and increased bureaucratization of science, peer review is getting more and more replaced or complemented with bibliometric methods. Central to the introduction of bibliometrics in research evaluation was the creation of the Science Citation Index (SCI) in the 1960s, a citation database initially developed for the retrieval of scientific information. Embedded in this database was the Impact Factor, first used as a tool for the selection of journals to cover in the SCI, which then became a synonym for journal quality and academic prestige. Over the last 10 years, this indicator became powerful enough to influence researchers’ publication patterns in so far as it became one of the most important criteria to select a publication venue. Regardless of its many flaws as a journal metric and its inadequacy as a predictor of citations on the paper level, it became the go-to indicator of research quality and was used and misused by authors, editors, publishers and research policy makers alike. The h-index, introduced as an indicator of both output and impact combined in one simple number, has experienced a similar fate, mainly due to simplicity and availability. Despite their massive use, these measures are too simple to capture the complexity and multiple dimensions of research output and impact. This chapter provides an overview of bibliometric methods, from the development of citation indexing as a tool for information retrieval to its application in research evaluation, and discusses their misuse and effects on researchers’ scholarly communication behavior.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    It should be mentioned that, despite the fact that bibliometrics should not be used alone for research evaluation, individual level disambiguation is often needed in order to assess groups of researchers.

  2. 2.

    http://www.researcherid.com

  3. 3.

    http://orcid.org

  4. 4.

    http://stadium.open.ac.uk/webcast-ou/documents/20130410_Jonathan_Adams_Presentation.ppt

  5. 5.

    http://admin-apps.webofknowledge.com/JCR/static_html/notices/notices.htm

  6. 6.

    Only recently the San Francisco Declaration on Research Assessment (DORA) took a stand against the use of the Impact Factor in article and author evaluation.

References

  • Abt HA (1992) Publication practices in various sciences. Scientometrics 24(3):441–447. doi:10.1007/BF02051040

    Article  Google Scholar 

  • Adam D (2002) The counting house. Nature 415(6873):726–729

    Article  Google Scholar 

  • Aksnes DW (2003) A macro study of self-citation. Scientometrics 56:235–246

    Article  Google Scholar 

  • Archambault E, Larivière V (2009) History of the journal impact factor: contingencies and consequences. Scientometrics 79(3):635–649. doi:10.1007/s11192-007-2036-x

    Article  Google Scholar 

  • Bergstrom CT (2007) Eigenfactor: measuring the value and prestige of scholarly journals. Coll Res Libr News 68(5):314–316

    Google Scholar 

  • Bhattacharjee Y (2011) Saudi universities offer cash in exchange for academic prestige. Science 334(6061):1344–1345

    Article  Google Scholar 

  • Biagioli M (2003) Rights or rewards? Changing frameworks of scientific authorship. In: Biagioli M, Galison P (eds) Scientific authorship: credit and intellectual property in science. Routledge, New York, pp 253–279

    Google Scholar 

  • Birnholtz J (2006) What does it mean to be an author? The intersection of credit, contribution and collaboration in science. J Am Soc Inf Sci Technol 57(13):1758–1770

    Article  Google Scholar 

  • Bornmann L, Mutz R (2011) Further steps towards an ideal method of measuring citation performance: the avoidance of citation (ratio) averages in field-normalization. J Informetrics 5(1):228–230. doi:10.1016/j.joi.2010.10.009

    Article  Google Scholar 

  • Bradford SC (1934) Sources of information on specific subjects. Engineering 137:85–86. doi:10.1177/016555158501000407

    Google Scholar 

  • Burton RE, Kebler RW (1960) The half-life of some scientific and technical literatures. Am Doc 11(1):18–22. doi:10.1002/asi.5090110105

    Article  Google Scholar 

  • Calver MC, Bradley JS (2009) Should we use the mean citations per paper to summarise a journal’s impact or to rank journals in the same field? Scientometrics 81(3):611–615. doi:10.1007/s11192-008-2229-y

    Article  Google Scholar 

  • Cameron BD (2005) Trends in the usage of ISI bibliometric data: uses, abuses, and implications. Portal Libr Acad 5(1):105–125. doi:10.1353/pla.2005.0003

    Article  Google Scholar 

  • Cole JR, Cole S (1967) Scientific output and recognition: a study in the operation of the reward system in science. Am Soc Rev 32(3):377–390

    Article  Google Scholar 

  • Cole JR, Cole S (1973) Social stratification in science. University of Chicago Press, Chicago

    Google Scholar 

  • Cole FJ, Eales NB (1917) The history of comparative anatomy. Part I: a statistical analysis of the literature. Sci Prog 11(43):578–596

    Google Scholar 

  • Costas R, van Leeuwen TN, Bordóns M (2010) Self-citations at the meso and individual levels: effects of different calculation methods. Scientometrics 82(3):517–537

    Article  Google Scholar 

  • De Bellis N (2009) Bibliometrics and citation analysis. From the science citation index to cybermetrics. The Scarecrow Press, Lanham

    Google Scholar 

  • De Lange C, Glänzel W (1997) Modelling and measuring multilateral coauthorship in international scientific collaboration. Part I. Development of a new model using a series expansion approach. Scientometrics 40(3):593–604

    Article  Google Scholar 

  • Engels TCE, Ossenblok TLB, Spruyt EHJ (2012) Changing publication patterns in the social sciences and humanities, 2000–2009. Scientometrics 93(2):373–390. doi:10.1007/s11192-012-0680-2

    Article  Google Scholar 

  • Favaloro EJ (2008) Measuring the quality of journals and journal articles: the impact factor tells but a portion of the story. Semin Thromb Hemost 34(1):7–25. doi:10.1055/s-2008-1066030

    Article  Google Scholar 

  • Flanagin A, Carey LA, Fontanarosa PB, Phillips SG, Pace BP, Lundberg GD, Rennie D (1998) Prevalance of articles with honorary authors and ghost authors in peer-reviewed medical journals. JAMA 280(3):222–224

    Article  Google Scholar 

  • Garfield E (1964) “Science citation index”—a new dimension in indexing. Science 144(3619):649–654

    Article  Google Scholar 

  • Garfield E (1972) Citation analysis as a tool in journal evaluation. Journals can be ranked by frequency and impact of citations for science policy studies. Science 178(4060):471–479

    Article  Google Scholar 

  • Garfield E (1979) Citation indexing - its theory and application in science, technology, and humanities. Wiley, New York

    Google Scholar 

  • Garfield E (1983) Citation indexing. Its theory and application in science, technology, and humanities. ISI Press, Philadelphia

    Google Scholar 

  • Gingras Y, Larivière V (2011) There are neither “king” nor “crown” in scientometrics: comments on a supposed “alternative” method of normalization. J Informetrics 5(1):226–227

    Article  Google Scholar 

  • Glänzel W, Thjis B (2004) The influence of author self-citations on bibliometric macro indicators. Scientometrics 59(3):281–310

    Article  Google Scholar 

  • Glänzel W, Debackere K, Thijs B, Schubert A (2006) A concise review on the role of author self-citations in information science, bibliometrics and science policy. Scientometrics 67:263–277

    Article  Google Scholar 

  • Glänzel W, Schubert A, Thijs B, Debackere K (2011) A priori vs. a posteriori normalisation of citation indicators. The case of journal ranking. Scientometrics 87(2):415–424. doi:10.1007/s11192-011-0345-6

    Article  Google Scholar 

  • Gonzalez-Pereira B, Guerrero-Bote VP, de Moya-Anegon F (2010) A new approach to the metric of journals’ scientific prestige: the SJR indicator. J Informetrics 4(3):379–391. doi:10.1016/j.joi.2010.03.002

    Article  Google Scholar 

  • Gross PLK, Gross EM (1927) College libraries and chemical education. Science 66(1713):385–389. doi:10.1126/science.66.1713.385

    Article  Google Scholar 

  • Haustein S (2012) Multidimensional journal evaluation. Analyzing scientific periodicals beyond the impact factor. De Gruyter Saur, Berlin

    Book  Google Scholar 

  • Hemmingsson A, Mygind T, Skjennald A, Edgren J (2002) Manipulation of impact factors by editors of scientific journals. Am J Roentgenol 178(3):767–767

    Article  Google Scholar 

  • Hicks D (2013) One size doesn’t fit all: on the co-evolution of national evaluation systems and social science publishing. Confero 1(1):67–90

    Article  Google Scholar 

  • Hirsch JE (2005) An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A 102(46):16569–16572. doi:10.1073/pnas.0507655102

    Article  Google Scholar 

  • Hvistendahl M (2013) China’s publication bazaar. Science 342(6162):1035–1039. doi:10.1126/science.342.6162.1035

    Article  Google Scholar 

  • Larivière V, Gingras Y (2010) On the prevalence and scientific impact of duplicate publications in different scientific fields (1980–2007). J Doc 66(2):179–190

    Article  Google Scholar 

  • Larivière V, Gingras Y (2011) Averages of ratios vs. ratios of averages: an empirical analysis of four levels of aggregation. J Informetrics 5(3):392–399. doi:10.1016/j.joi.2011.02.001

    Article  Google Scholar 

  • Larivière V, Gingras Y, Archambault E (2006) Canadian collaboration networks: a comparative analysis of the natural sciences, social sciences and the humanities. Scientometrics 68(3):519–533

    Article  Google Scholar 

  • Larivière V, Archambault É, Gingras Y (2008) Long-term variations in the aging of scientific literature: from exponential growth to steady-state science (1900–2004). J Am Soc Inf Sci Technol 59(2):288–296

    Article  Google Scholar 

  • Leydesdorff L (2009) How are new citation-based journal indicators adding to the bibliometric toolbox? J Am Soc Inf Sci Technol 60(7):1327–1336. doi:10.1002/asi.21024

    Article  Google Scholar 

  • Leydesdorff L, Opthof T (2010) Normalization at the field level: fractional counting of citations. J Informetrics 4(4):644–646. doi:10.1016/j.joi.2010.05.003

    Article  Google Scholar 

  • Lotka AJ (1926) The frequency distribution of scientific productivity. J Wash Acad Sci 16(12):317–323

    Google Scholar 

  • Lozano GA, Larivière V, Gingras Y (2012) The weakening relationship between the impact factor and papers’ citations in the digital age. J Am Soc Inf Sci Technol 63(11):2140–2145

    Article  Google Scholar 

  • Martin BR (2013) Whither research integrity? Plagiarism, self-plagiarism and coercive citation in an age of research assessment. Res Policy 42:1005–1014. doi:10.1016/j.respol.2013.03.011

    Article  Google Scholar 

  • Martin BR, Irvine J (1983) Assessing basic research. Some partial indicators of scientific progress in radio astronomy. Res Policy 12:61–90

    Article  Google Scholar 

  • Martyn J, Gilchrist A (1968) An evaluation of British scientific journals. Aslib, London

    Google Scholar 

  • Merton RK (1973) The sociology of science: theoretical and empirical investigations. University of Chicago Press, Chicago

    Google Scholar 

  • Merton RK (1977) The sociology of science: an episodic memoir. In: Merton RK, Gaston J (eds) The sociology of science in Europe. Southern Illinois University Press, Carbondale, pp 3–141

    Google Scholar 

  • Merton RK (1988) The Matthew effect in science. II. Cummulative advantage and the symbolism of intellectual property. ISIS 79:606–623

    Article  Google Scholar 

  • Moed HF (2002) The impact-factors debate: the ISI’s uses and limits. Nature 415(6873):731–732

    Article  Google Scholar 

  • Moed HF (2005) Citation analysis in research evaluation. Springer, Dordrecht

    Google Scholar 

  • Moed HF (2010) Measuring contextual citation impact of scientific journals. J Informetrics 4(3):265–277. doi:10.1016/j.joi.2010.01.002

    Article  Google Scholar 

  • Moed HF, van Leeuwen TN (1995) Improving the accuracy of institute for scientific information’s journal impact factors. J Am Soc Inf Sci 46(6):461–467

    Article  Google Scholar 

  • Norris M, Oppenheim C (2003) Citation counts and the research assessment exercise V: archaeology and the 2001 research assessment exercise. J Doc 59(6):709–730

    Article  Google Scholar 

  • Otlet P (1934) Traité de documentation: le livre sur le livre, théorie et pratique. Editiones Mundaneum, Brussels

    Google Scholar 

  • Pontille D (2004) La Signature Scientifique: une Sociologie Pragmatique de l’Attribution. CNRS Éditions, Paris

    Google Scholar 

  • Price DJDS (1963) Little science, big science. Columbia University Press, New York

    Google Scholar 

  • Price DDS (1981) Letter to the editor: multiple authorship. Science 212(4498):984–1066

    Article  Google Scholar 

  • Pritchard A (1969) Statistical bibliography or bibliometrics? J Doc 25(4):348–349

    Google Scholar 

  • Rinia EJ, van Leeuwen TN, van Vuren HG, van Raan AFJ (1998) Comparative analysis of a set of bibliometric indicators and central peer review criteria. Evaluation of condensed matter physics in the Netherlands. Res Policy 27:95–107

    Article  Google Scholar 

  • Rogers LF (2002) Impact factor: the numbers game. Am J Roentgenol 178(3):541–542. doi:10.2214/ajr.178.3.1780541

    Article  Google Scholar 

  • Rowlands I, Nicholas D (2005) New journal publishing models – an international survey of senior researchers. CIBER, University College London, London. Available from: http://www.homepages.ucl.ac.uk/~uczciro/pa_stm_final_report.pdf

  • Schubert A, Braun T (1996) Cross-field normalization of scientometric indicators. Scientometrics 36(3):311–324

    Article  Google Scholar 

  • Seglen PO (1992) The skewness of science. J Am Soc Inf Sci 43(9):628–638

    Article  Google Scholar 

  • Seglen PO (1997a) Citations and journal impact factors: questionable indicators of research quality. Allergy 52(11):1050–1056

    Article  Google Scholar 

  • Seglen PO (1997b) Why the impact factor of journals should not be used for evaluating research. BMJ 314(7079):498–502. doi:10.1136/bmj.314.7079.497

    Article  Google Scholar 

  • Smalheiser NR, Torvik VI (2009) Author name disambiguation. Annu Rev Inf Sci Technol 43:1–43. doi:10.1002/aris.2009.1440430113

    Article  Google Scholar 

  • Small H (1987) The significance of bibliographic references. Scientometrics 8:321–340

    Article  Google Scholar 

  • Snyder H, Bonzi S (1998) Patterns of self-citation across disciplines. J Inf Sci 24(6):431–435

    Article  Google Scholar 

  • The PLoS Medicine editors (2006) The impact factor game—it is time to find a better way to assess the scientific literature. PLoS Med 3(6):e291. doi:10.1371/journal.pmed.0030291

  • Tijssen RJW, Visser MS, van Leeuwen TN (2002) Benchmarking international scientific excellence: are highly cited research papers an appropriate frame of reference? Scientometrics 54(3):381–397

    Article  Google Scholar 

  • Todorov R, Glänzel W (1988) Journal citation measures—a concise review. J Inf Sci 14(1):47–56. doi:10.1177/016555158801400106

    Article  Google Scholar 

  • Van Norden R (2013) Brazilian citation scheme outed. Thomson Reuters suspends journals from its rankings for ‘citation stacking’. Nature 500(7464):510–511

    Article  Google Scholar 

  • Waltman L, van Eck NJ (2010) The relation between Eigenfactor, audience factor, and influence weight. J Am Soc Inf Sci Technol 61(7):1476–1486. doi:10.1002/asi.21354

    Article  Google Scholar 

  • Waltman L, van Eck NJ (2012) The inconsistency of the h-index. J Am Soc Inf Sci Technol 63(2):406–415. doi:10.1002/asi.21678

    Article  Google Scholar 

  • Weingart P (2005) Impact of bibliometrics upon the science system: inadvertent consequences? Scientometrics 62(1):117–131. doi:10.1007/s11192-005-0007-7

    Article  Google Scholar 

  • Weller K (2015) Social media and altmetrics: an overview of current alternative approaches to measuring scholarly impact. In: Welpe IM, Wollersheim J, Ringelhan S, Osterloh M (eds) Incentives and performance - governance of research organizations. Springer, Cham

    Google Scholar 

  • Wislar JS, Flanagin A, Fontanarosa PB, DeAngelis CD (2011) Honorary and ghost authorship in high impact biomedical journals: a cross sectional survey. BMJ 343:6128. doi:10.1136/bmj.d6128

    Article  Google Scholar 

  • Zipf GK (1949) Human behaviour and the principle of least effort. Addison-Wesley, Reading, MA

    Google Scholar 

  • Zitt M (2010) Citing-side normalization of journal impact: a robust variant of the audience factor. J Informetrics 4(3):392–406

    Article  Google Scholar 

  • Zuckerman H (1987) Citation analysis and the complex problem of intellectual influence. Scientometrics 12:329–338

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. École de bibliothéconomie et des sciences de l’information (EBSI), Université de Montréal, Montréal, QC, Canada

    Stefanie Haustein

  2. École de bibliothéconomie et des sciences de l’information (EBSI), Université de Montréal, Observatoire des sciences et des technologies (OST), Centre interuniversitaire de recherche sur la science et la technologie (CIRST), Université du Québec à Montréal, Montréal, QC, Canada

    Vincent Larivière

Authors
  1. Stefanie Haustein
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vincent Larivière
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefanie Haustein .

Editor information

Editors and Affiliations

  1. TUM School of Management, Technische Universität München, Munich, Germany

    Isabell M. Welpe

  2. TUM School of Management, Technische Universität München, Munich, Germany

    Jutta Wollersheim

  3. TUM School of Management, Technische Universität München, Munich, Germany

    Stefanie Ringelhan

  4. Zeppelin University, Friedrichshafen, Germany, and University of Zurich, Zurich, Switzerland

    Margit Osterloh

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Haustein, S., Larivière, V. (2015). The Use of Bibliometrics for Assessing Research: Possibilities, Limitations and Adverse Effects. In: Welpe, I., Wollersheim, J., Ringelhan, S., Osterloh, M. (eds) Incentives and Performance. Springer, Cham. https://doi.org/10.1007/978-3-319-09785-5_8

Download citation

Publish with us

Policies and ethics

Search

Navigation