Skip to main content
SpringerLink
Log in

Google Scholar as a Data Source for Research Assessment

  • Chapter
Book cover Springer Handbook of Science and Technology Indicators

Part of the book series: Springer Handbooks ((SHB))

Abstract

The launch of Google Scholar ( ) marked the beginning of a revolution in the scientific information market. This search engine, unlike traditional databases, automatically indexes information from the academic web. Its ease of use, together with its wide coverage and fast indexing speed, have made it the first tool most scientists currently turn to when they need to carry out a literature search. Additionally, the fact that its search results were accompanied from the beginning by citation counts, as well as the later development of secondary products that leverage this citation data (such as Google Scholar Metrics and Google Scholar Citations), made many scientists wonder about its potential as a source of data for bibliometric analyses. The goal of this chapter is to lay the foundations for the use of GS as a supplementary source (and in some disciplines, arguably the best alternative) for scientific evaluation. First, we present a general overview of how GS works. Second, we present empirical evidences about its main characteristics (size, coverage, and growth rate). Third, we carry out a systematic analysis of the main limitations this search engine presents as a tool for the evaluation of scientific performance. Lastly, we discuss the main differences between GS and other more traditional bibliographic databases in light of the correlations found between their citation data. We conclude that GS presents a broader view of the academic world because it has brought to light a great amount of sources that were not previously visible.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 299.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 379.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • E. Garfield, I.H. Sher, R.J. Torpie: The Use of Citation Data in Writing the History of Science (Institute for Scientific Information, Philadelphia 1964)

    Book  Google Scholar 

  • E. Garfield: Science citation index: A new dimension in indexing, Science 144(3619), 649–654 (1964)

    Article  Google Scholar 

  • J.L. Ortega: Academic Search Engines: A Quantitative Outlook (Elsevier, Oxford 2014)

    Book  Google Scholar 

  • E. Orduna-Malea, A. Martín-Martín, J.M. Ayllón, E. Delgado López-Cózar: La revolución Google Scholar: abriendo la caja de Pandora académica (UNE, Granada 2016)

    Google Scholar 

  • Google Scholar: My citation counts have gone down. Help!, https://scholar.google.com/intl/en/scholar/help.html#corrections (2017)

  • S. Baggett, M. Williams: Budgets stretched, staff stressed, usage stalled…Something's gotta give! In: Proc. Charleston Library Conf., Charleston, ed. by B.R. Bernhardt, L.H. Hinds, K.P. Strauch (2012) pp. 324–327

    Google Scholar 

  • B. Kramer, J. Bosman: 101 Innovations in Scholarly Communication—The Changing Research Workflow (Universitait Utrecht, Utrecht 2015)

    Google Scholar 

  • S. Brown, S.A. Swan: Researchers' use of academic libraries and their services. A report commissioned by the Research Information Network and the Consortium of Research Libraries (2007)

    Google Scholar 

  • J. Mussell, R. Croft: Discovery layers and the distance student: Online search habits of students, J. Lib. Inf. Serv. Distance Learn. 7(1/2), 18–39 (2013)

    Article  Google Scholar 

  • D. Nicholas, C. Boukacem-Zeghmouri, B. Rodríguez-Bravo, J. Xu, A. Watkinson, A. Abrizah, E. Herman, M. Świgoń: Where and how early career researchers find scholarly information, Learn. Publ. 30(1), 19–29 (2017)

    Article  Google Scholar 

  • R. Van Noorden: Online collaboration: Scientists and the social network, Nature 512(7513), 126–169 (2014)

    Article  Google Scholar 

  • P. Jacsó: Google Scholar Author Citation Tracker: Is it too little, too late?, Online Inf. Rev. 36(1), 126–141 (2012)

    Article  Google Scholar 

  • E. Delgado López-Cózar, A. Cabezas-Clavijo: Google Scholar Metrics: An unreliable tool for assessing scientific journals, Prof. Inf. 21(4), 419–427 (2012)

    Google Scholar 

  • E. Delgado López-Cózar, E. Orduna-Malea, A. Martín-Martín: Supplementary material to ‘Google Scholar as a data source for research assessment', https://doi.org/10.17605/OSF.IO/PQR53 (2017)

  • S.C. Bradford: Sources of information of specific subjects, Engineering 137, 85–86 (1934)

    Google Scholar 

  • Google Scholar: Google Scholar's inclusion guidelines for webmasters, https://scholar.google.com/intl/en/scholar/inclusion.html#indexing (2017)

  • Google Scholar: Google Scholar's About page, https://scholar.google.com/intl/en/scholar/about.html (2017)

  • Google Scholar: Google Scholar's search tips: Content coverage. What specific journals do you cover?, https://scholar.google.com/intl/en/scholar/help.html#coverage (2017)

  • E. Delgado López-Cózar, A. Cabezas-Clavijo: Ranking journals: Could Google Scholar Metrics be an alternative to journal citation reports and Scimago journal rank?, Learn. Publ. 26(2), 101–113 (2013)

    Article  Google Scholar 

  • E. Delgado López-Cózar, N. Robinson-García: Repositories in Google Scholar Metrics or what is this document type doing in a place as such, Cybermetrics 16(1) (2012)

    Google Scholar 

  • A. Martín-Martín, J.M. Ayllón, E. Orduna-Malea, E. Delgado López-Cózar: Google Scholar Metrics 2014: A low cost bibliometric tool, EC3 Working Papers 17 (2014)

    Google Scholar 

  • A. Martín-Martín, J.M. Ayllón, E. Orduna-Malea, E. Delgado López-Cózar: 2016 Google Scholar Metrics released: A matter of languages… and something else, EC3 Working Papers 22 (2016)

    Google Scholar 

  • J. Bar-Ilan: Search engine results over time: A case study on search engine stability, Cybermetrics 2(3) (1999)

    Google Scholar 

  • R. Rousseau: Daily time series of common single word searches in Altavista and Northernlight, Cybermetrics 2(3) (1999)

    Google Scholar 

  • M. Thelwall: Quantitative comparisons of search engine results, J. Am. Soc. Inform. Sci. Technol. 59(11), 1702–1710 (2008)

    Article  Google Scholar 

  • A. Uyar: Investigation of the accuracy of search engine hit counts, J. Inform. Sci. 35(4), 469–480 (2009)

    Article  Google Scholar 

  • W. Koehler: Web page change and persistence–A four-year longitudinal study, J. Am. Soc. Inform. Sci. Technol. 53(2), 162–171 (2002)

    Article  Google Scholar 

  • J.L. Ortega, I.F. Aguillo, J.A. Prieto: Longitudinal study of content and elements in the scientific web environment, J. Inform. Sci. 32(4), 344–351 (2006)

    Article  Google Scholar 

  • N. Payne, M. Thelwall: Longitudinal trends in academic web links, J. Inform. Sci. 34(1), 3–14 (2008)

    Article  Google Scholar 

  • A. Bosch, T. Bogers, T.M. Kunder: Estimating search engine index size variability: A 9-year longitudinal study, Scientometrics 107(2), 839–856 (2016)

    Article  Google Scholar 

  • P. Jacsó: As we may search–comparison of major features of the Web of Science, Scopus, and Google Scholar, Curr. Sci. 89(9), 1537–1547 (2005)

    Google Scholar 

  • I.F. Aguillo: Is Google Scholar useful for bibliometrics? A webometric analysis, Scientometrics 91(2), 343–351 (2012)

    Article  Google Scholar 

  • I.F. Aguillo: Why Google Scholar? An Empirical Comparison of the Main Evaluation Tools (Google Scholar Day, Madrid 2017)

    Google Scholar 

  • E. Orduna-Malea, J.M. Ayllón, A. Martín-Martín, E. Delgado López-Cózar: Methods for estimating the size of Google Scholar, Scientometrics 104(3), 931–949 (2015)

    Article  Google Scholar 

  • M. Khabsa, C.L. Giles: The number of scholarly documents on the public web, PLoS ONE 9(5), e93949 (2014)

    Article  Google Scholar 

  • E. Orduna-Malea, A. Martín-Martín, E. Delgado López-Cózar: Google Scholar as a source for scholarly evaluation: A bibliographic review of database errors, Rev. Esp. Doc. Cient. 40(4), e185 (2017)

    Article  Google Scholar 

  • A. Martín-Martín, E. Orduna-Malea, J.M. Ayllón, E. Delgado López-Cózar: A two-sided academic landscape: Snapshot of highly-cited documents in Google Scholar (1950–2013), Rev. Esp. Doc. Cient. 39(4), e149 (2016)

    Article  Google Scholar 

  • H.R. Jamali, M. Nabavi: Open access and sources of full-text articles in Google Scholar in different subject fields, Scientometrics 105(3), 1635–1651 (2015)

    Article  Google Scholar 

  • A. Verstak, A. Acharya: Identifying a primary version of a document, US Patent US8095876B1 (Washington DC 2013)

    Google Scholar 

  • ResearchGate: About, https://www.researchgate.net/about (2017)

  • E. Orduna-Malea, E. Delgado López-Cózar: Google Scholar metrics evolution: An analysis according to languages, Scientometrics 98(3), 2353–2367 (2014)

    Article  Google Scholar 

  • G.A. Barnett, C.J. Chung, H.W. Park: Uncovering transnational hyperlink patterns and web-mediated contents: A new approach based on cracking. com domain, Soc. Sci. Comput. Rev. 29(3), 369–384 (2011)

    Article  Google Scholar 

  • E. Delgado López-Cózar, R. Repiso-Caballero: El impacto de las revistas de comunicación: Comparando Google Scholar metrics, Web of Science and Scopus, Comunicar 21(41), 45–52 (2013)

    Article  Google Scholar 

  • L.M. Reina Leal, R. Repiso-Caballero, E. Delgado López-Cózar: H Index of scientific Nursing journals according to Google Scholar metrics (2007–2011), EC3 Reports 5 (2013)

    Google Scholar 

  • E. Delgado López-Cózar, E. Orduna-Malea, D.M. Cartagena, E. Jiménez-Contreras, R. Ruiz-Pérez: Journal Scholar: Una alternativa internacional, gratuita y de libre acceso para medir el impacto de las revistas de Arte, Humanidades y Ciencias Sociales, EC3 Working Papers 5 (2012)

    Google Scholar 

  • A. Martin-Martin, E. Orduna-Malea, A.W. Harzing, E. Delgado López-Cózar: Can we use Google Scholar to identify highly-cited documents?, J. Inform. 11(1), 152–163 (2017)

    Article  Google Scholar 

  • E. Delgado López-Cózar, D.M. Cartagena, E. Jiménez-Contreras, R. Ruiz-Pérez: Índice H de las revistas españolas de Ciencias Sociales y Jurídicas según Google Scholar (2002–2011), EC3 Reports 4 (2013)

    Google Scholar 

  • Grupo de Evaluación de la Ciencia y de la Comunicación Científica (EC3): IN-RECS database, http://ec3.ugr.es/in-recs (2017)

  • J.M. Ayllón, A. Martín-Martín, E. Orduna-Malea, E. Delgado López-Cózar: Índice H de las revistas científicas españolas según Google Scholar metrics (2011–2015), EC3 Reports 17 (2016)

    Google Scholar 

  • X. Gu, K. Blackmore: Characterisation of academic journals in the digital age, Scientometrics 110(3), 1333–1350 (2017)

    Article  Google Scholar 

  • K. Kousha, M. Thelwall: Google Scholar citations and Google Web/URL citations: A multi-discipline exploratory analysis, J. Am. Soc. Inform. Sci. Technol. 58(7), 1055–1065 (2007)

    Article  Google Scholar 

  • J.D.F. De Winter, A.A. Zadpoor, D. Dodou: The expansion of Google Scholar versus Web of Science: A longitudinal study, Scientometrics 98(2), 1547–1565 (2014)

    Article  Google Scholar 

  • A.W. Harzing: A preliminary test of Google Scholar as a source for citation data: A longitudinal study of Nobel Prize winners, Scientometrics 94(3), 1057–1075 (2013)

    Article  Google Scholar 

  • A.W. Harzing, S. Alakangas: Google Scholar, Scopus and the Web of Science: A longitudinal and cross-disciplinary comparison, Scientometrics 106(2), 787–804 (2016)

    Article  Google Scholar 

  • A.W. Harzing, S. Alakangas: Microsoft Academic: Is the phoenix getting wings?, Scientometrics 110(1), 371–383 (2016)

    Article  Google Scholar 

  • H. Moed, J. Bar-Ilan, G. Halevi: A new methodology for comparing Google Scholar and Scopus, J. Inform. 10(2), 533–551 (2016)

    Article  Google Scholar 

  • L. Bornmann, W. Marx, H. Schier, E. Rahm, A. Thor, H.D. Daniel: Convergent validity of bibliometric Google Scholar data in the field of chemistry citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts, J. Inform. 3(1), 27–35 (2009)

    Article  Google Scholar 

  • A. Martín-Martín, E. Orduna-Malea, E. Delgado López-Cózar: Highly cited documents in Google Scholar by language and year of publication (January 2017), https://doi.org/10.6084/m9.figshare.5024465.v1 (2017)

  • A.W. Harzing: A longitudinal study of Google Scholar coverage between 2012 and 2013, Scientometrics 98(1), 565–575 (2014)

    Article  Google Scholar 

  • M. Thelwall, K. Kousha: ResearchGate versus Google Scholar: Which finds more early citations?, Scientometrics 112(2), 1125–1131 (2017)

    Article  Google Scholar 

  • P. Jacsó: Google Scholar: The pros and the cons, Online Inf. Rev. 29(2), 208–214 (2005)

    Article  Google Scholar 

  • P. Jacsó: Deflated, inflated and phantom citation counts, Online Inf. Rev. 30(3), 297–309 (2006)

    Article  Google Scholar 

  • P. Jacsó: Google Scholar revisited, Online Inf. Rev. 32(1), 102–114 (2008)

    Article  Google Scholar 

  • P. Jacsó: Testing the calculation of a realistic h-index in Google Scholar, Scopus and Web of Science for F.W. Lancaster, Libr. Trends 56(4), 784–815 (2008)

    Article  Google Scholar 

  • P. Jacsó: The pros and cons of computing the h-index using Google Scholar, Online Inf. Rev. 32(3), 437–452 (2008)

    Article  Google Scholar 

  • P. Jacsó: Calculating the h-index and other bibliometric and scientometric indicators from Google Scholar with the publish or perish software, Online Inf. Rev. 33(6), 1189–1200 (2009)

    Article  Google Scholar 

  • P. Jacsó: Google Scholar's Ghost Authors, Libr. J. 134(18), 26–27 (2009)

    Google Scholar 

  • P. Jacsó: Metadata mega mess in Google Scholar, Online Inf. Rev. 34(1), 175–191 (2010)

    Article  Google Scholar 

  • P. Jacsó: Google Scholar duped and deduped—The aura of ‘robometrics', Online Inf. Rev. 35(1), 154–160 (2011)

    Article  Google Scholar 

  • M. Thelwall, K. Kousha: Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web, El profesional de la información 24(5), 587–606 (2015)

    Article  Google Scholar 

  • A. Martín-Martín, E. Orduna-Malea, J.M. Ayllón, E. Delgado López-Cózar: Does Google Scholar contain all highly cited documents (1950–2013)?, EC3 Working Papers 19 (2014)

    Google Scholar 

  • J. Wilsdon, L. Allen, E. Belfiore, P. Campbell, S. Curry, S. Hill: The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management (HEFCE, UK 2015)

    Book  Google Scholar 

  • L.I. Meho, K. Yang: Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar, J. Am. Soc. Inform. Sci. Technol. 58(13), 2105–2125 (2007)

    Article  Google Scholar 

  • J. Bar-Ilan: Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar, Scientometrics 82(3), 495–506 (2010)

    Article  Google Scholar 

  • G. Doğan, İ. Şencan, Y. Tonta: Does dirty data affect google scholar citations?, Proc. Assoc. Inform. Sci. Technol. 53(1), 1–4 (2016)

    Article  Google Scholar 

  • J. Beel, B. Gipp: Academic search engine spam and Google Scholar's resilience against it, J. Electron. Publ. (2010), https://doi.org/10.3998/3336451.0013.305

    Article  Google Scholar 

  • C. Labbé, D. Labbé: Duplicate and fake publications in the scientific literature: How many SCIgen papers in computer science?, Scientometrics 94(1), 379–396 (2013)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Información y Documentación, Universidad de Granada, Granada, Spain

    Emilio Delgado López-Cózar & Alberto Martín-Martín

  2. Audiovisual Communication, Documentation, and History of Art, Universitat Politècnica de València, Valencia, Spain

    Enrique Orduña-Malea

Authors
  1. Emilio Delgado López-Cózar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Enrique Orduña-Malea
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alberto Martín-Martín
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Emilio Delgado López-Cózar .

Editor information

Editors and Affiliations

  1. ECOOM and Faculty of Economics and Business, KU Leuven, Leuven, Belgium

    Wolfgang Glänzel

  2. Amsterdam, The Netherlands

    Henk F. Moed

  3. Competence Center Policy – Industry – Innovation, Fraunhofer Institute for Systems and Innovation Research ISI, Karlsruhe, Germany

    Ulrich Schmoch

  4. Faculty of Science and Engineering, University of Wolverhampton, Wolverhampton, UK

    Mike Thelwall

1 Supplementary Material to Book Chapter

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer International Publishing AG, part of Springer Nature

About this chapter

Cite this chapter

Delgado López-Cózar, E., Orduña-Malea, E., Martín-Martín, A. (2019). Google Scholar as a Data Source for Research Assessment. In: Glänzel, W., Moed, H.F., Schmoch, U., Thelwall, M. (eds) Springer Handbook of Science and Technology Indicators. Springer Handbooks. Springer, Cham. https://doi.org/10.1007/978-3-030-02511-3_4

Download citation

Publish with us

Policies and ethics

Search

Navigation