Skip to main content
SpringerLink
Log in

Using Altmetrics to Support Research Evaluation

  • Conference paper
  • First Online:
Altmetrics for Research Outputs Measurement and Scholarly Information Management (AROSIM 2018)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 856))

Abstract

Altmetrics are indicators that have been proposed as alternatives to citation counts for academic publication impact assessment. Altmetrics may be valued for their speed or ability to reflect the non-scholarly or societal impacts of research. Evidence supports these claims for some altmetrics but many are limited in coverage (the proportion of outputs that have non-zero values) or ability to reflect societal impact. This article describes data sources for altmetrics, indicator formulae, and strategies for applying them for different tasks. It encompasses traditional altmetrics as well webometric and usage indicators.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Priem, J., Taraborelli, D., Groth, P., Neylon, C.: Altmetrics: a manifesto (2010). http://altmetrics.org/manifesto/

  2. Holmberg, K.: Altmetrics for Information Professionals Past, Present and Future. Chandos, Oxford (2015)

    Google Scholar 

  3. Vaughan, L., Hysen, K.: Relationship between links to journal Web sites and impact factors. Aslib Proc. 54(6), 356–361 (2002)

    Article  Google Scholar 

  4. Thelwall, M., Kousha, K.: Web indicators for research evaluation, Part 1: citations and links to academic articles from the web. El Profesional de la Información 24(5), 587–606 (2015). https://doi.org/10.3145/epi.2015.sep.08

    Article  Google Scholar 

  5. Kousha, K., Thelwall, M.: Are Wikipedia citations important evidence of the impact of scholarly articles and books? J. Assoc. Inf. Sci. Technol. 68(3), 762–779 (2017). https://doi.org/10.1002/asi.23694

    Article  Google Scholar 

  6. Kousha, K., Thelwall, M.: Web indicators for research evaluation, Part 3: books and non-standard outputs. El Profesional de la Información 24(6), 724–736 (2015)

    Article  Google Scholar 

  7. Sugimoto, C.R., Work, S., Larivière, V., Haustein, S.: Scholarly use of social media and altmetrics: a review of the literature. J. Assoc. Inf. Sci. Technol. 68(9), 2037–2062 (2017)

    Article  Google Scholar 

  8. Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S.: The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management. HEFCE, London (2015)

    Book  Google Scholar 

  9. Haustein, S., Siebenlist, T.: Applying social bookmarking data to evaluate journal usage. J. Informetr. 5, 446–457 (2011)

    Google Scholar 

  10. Piwowar, H., Priem, J.: The power of altmetrics on a CV. Bull. Assoc. Inf. Sci. Technol. 39(4), 10–13 (2013)

    Article  Google Scholar 

  11. Merton, R.K.: The Sociology of Science: Theoretical and Empirical Investigations. University of Chicago Press, Chicago (1973)

    Google Scholar 

  12. Krampen, G., Becker, R., Wahner, U., Montada, L.: On the validity of citation counting in science evaluation: content analyses of references and citations in psychological publications. Scientometrics 71(2), 191–202 (2007)

    Article  Google Scholar 

  13. Campbell, F.M.: National bias: a comparison of citation practices by health professionals. Bull. Med. Libr. Assoc. 78(4), 376 (1990)

    Google Scholar 

  14. Pasterkamp, G., Rotmans, J., de Kleijn, D., Borst, C.: Citation frequency: a biased measure of research impact significantly influenced by the geographical origin of research articles. Scientometrics 70(1), 153–165 (2007)

    Article  Google Scholar 

  15. Seglen, P.O.: Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthopaedica Scandinavica 69(3), 224–229 (1998)

    Article  Google Scholar 

  16. Althouse, B.M., West, J.D., Bergstrom, C.T., Bergstrom, T.: Differences in impact factor across fields and over time. J. Assoc. Inf. Sci. Technol. 60(1), 27–34 (2009)

    Article  Google Scholar 

  17. van Driel, M.L., Maier, M., Maeseneer, J.D.: Measuring the impact of family medicine research: scientific citations or societal impact? Fam. Pract. 24(5), 401–402 (2007)

    Article  Google Scholar 

  18. Glänzel, W., Schubert, A.: A new classification scheme of science fields and subfields designed for scientometric evaluation purposes. Scientometrics 56(3), 357–367 (2003)

    Article  Google Scholar 

  19. Dinsmore, A., Allen, L., Dolby, K.: Alternative perspectives on impact: the potential of ALMs and altmetrics to inform funders about research impact. PLoS Biol. 12(11), e1002003 (2014)

    Article  Google Scholar 

  20. Fenner, M.: What can article-level metrics do for you? PLoS Biol. 11(10), e1001687 (2013)

    Article  Google Scholar 

  21. Maggio, L.A., Meyer, H.S., Artino, A.R.: Beyond citation rates: a real-time impact analysis of health professions education research using altmetrics. Acad. Med. 92(10), 1449–1455 (2017)

    Article  Google Scholar 

  22. Colquhoun, D., Plested, A.: Why you should ignore altmetrics and other bibliometric nightmares (2014). http://www.dcscience.net/2014/01/16/why-you-should-ignore-altmetrics-and-other-bibliometric-nightmares/

  23. Livas, C., Delli, K.: Looking beyond traditional metrics in orthodontics: an altmetric study on the most discussed articles on the web. Eur. J. Orthod. (2017). https://doi.org/10.1093/ejo/cjx050

  24. Mas-Bleda, A., Thelwall, M.: Can alternative indicators overcome language biases in citation counts? A comparison of Spanish and UK research. Scientometrics 109(3), 2007–2030 (2016)

    Article  Google Scholar 

  25. Ravenscroft, J., Liakata, M., Clare, A., Duma, D.: Measuring scientific impact beyond academia: an assessment of existing impact metrics and proposed improvements. PLoS ONE 12(3), e0173152 (2017)

    Article  Google Scholar 

  26. Wouters, P., Costas, R.: Users, narcissism and control: tracking the impact of scholarly publications in the 21st century. In: Science and Technology Indicators 2012 (STI 2012), pp. 847–857. SURF Foundation, Utrecht (2012)

    Google Scholar 

  27. NISO: Outputs of the NISO Alternative Assessment Metrics Project (2016). http://www.niso.org/apps/group_public/download.php/17091/NISO%20RP-25-2016%20Outputs%20of%20the%20NISO%20Alternative%20Assessment%20Project.pdf

  28. Wilsdon, J., Bar-Ilan, J., Frodeman, R., Lex, E., Peters, I., Wouters, P.: Next-generation metrics: responsible metrics and evaluation for open science (2017). https://ec.europa.eu/research/openscience/index.cfm?pg=altmetrics_eg

  29. Robinson-García, N., Torres-Salinas, D., Zahedi, Z., Costas, R.: New data, new possibilities: exploring the insides of Altmetric.com. El Profesional de La Información 23(4), 359–366 (2014)

    Article  Google Scholar 

  30. Waltman, L., van Eck, N.J., van Leeuwen, T.N., Visser, M.S., van Raan, A.F.: Towards a new crown indicator: an empirical analysis. Scientometrics 87(3), 467–481 (2011)

    Article  Google Scholar 

  31. Thelwall, M.: Three practical field normalised alternative indicator formulae for research evaluation. J. Informetr. 11(1), 128–151 (2017). https://doi.org/10.1016/j.joi.2016.12.002

    Article  Google Scholar 

  32. Haustein, S., Larivière, V., Thelwall, M., Amyot, D., Peters, I.: Tweets vs. Mendeley readers: how do these two social media metrics differ? IT Inf. Technol. 56(5), 207–215 (2014)

    Google Scholar 

  33. Erdt, M., Nagarajan, A., Sin, S.C.J., Theng, Y.L.: Altmetrics: an analysis of the state-of-the-art in measuring research impact on social media. Scientometrics 109(2), 1117–1166 (2016)

    Article  Google Scholar 

  34. Thelwall, M., Sud, P.: Mendeley readership counts: an investigation of temporal and disciplinary differences. J. Assoc. Inf. Sci. Technol. 57(6), 3036–3050 (2016). https://doi.org/10.1002/asi.2355

    Article  Google Scholar 

  35. Borrego, A., Fry, J.: Measuring researchers’ use of scholarly information through social bookmarking data: a case study of BibSonomy. J. Inf. Sci. 38(3), 297–308 (2012)

    Article  Google Scholar 

  36. Costas, R., Zahedi, Z., Wouters, P.: Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. J. Assoc. Inf. Sci. Technol. 66(10), 2003–2019 (2015)

    Article  Google Scholar 

  37. Thelwall, M., Haustein, S., Larivière, V., Sugimoto, C.: Do altmetrics work? Twitter and ten other candidates. PLoS ONE 8(5), e64841 (2013). https://doi.org/10.1371/journal.pone.0064841

    Article  Google Scholar 

  38. Thelwall, M., Kousha, K.: Online presentations as a source of scientific impact? An analysis of PowerPoint files citing academic journals. J. Am. Soc. Inf. Sci. Technol. 59(5), 805–815 (2008)

    Article  Google Scholar 

  39. Thelwall, M., Kousha, K.: SlideShare presentations, citations, users and trends: a professional site with academic and educational uses. J. Assoc. Inf. Sci. Technol. 68(8), 1989–2003 (2017)

    Article  Google Scholar 

  40. Kousha, K., Thelwall, M.: Patent citation analysis with Google. J. Assoc. Inf. Sci. Technol. 68(1), 48–61 (2017)

    Article  Google Scholar 

  41. Shema, H., Bar-Ilan, J., Thelwall, M.: Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics. J. Am. Soc. Inf. Sci. Technol. 65(5), 1018–1027 (2014)

    Article  Google Scholar 

  42. Thelwall, M., Kousha, K., Abdoli, M.: Is medical research informing professional practice more highly cited? Evidence from AHFS DI Essentials in Drugs.com. Scientometrics 112(1), 509–527 (2017)

    Article  Google Scholar 

  43. Thelwall, M., Maflahi, N.: Guideline references and academic citations as evidence of the clinical value of health research. J. Assoc. Inf. Sci. Technol. 67(4), 960–966 (2016)

    Article  Google Scholar 

  44. Haustein, S., Bowman, T.D., Holmberg, K., Tsou, A., Sugimoto, C.R., Larivière, V.: Tweets as impact indicators: examining the implications of automated “bot” accounts on Twitter. J. Assoc. Inf. Sci. Technol. 67(1), 232–238 (2016)

    Article  Google Scholar 

  45. Sud, P., Thelwall, M.: Evaluating altmetrics. Scientometrics 98(2), 1131–1143 (2014)

    Article  Google Scholar 

  46. Thelwall, M.: Interpreting correlations between citation counts and other indicators. Scientometrics 108(1), 337–347 (2016). https://doi.org/10.1007/s11192-016-1973-7

    Article  Google Scholar 

  47. Thelwall, M.: Are Mendeley reader counts useful impact indicators in all fields? Scientometrics 113(3), 1721–1731 (2017). https://doi.org/10.1007/s11192-017-2557-x

    Article  Google Scholar 

  48. HEFCE: The Metric Tide: Correlation Analysis of REF2014 Scores and Metrics (Supplementary Report II to the Independent Review of the Role of Metrics in Research Assessment and Management) (2015). http://www.hefce.ac.uk/pubs/rereports/Year/2015/metrictide/Title,104463,en.html

  49. Halevi, G., Moed, H.F.: Usage patterns of scientific journals and their relationship with citations. In: Context Counts: Pathways to Master Big and Little Data, pp. 241–251 (2014)

    Google Scholar 

  50. Mohammadi, E., Thelwall, M., Kousha, K.: Can Mendeley bookmarks reflect readership? A survey of user motivations. J. Assoc. Inf. Sci. Technol. 67(5), 1198–1209 (2016). https://doi.org/10.1002/asi.23477

    Article  Google Scholar 

  51. Mohammadi, E., Thelwall, M., Haustein, S., Larivière, V.: Who reads research articles? An altmetrics analysis of Mendeley user categories. J. Assoc. Inf. Sci. Technol. 66(9), 1832–1846 (2015)

    Article  Google Scholar 

  52. Thelwall, M.: Why do papers have many Mendeley readers but few Scopus-indexed citations and vice versa? J. Librariansh. Inf. Sci. 49(2), 144–151 (2017). https://doi.org/10.1177/0961000615594867

    Article  Google Scholar 

  53. Thelwall, M., Tsou, A., Weingart, S., Holmberg, K., Haustein, S.: Tweeting links to academic articles. Cybermetrics 17(1) (2013). http://cybermetrics.cindoc.csic.es/articles/v17i1p1.html

  54. Tsou, A., Bowman, T.D., Ghazinejad, A., Sugimoto, C.R.: Who tweets about science? In: Proceedings of ISSI 2015 - 15th International Conference of the International Society for Scientometrics and Informetricspp, pp. 95–100. Boğaziçi University Printhouse, Istanbul (2015)

    Google Scholar 

  55. Shema, H., Bar-Ilan, J., Thelwall, M.: How is research blogged? A content analysis approach. J. Assoc. Inf. Sci. Technol. 66(6), 1136–1149 (2015). https://doi.org/10.1002/asi.23239

    Article  Google Scholar 

  56. Schloegl, C., Gorraiz, J.: Comparison of citation and usage indicators: the case of oncology journals. Scientometrics 82(3), 567–580 (2010)

    Article  Google Scholar 

  57. Moed, H.F.: Statistical relationships between downloads and citations at the level of individual documents within a single journal. J. Assoc. Inf. Sci. Technol. 56(10), 1088–1097 (2005)

    Article  Google Scholar 

  58. Wilkinson, D., Sud, P., Thelwall, M.: Substance without citation: evaluating the online impact of grey literature. Scientometrics 98(2), 797–806 (2014)

    Article  Google Scholar 

  59. Moed, H.F., Halevi, G.: On full text download and citation distributions in scientific-scholarly journals. J. Assoc. Inf. Sci. Technol. 67(2), 412–431 (2016)

    Article  Google Scholar 

  60. Kudlow, P., Cockerill, M., Toccalino, D., Dziadyk, D.B., Rutledge, A., Shachak, A., Eysenbach, G.: Online distribution channel increases article usage on Mendeley: a randomized controlled trial. Scientometrics 112(3), 1537–1556 (2017)

    Article  Google Scholar 

  61. Larivière, V., Archambault, É., Gingras, Y.: Long-term variations in the aging of scientific literature: from exponential growth to steady-state science (1900–2004). J. Assoc. Inf. Sci. Technol. 59(2), 288–296 (2008)

    Article  Google Scholar 

  62. Maflahi, N., Thelwall, M.: How quickly do publications get read? The evolution of Mendeley reader counts for new articles. J. Assoc. Inf. Sci. Technol. 69(1), 158–167 (2018). https://doi.org/10.1002/asi.23909

    Article  Google Scholar 

  63. Thelwall, M.: Are Mendeley reader counts high enough for research evaluations when articles are published? Aslib J. Inf. Manag. 69(2), 174–183 (2017). https://doi.org/10.1108/AJIM-01-2017-0028

    Article  Google Scholar 

  64. Thelwall, M.: Early Mendeley readers correlate with later citation counts. Scientometrics (in press). https://doi.org/10.1007/s11192-018-2715-9

  65. Eysenbach, G.: Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J. Med. Internet Res. 13(4), e123 (2011)

    Article  Google Scholar 

  66. Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., Rafols, I.: The Leiden Manifesto for research metrics. Nature 520(7548), 429–431 (2015)

    Article  Google Scholar 

  67. Kousha, K., Thelwall, M.: Assessing the impact of disciplinary research on teaching: an automatic analysis of online syllabuses. J. Am. Soc. Inform. Sci. Technol. 59(13), 2060–2069 (2008)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Wolverhampton, Wolverhampton, UK

    Mike Thelwall

Authors
  1. Mike Thelwall
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mike Thelwall .

Editor information

Editors and Affiliations

  1. Nanyang Technological University, Singapore, Singapore

    Mojisola Erdt

  2. Nanyang Technological University, Singapore, Singapore

    Aravind Sesagiri Raamkumar

  3. University of British Columbia, Vancouver, British Columbia, Canada

    Edie Rasmussen

  4. Nanyang Technological University, Singapore, Singapore

    Yin-Leng Theng

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Thelwall, M. (2018). Using Altmetrics to Support Research Evaluation. In: Erdt, M., Sesagiri Raamkumar, A., Rasmussen, E., Theng, YL. (eds) Altmetrics for Research Outputs Measurement and Scholarly Information Management. AROSIM 2018. Communications in Computer and Information Science, vol 856. Springer, Singapore. https://doi.org/10.1007/978-981-13-1053-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-1053-9_2

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-1052-2

  • Online ISBN: 978-981-13-1053-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation