Skip to main content
SpringerLink
Log in

Grand challenges in altmetrics: heterogeneity, data quality and dependencies

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

With increasing uptake among researchers, social media are finding their way into scholarly communication and, under the umbrella term altmetrics, are starting to be utilized in research evaluation. Fueled by technological possibilities and an increasing demand to demonstrate impact beyond the scientific community, altmetrics have received great attention as potential democratizers of the scientific reward system and indicators of societal impact. This paper focuses on the current challenges for altmetrics. Heterogeneity, data quality and particular dependencies are identified as the three major issues and discussed in detail with an emphasis on past developments in bibliometrics. The heterogeneity of altmetrics reflects the diversity of the acts and online events, most of which take place on social media platforms. This heterogeneity has made it difficult to establish a common definition or conceptual framework. Data quality issues become apparent in the lack of accuracy, consistency and replicability of various altmetrics, which is largely affected by the dynamic nature of social media events. Furthermore altmetrics are shaped by technical possibilities and are particularly dependent on the availability of APIs and DOIs, strongly dependent on data providers and aggregators, and potentially influenced by the technical affordances of underlying platforms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. http://www.niso.org/topics/tl/altmetrics_initiative/.

  2. http://www.niso.org/publications/newsline/2015/working_group_connection_apr2015.html#bi0.

  3. The loss might be avoided or at least mitigated by maintaining a dark archive, which was mentioned by Altmetric founder Euan Adie in a tweet: https://twitter.com/stew/status/595527260817469440.

  4. In the short history of altmetrics, such loss can already be observed for Connotea.

References

  • Alperin, J. P. (2015). Geographic variation in social media metrics: An analysis of Latin American journal articles. Aslib Journal of Information Management, 67(3), 289–304. doi:10.1108/AJIM-12-2014-0176.

    Article  MathSciNet  Google Scholar 

  • Bar-Ilan, J. (2014). JASIST@ Mendeley revisited. In altmetrics14: expanding impacts and metrics, workshop at web science conference 2014. Retrieved from http://files.figshare.com/1504021/JASIST_new_revised.pdf

  • Björneborn, L., & Ingwersen, P. (2004). Toward a basic framework for webometrics. Journal of the American Society for Information Science and Technology, 55(14), 1216–1227. doi:10.1002/asi.20077.

    Article  Google Scholar 

  • Bornmann, L., & Leydesdorff, L. (2013). The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000. Journal of Informetrics, 7(2), 286–291. doi:10.1016/j.joi.2012.12.003.

    Article  Google Scholar 

  • Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019. doi:10.1002/asi.23309.

    Article  Google Scholar 

  • Cronin, B. (2016). The Incessant chattering of texts. In C. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication. A Festschrift in Honor of Blaise Cronin (pp. 13–19). Berlin: De Gruyter.

    Google Scholar 

  • Cronin, B., Snyder, H. W., Rosenbaum, H., Martinson, A., & Callahan, E. (1998). Invoked on the web. Journal of the American Society for Information Science, 49(14), 1319–1328. doi:10.1002/(SICI)1097-4571(1998)49:14<1319:AID-ASI9>3.0.CO;2-W.

    Article  Google Scholar 

  • Dahler-Larsen, P. (2012). The evaluation society. Stanford, CA: Stanford Business Books, an imprint of Stanford University Press.

    Google Scholar 

  • Dinsmore, A., Allen, L., & Dolby, K. (2014). Alternative perspectives on impact: The potential of ALMs and altmetrics to inform funders about research impact. PLoS Biology, 12(11), e1002003.

    Article  Google Scholar 

  • Garfield, E. (1955). Citation indexes for science. A new dimension in documentation through association of ideas. Science, 122, 108–111.

    Article  Google Scholar 

  • Gilbert, G. N. (1977). Referencing as persuasion. Social Studies of Science, 7(1), 113–122.

    Article  Google Scholar 

  • Glänzel, W., & Gorraiz, J. (2015). Usage metrics versus altmetrics: Confusing terminology? Scientometrics, 102(3), 2161–2164. doi:10.1007/s11192-014-1472-7.

    Article  Google Scholar 

  • Gross, P. L. K., & Gross, E. M. (1927). College libraries and chemical education. Science, 66(1713), 385–389.

    Article  Google Scholar 

  • Haustein, S., Bowman, T. D., & Costas, R. (2015a). When is an article actually published? An analysis of online availability, publication, and indexation dates. In Proceedings of the 15th international society of scientometrics and informetrics conference (pp. 1170–1179). Istanbul, Turkey.

  • Haustein, S., Bowman, T. D., & Costas, R. (2016a). Interpreting “altmetrics”: viewing acts on social media through the lens of citation and social theories. In C. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication. A Festschrift in Honor of Blaise Cronin (pp. 372–405). Berlin: De Gruyter. Retrieved from http://arxiv.org/abs/1502.05701

  • Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R., & Larivière, V. (2016b). Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology, 67(1), 232–238. doi:10.1002/asi.23456.

    Article  Google Scholar 

  • Haustein, S., Costas, R., & Larivière, V. (2015b). Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(3), e0120495. doi:10.1371/journal.pone.0120495.

    Article  Google Scholar 

  • Haustein, S., & Larivière, V. (2015). The use of bibliometrics for assessing research: possibilities, limitations and adverse effects. In I. M. Welpe, J. Wollersheim, S. Ringelhan, & M. Osterloh (Eds.), Incentives and performance: Governance of knowledge-intensive organizations (pp. 121–139). Springer International Publishing. Retrieved from http://link.springer.com/chapter/10.1007/978-3-319-09785-5_8

  • Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It—Information Technology, 56(5), 207–215. doi:10.1515/itit-2014-1048.

    Google Scholar 

  • Haustein, S., Sugimoto, C., & Larivière, V. (2015c). Guest editorial: Social media in scholarly communication. Aslib Journal of Information Management,. doi:10.1108/AJIM-03-2015-0047.

    Google Scholar 

  • Higher Education Funding Council for England. (2011). Decisions on assessing research impact. Research Excellent Framework (REF) 2014 (Research Excellence Framework No. 01.2011). Retrieved from http://www.ref.ac.uk/media/ref/content/pub/decisionsonassessingresearchimpact/01_11.pdf

  • Jobmann, A., Hoffmann, C. P., Künne, S., Peters, I., Schmitz, J., & Wollnik-Korn, G. (2014). Altmetrics for large, multidisciplinary research groups: Comparison of current tools. Bibliometrie-Praxis Und Forschung, 3. Retrieved from http://www.bibliometrie-pf.de/index.php/bibliometrie/article/view/205

  • Lazarsfeld, P. F. (1993). On social research and its language. Chicago: University of Chicago Press.

    Google Scholar 

  • Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461–471. doi:10.1007/s11192-011-0580-x.

    Article  Google Scholar 

  • Lin, J., & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Information Standards Quarterly, 25(2), 20–26.

    Article  Google Scholar 

  • Mas-Bleda, A., Thelwall, M., Kousha, K., & Aguillo, I. F. (2014). Do highly cited researchers successfully use the social web? Scientometrics, 101(1), 337–356. doi:10.1007/s11192-014-1345-0.

    Article  Google Scholar 

  • Moed, H. F. (2016). Altmetrics as traces of the computerization of the research process. In C. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication. A Festschrift in Honor of Blaise Cronin (pp. 360–371). Berlin: De Gruyter.

    Google Scholar 

  • Moed, H. F., Burger, W. J. M., Frankfort, J. G., & Van Raan, A. F. J. (1985). The use of bibliometric data for the measurement of university research performance. Research Policy, 14(3), 131–149. doi:10.1016/0048-7333(85)90012-5.

    Article  Google Scholar 

  • Piwowar, H. (2013). Value all research products. Nature, 493, 159. doi:10.1038/493159a.

    Google Scholar 

  • Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: harnessing multidimensional indicators of performance (pp. 263–287). Cambridge, MA: MIT Press.

    Google Scholar 

  • Priem, J., & Hemminger, B. M. (2010). Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web. First Monday, 15(7). Retrieved from http://pear.accc.uic.edu/ojs/index.php/fm/rt/printerFriendly/2874/2570

  • Priem, J., Piwowar, H. A., & Hemminger, B. M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv Print, 1–17.

  • Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010, October 26). Altmetrics: A manifesto. Retrieved from http://altmetrics.org/manifesto/

  • Rousseau, R., & Ye, F. Y. (2013). A multi-metric approach for research evaluation. Chinese Science Bulletin, 58(26), 3288–3290. doi:10.1007/s11434-013-5939-3.

    Article  Google Scholar 

  • Rowlands, I., Nicholas, D., Russell, B., Canty, N., & Watkinson, A. (2011). Social media use in the research workflow. Learned Publishing, 24(3), 183–195. doi:10.1087/20110306.

    Article  Google Scholar 

  • Sugimoto, C. R. (Ed.). (2016). Theories of informetrics and scholarly communication. Berlin: De Gruyter.

    Google Scholar 

  • Taylor, M. (2013). Towards a common model of citation: Some thoughts on merging altmetrics and bibliometrics. Research Trends, 35, 1–6.

    Google Scholar 

  • Thelwall, M., & Wilson, P. (2015). Mendeley readership altmetrics for medical articles: An analysis of 45 fields. Journal of the Association for Information Science and Technology,. doi:10.1002/asi.23501.

    Google Scholar 

  • Van Noorden, R. (2014). Online collaboration: Scientists and the social network. Nature, 512(7513), 126–129. doi:10.1038/512126a.

    Article  Google Scholar 

  • Waltman, L., & Costas, R. (2014). F1000 recommendations as a potential new data source for research evaluation: A comparison with citations. Journal of the Association for Information Science and Technology, 65(3), 433–445. doi:10.1002/asi.23040.

    Article  Google Scholar 

  • Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., et al. (2015). The metric tide: report of the independent review of the role of metrics in research assessment and management.,. doi:10.13140/RG.2.1.4929.1363.

    Google Scholar 

  • Zahedi, Z., Bowman, T. D., & Haustein, S. (2014a). Exploring data quality and retrieval strategies for Mendeley reader counts. Presented at the SIG/MET workshop, ASIS&T 2014 annual meeting, Seattle. Retrieved from http://www.asis.org/SIG/SIGMET/data/uploads/sigmet2014/zahedi.pdf

  • Zahedi, Z., Costas, R., & Wouters, P. (2014b). Assessing the impact of publications saved by Mendeley users: Is there any different pattern among users? In IATUL conference, Espoo, Finland, June 2–5 2014. Retrieved from http://docs.lib.purdue.edu/iatul/2014/altmetrics/4

  • Zahedi, Z., Fenner, M., & Costas, R. (2014c). How consistent are altmetrics providers? Study of 1000 PLOS ONE publications using the PLOS ALM, Mendeley and Altmetric. com APIs. In altmetrics 14. Workshop at the web science conference, Bloomington, USA. Retrieved from http://files.figshare.com/1945874/How_consistent_are_altmetrics_providers__5_.pdf

  • Zahedi, Z., Fenner, M., & Costas, R. (2015). Consistency among altmetrics data provider/aggregators: What are the challenges? In altmetrics15: 5 years in, what do we know? Amsterdam, The Netherlands. Retrieved from http://altmetrics.org/wp-content/uploads/2015/09/altmetrics15_paper_14.pdf

Download references

Acknowledgments

The author acknowledges funding from the Alfred P. Sloan Foundation Grant # 2014-3-25. She would also like to thank Vincent Larivière for stimulating discussions and helpful suggestions on the manuscript, as well as Sam Work for proofreading.

Author information

Authors and Affiliations

  1. EBSI, Université de Montréal, CP 6128, Station Centre-Ville, Montreal, QC, H3C 3J7, Canada

    Stefanie Haustein

Authors
  1. Stefanie Haustein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefanie Haustein.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haustein, S. Grand challenges in altmetrics: heterogeneity, data quality and dependencies. Scientometrics 108, 413–423 (2016). https://doi.org/10.1007/s11192-016-1910-9

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-016-1910-9

Keywords

Search

Navigation