Research productivity of health care policy faculty: a cohort study of Harvard Medical School

Abstract

In today’s publish or perish environment, the scholarly impact of a research article holds great importance. The present study examined 2343 articles having both altmetric attention scores and citations published by 22 core health care policy faculty members at Harvard Medical School. Web of Science was used to retrieve the citations, whereas Altmetric Explorer was used to determine the altmetric attention score. The evaluation metrics in this study were focused on article-level information to determine each faculty member’s contribution to the health care policy department collected in November 2018. This paper further determined the predictive capability of altmetric attention score for the increase of citations from November 2018 to January 2020. The findings showed J. Michael McWilliams (Professor); Anupam B. Jena (Associate Professor); and Zirui Song (Assistant Professor) were the faculty members with highest altmetric attention score while Ronald C. Kessler (Professor); Nicole Maestas (Associate Professor); and Zirui Song (Assistant Professor) were the faculty members with the highest citation value. The lowest percentage (5.84%) of articles without any citations were published by the professors, whereas the lowest percentage (4.84%) of articles without any social media mention were published by the assistant professors. The health care faculty had the highest percentage of Mendeley readerships followed by tweets, news, and blogs mention in comparison to a meager percentage of policy-related documents mention for their retrieved articles. A significant strong positive correlation (\(r_{\text{R}}\) > 0.4) was observed between the aggregated ranked altmetric attention scores and ranked citation/increased citation values for all the faculty members. This study can be extended to determine the research productivity of any department, discipline, or group of faculty members.

This is a preview of subscription content, log in to check access.

Fig. 1

References

  1. Akoglu, H. (2018). User’s guide to correlation coefficients. Turkish Journal of Emergency Medicine,18(3), 91–93. https://doi.org/10.1016/j.tjem.2018.08.001.

    Article  Google Scholar 

  2. Alhoori, H., Choudhury, S. R., Kanan, T., Tech, V., Fox, E. A., Tech, V., et al. (2015). On the relationship between open access and altmetrics. https://www.ideals.illinois.edu/bitstream/handle/2142/73451/212_ready.pdf?sequence=2. Retrieved February 3, 2020.

  3. Altmetric. (2020). https://www.altmetric.com/. Retrieved February 2, 2020.

  4. Antelman, K. (2004). Do open-access articles have a greater research impact? College & Research Libraries,65(5), 372–382. https://doi.org/10.5860/crl.65.5.372.

    Article  Google Scholar 

  5. Aoun, S. G., Bendok, B. R., Rahme, R. J., Dacey, R. G., & Batjer, H. H. (2013). Standardizing the evaluation of scientific and academic performance in neurosurgery—Critical review of the “h” index and its variants. World Neurosurgery,80(5), e85–e90.

    Article  Google Scholar 

  6. Azer, S. A., & Azer, S. (2019). Top-cited articles in medical professionalism: A bibliometric analysis versus altmetric scores. British Medical Journal Open,9(7), e029433. https://doi.org/10.1136/bmjopen-2019-029433.

    Article  Google Scholar 

  7. Bornmann, L., & Haunschild, R. (2016). How to normalize Twitter counts? A first attempt based on journals in the Twitter Index. Scientometrics,107(3), 1405–1422. https://doi.org/10.1007/s11192-016-1893-6.

    Article  Google Scholar 

  8. Choi, M., Holliday, E. B., Jagsi, R., Wilson, L. D., Fuller, C. D., & Thomas, C. R. (2014). Citation-based estimation of scholarly activity among domestic academic radiation oncologists: Five-year update. Journal of Radiation Oncology,3(1), 115–122.

    Article  Google Scholar 

  9. Costello, L. C. (2010). Perspective: Is NIH funding the “best science by the best scientists”? A critique of the NIH R01 research grant review policies. Academic Medicine,85(5), 775–779.

    Article  Google Scholar 

  10. Danthi, N., Wu, C. O., Shi, P., et al. (2014). Percentile ranking and citation impact of a large cohort of National Heart, Lung, and Blood Institute-funded cardiovascular R01 grants. Circulation Research,114(4), 600–606.

    Article  Google Scholar 

  11. Harnad, S., & Brody, T. (2004). Comparing the impact of open access vs. non OA articles in the same journals. D-Lib Magazine, 10(6). http://www.dlib.org/dlib/june04/harnad/06harnad.html. Retrieved February 3, 2020.

  12. Hassan, S.-U., Imran, M., Gillani, U., Aljohani, N. R., Bowman, T. D., & Didegah, F. (2017). Measuring social media activity of scientific literature: An exhaustive comparison of scopus and novel altmetrics big data. Scientometrics,113(2), 1037–1057. https://doi.org/10.1007/s11192-017-2512-x.

    Article  Google Scholar 

  13. Haunschild, R., & Bornmann, L. (2016). Normalization of Mendeley reader counts for impact assessment. Journal of Informatics,10(1), 62–73. https://doi.org/10.1016/j.joi.2015.11.003.

    Article  Google Scholar 

  14. Haunschild, R., & Bornmann, L. (2017). How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics,110(3), 1209–1216. https://doi.org/10.1007/s11192-016-2237-2.

    Article  Google Scholar 

  15. Health Care Policy. (2020). Harvard University. https://healthcarepolicy.hms.harvard.edu/about. Retrieved February 2, 2020.

  16. Heydari, S., Shekofteh, M., & Kazerani, M. (2019). Relationship between altmetrics and citations a study on the highly cited research papers. DESIDOC Journal of Library & Information Technology,39(4), 169–174. https://doi.org/10.14429/djlit.39.4.14204.

    Article  Google Scholar 

  17. Holmberg, K., & Park, H. W. (2018). An altmetric investigation of the online visibility of South Korea-based scientific journals. Scientometrics,117(1), 603–613. https://doi.org/10.1007/s11192-018-2874-8.

    Article  Google Scholar 

  18. Huang, W., Wang, P., & Wu, Q. (2018). A correlation comparison between Altmetric Attention Scores and citations for six PLOS journals. PLoS ONE,13(4), e0194962. https://doi.org/10.1371/journal.pone.0194962.

    Article  Google Scholar 

  19. Kousha, K., & Abdoli, M. (2010). The citation impact of open access agricultural research: A comparison between OA and non-OA publications. Online Information Review,34(5), 772–785. https://doi.org/10.1108/14684521011084618.

    Article  Google Scholar 

  20. Lawrence, S. (2001). Online or invisible? Nature,411(6837), 521.

    Article  Google Scholar 

  21. Livas, C., & Delli, K. (2018). Looking beyond traditional metrics in orthodontics: An altmetric study on the most discussed articles on the web. European Journal of Orthodontics,40(2), 193–199. https://doi.org/10.1093/ejo/cjx050.

    Article  Google Scholar 

  22. Maggio, L. A., Leroux, T., Meyer, H. S., & Artino, A. R. (2018). Exploring the relationship between altmetrics and traditional measures of dissemination in health professions education. bioRxiv. https://doi.org/10.1101/260059.

    Article  Google Scholar 

  23. Merton, R. K. (1968). The matthew effect in science: The reward and communication systems of science are considered. Science,159(3810), 56–63. https://doi.org/10.1126/science.159.3810.56.

    Article  Google Scholar 

  24. Ortega, J. L. (2018). Reliability and accuracy of altmetric providers: A comparison among Altmetric.com, PlumX and Crossref Event Data. Scientometrics,116(3), 2123–2138. https://doi.org/10.1007/s11192-018-2838-z.

    Article  Google Scholar 

  25. Salajegheh, M., & Dayari, S. (2019). Comparing the citations counts and altmetrics of the top medical science journals in scopus. International Journal of Information Science and Management,17(1), 59–72.

    Google Scholar 

  26. Shu, F., Quan, W., Chen, B., Qiu, J., Sugimoto, C. R., & Larivière, V. (2020). The role of web of science publications in China’s tenure system. Scientometrics. https://doi.org/10.1007/s11192-019-03339-x.

    Article  Google Scholar 

  27. Susarla, H. K., Dhar, V., Karimbux, N. Y., & Tinanoff, N. (2017). Do standard bibliometric measures correlate with academic rank of full-time pediatric dentistry faculty members? Journal of Dental Education,81(4), 427–432.

    Article  Google Scholar 

  28. Syamili, C., & Rekha, R. V. (2017). Do altmetric correlate with citation? A study based on PLOS ONE journal. COLLNET Journal of Scientometrics and Information Management,11(1), 103–117. https://doi.org/10.1080/09737766.2016.1260815.

    Article  Google Scholar 

  29. Thelwall, M. (2018). Using altmetrics to support research evaluation. In M. Erdt, A. Sesagiri Raamkumar, E. Rasmussen, & Y.-L. Theng (Eds.), Altmetrics for research outputs measurement and scholarly information management (pp. 11–28). Singapore: Springer.

    Google Scholar 

  30. Web of Science. (2020). Clarivate analytics. http://login.webofknowledge.com/error/Error?Error=IPError&PathInfo=%2F&RouterURL=http%3A%2F%2Fwww.webofknowledge.com%2F&Domain=.webofknowledge.com&Src=IP&Alias=WOK5. Retrieved February 2, 2020.

  31. Zhang, X., Wang, X., Zhao, H., Ordóñez de Pablos, P., Sun, Y., & Xiong, H. (2019). An effectiveness analysis of altmetrics indices for different levels of artificial intelligence publications. Scientometrics,119(3), 1311–1344. https://doi.org/10.1007/s11192-019-03088-x.

    Article  Google Scholar 

Download references

Acknowledgements

The author wishes to thank Altmetric for providing this study’s data free of charge for the research purpose.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Manika Lamba.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (XLSX 272 kb)

Appendix 1: sources and their weights in calculating altmetric attention score

Appendix 1: sources and their weights in calculating altmetric attention score

Sources Weight
News 8
Blogs 5
Wikipedia pages 3
Policy documents (per-source) 3
Twitter 1
Sina Weibo 1
F1000/Publons/Pubpeer 1
LinkedIna,b 0.5
Q&Aa 0.25
Facebook (public pages)a 0.25
Video/YouTubea 0.25
Reddit/Pinteresta,b 0.25
Open Syllabus 1
Google+ 1
Patents 3
  1. Data are from https://help.altmetric.com/support/solutions/articles/6000060969-how-is-the-altmetric-scorecalculated—(accessed on 01 February 2019)
  2. aThese scores are rounded up to a whole number at the aggregated level. For example, five, six, seven or eight Facebook posts will have the same score 2
  3. bLinkedIn and Pinterest have since been deprecated as sources due to login requirements

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lamba, M. Research productivity of health care policy faculty: a cohort study of Harvard Medical School. Scientometrics 124, 107–130 (2020). https://doi.org/10.1007/s11192-020-03433-5

Download citation

Keywords

  • Altmetric explorer
  • Web of science
  • Comparison analysis
  • Author productivity
  • Scholarly communication
  • Research evaluation