Advertisement

Scientometrics

, Volume 121, Issue 1, pp 261–283 | Cite as

Building journal’s long-term impact: using indicators detected from the sustained active articles

  • Mingyang Wang
  • Shijia Jiao
  • Kah-Hin Chai
  • Guangsheng ChenEmail author
Article
  • 70 Downloads

Abstract

The Journal’s Impact Factor is an appropriate measure of recent concern rather than an effective measure of long-term impact of journals. This paper is mainly to find indicators that can effectively quantify the long-term impact of journal, with the aim to provide more useful supplementary information for journal evaluation. By examining the correlation between articles’ past citations and their future citations in different time windows, we found that the articles which were referenced in the past years will yield useful information also in the future. The age characteristics of these sustained active articles in journals provide clues for establishing long-term impact metrics for journals. A new indicator: h1-index was proposed to extract the active articles with at least the same number of citations as the h1-index in the statistical year. On this basis, four indicators describing the age characteristics of active articles were proposed to quantify the long-term impact of journals. The experimental results show that these indicators have a high correlation with the journal’s total citations, indicating that it is appropriate for these indicators to express the impact of the journal. Combining the average age of the active articles with the impact factors of journals, we found that some journals with short-term attraction strategies can also build long-term impact. The indicators presented in this paper that describe the long-term impact of journals will be a useful complement to journal quality assessment.

Keywords

Impact evaluation Long-term impact Active articles h1-Index Impact factor 

Notes

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant No. 71473034), the Heilongjiang Provincial Natural Science Foundation of China (Grant No. LH2019G001), and the financial assistance from Postdoctoral Scientific Research Developmental Fund of Heilongjiang Province (Grant No. LBH-Q16003).

Supplementary material

11192_2019_3196_MOESM1_ESM.docx (773 kb)
Supplementary material 1 (DOCX 773 kb)

References

  1. Andrei, T., Teodorescu, D., & Mirica, A. (2016). Beyond the impact factor: Measuring the international visibility of Romanian social sciences journals. Scientometrics, 108(1), 1–20.Google Scholar
  2. Antonoyiannakis, M. (2018). Impact factors and the central limit theorem: Why citation average are scale dependent. Journal of Informetrics, 12(4), 1072–1088.Google Scholar
  3. Archambault, E., & Larivière, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649.Google Scholar
  4. Asnafi, S., Gunderson, T., McDonald, R. J., & Kallmes, D. F. (2017). Association of h-index of editorial board members and impact factor among radiology journals. Academic Radiology, 24(2), 119–123.Google Scholar
  5. Bador, P., & Lafouge, T. (2010). Comparative analysis between impact factor and h-index for pharmacology and psychiatry journals. Scientometrics, 84(1), 65–69.Google Scholar
  6. Barker, K. L., & Thyer, B. A. (2005). An empirical evaluation of the editorial practices of social work journals: Voices of authors pubished in 2000. Journal of Social Service Research, 32(1), 17–31.Google Scholar
  7. Bertoli-Barsotti, L., & Lando, T. (2017a). The h-index as an almost-exact function of some basic statistics. Scientometrics, 113(2), 1209–1228.Google Scholar
  8. Bertoli-Barsotti, L., & Lando, T. (2017b). A theoretical model of the relationship between the h-index and other simple citation indicators. Scientometrics, 111(3), 1415–1448.Google Scholar
  9. Bogocz, J., Bak, A., & Polanski, J. (2014). No free lunches in nature? An analysis of the regional distribution of the affiliations of nature publications. Scientometrics, 101(1), 547–568.Google Scholar
  10. Bornmann, L. (2017). Confidence intervals for Journal Impact Factors. Scientometrics, 111(3), 1869–1871.Google Scholar
  11. Bornmann, L., & Williams, R. (2017). Can the journal impact factor be used as a criterion for the selection of junior researchers? A large-scale empirical study based on researcher ID data. Journal of Informetrics, 11(3), 788–799.Google Scholar
  12. Braun, T., Glänzel, W., & Schubert, A. (2005). A Hirsch-type index for journals. The Scientist, 19(22), 8.Google Scholar
  13. Braun, T., Glänzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169–173.Google Scholar
  14. Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: Unintended consequences of journal rank. Frontiers in Human Neuroscience, 7, 291.Google Scholar
  15. Cameron, B. F. (2005). Trends in the usage of ISI bibliometric data: Uses, abuses, and implications. Portal: Libraries and the Academy, 5(1), 105–125.Google Scholar
  16. Campanario, J. M. (2018). Are leaders really leading? Journals that are first in web of science subject categories in the context of their groups. Scientometrics, 115(1), 111–130.Google Scholar
  17. Chi, P. S., & Glänzel, W. (2018). Comparison of citation and usage indicators in research assessment in scientific disciplines and journals. Scientometrics, 116(1), 537–554.Google Scholar
  18. Crookes, P. A., Reis, S. L., & Jones, S. C. (2010). The development of a ranking tool for refereed journals in which nursing and midwifery researchers publish their work. Nurse Education Today, 30(5), 420–427.Google Scholar
  19. da Silva, J. A. T. (2017). Does China need to rethink its metrics-and citation-based research rewards policies? Scientometrics, 112(3), 1853–1857.Google Scholar
  20. de Araujo, C. G. S., & Sardinha, A. (2011). H-index of the citing articles: A contribution to the evaluation of scientific production of experienced researchers. Revista Brasileira de Medicina do Esporte, 17(5), 358–362.Google Scholar
  21. Dempsey, J. A. (2009). Impact factor and its role in academic promotion: A statement adopted by the international Respiratory Journal Editors Roundtable. Journal of Applied Physiology, 107(4), 1005.Google Scholar
  22. Dorta-González, P., & Dorta-González, M. I. (2013). Impact maturity times and citation time windows: The 2-year maximum journal impact factor. Journal of Informetrics, 7(3), 593–602.Google Scholar
  23. Editors. (2006). The impact factor game. It is time to find a better way to assess the scientific literature. PLoS Medicine, 3(6), 707–708.Google Scholar
  24. Epstein, W. M. (2004). Confirmational response bias and the quality of the editorial processes among American social work journals. Research on Social Work Practice, 14(6), 450–458.Google Scholar
  25. Eyre-Walker, A., & Stoletzki, N. (2013). The assessment of science: The relative merits of post-publication review, the impact factor, and the number of citations. PLoS Biology, 11(10), e1001675.Google Scholar
  26. Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., & Karageorgopoulos, D. E. (2008). Comparison of SCImago journal rank indicator with journal impact factor. Faseb Journal, 22(8), 2623–2628.Google Scholar
  27. Fiala, J., Mares, J. J., & Sestak, J. (2017). Reflections on how to evaluate the professional value of scientific papers and their corresponding citations. Scientometrics, 112(1), 697–709.Google Scholar
  28. Fischer, I., & Steiger, H. J. (2018). Dynamics of journal impact factors and limits to their inflation. Journal of Scholarly Publishing, 50(1), 26–36.Google Scholar
  29. Fuyono, I., & Cyranoski, D. (2006). Cash for papers: Putting a premium on publication. Nature, 441(7095), 792.Google Scholar
  30. Garfield, E. (1955). Citation indexes to science: A new dimension in documentation through association of ideas. Science, 122, 108–111.Google Scholar
  31. Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.Google Scholar
  32. Garfield, E. (1996). How can impact factors be improved? British Medical Journal, 313(7054), 411–413.Google Scholar
  33. Garfield, E. (2001). Impact factors, and why they won’t go away. Nature, 411(6837), 522.Google Scholar
  34. Glänzel, W. (2008). On some new bibliometric applications of statistics related to the h-index. Scientometrics, 77(1), 187–196.Google Scholar
  35. Glänzel, W. (2009). The multi-dimensionality of journal impact. Scientometrics, 78(2), 355–374.Google Scholar
  36. Glänzel, W. (2010a). The role of the h-index and the characteristic scores and scales in testing the tail properties of scientometric distributions. Scientometrics, 83(3), 697–709.Google Scholar
  37. Glänzel, W. (2010b). On reliability and robustness of scientometrics indicators based on stochastic models. An evidence-based opinion paper. Journal of Informetrics, 4(3), 313–319.Google Scholar
  38. Glänzel, W. (2011). The application of characteristic scores and scales to the evaluation and ranking of scientific journals. Journal of Information Science, 37(1), 40–48.MathSciNetGoogle Scholar
  39. Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193.Google Scholar
  40. Glänzel, W., & Schubert, A. (2010). Hirsch-type characteristics of the tail of distributions. The genralised h-index. Journal of Informetrics, 4(1), 118–123.Google Scholar
  41. Glänzel, W., & Thijs, B. (2018). The role of baseline granularity for benchmarking citation impact. The case of CSS profiles. Scientometrics, 116(1), 521–536.Google Scholar
  42. Goncalves, R. R., Kieling, C., Bressan, R. A., Mari, J. J., & Rohde, L. A. (2009). The evaluation of scientific productivity in Brazil: An assessment of the mental health field. Scientometrics, 80(2), 529–537.Google Scholar
  43. Ha, T. C., Tan, S. B., & Soo, K. C. (2006). The journal impact factor: Too much of an impact? Annals of the Academy of Medicine, Singapore, 35(12), 911–916.Google Scholar
  44. Haghdoost, A., Zare, M., & Bazrafshan, A. (2014). How variable are the journal impact measures? Online Information Review, 38(6), 723–737.Google Scholar
  45. Hartley, J. (2012). To cite or not to cite: Author self-citations and the impact factor. Scientometrics, 92(2), 313–317.Google Scholar
  46. Harzing, A. W., & van der Wal, R. (2009). A Google Scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41–46.Google Scholar
  47. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.zbMATHGoogle Scholar
  48. Hsu, W. C., Tasi, C. F., & Li, J. H. (2015). A hybrid indicator for journal ranking: An example from the field of health care sciences and services. Online Information Review, 39(7), 858–869.Google Scholar
  49. Hunt, G. E., Cleary, M., & Walter, G. (2010). Psychiatry and the Hirsch h-index: The relationship between journal impact factors and accrued citations. Harvard Review of Psychiatry, 18(4), 207–219.Google Scholar
  50. Iglesias, J. E., & Pecharroman, C. (2007). Scaling the h-index for different scientific ISI fields. Scientometrics, 73(3), 303–320.Google Scholar
  51. Jacob, J. H., Lehrl, S., & Henkel, A. W. (2007). Early recognition of high quality researchers of the German psychiatry by worldwide accessible bibliometric indicators. Scientometrics, 73(2), 117–130.Google Scholar
  52. Jung, H., Seo, I., Kim, J., & Kim, B. K. (2017). Factors affecting government-funded research quality. Asian Journal of Technology Innovation, 25(3), 447–467.Google Scholar
  53. Kumaran, M., & Ha, C. (2017). Knowledge of journal impact factors among nursing faculty: A cross-sectional study. Journal of the Medical Library Association, 105(2), 140–144.Google Scholar
  54. Leydesdorff, L. (2008). Caveats for the use of citation indicators in research and journal evaluations. Journal of the American Society for Information Science and Technology, 59(2), 278–287.Google Scholar
  55. Leydesdorff, L. (2012). Alternatives to the journal impact factor: 13 and the top-10% (or top-25%) of the most-highly cited papers. Scientometrics, 92(2), 355–365.Google Scholar
  56. Leydesdorff, L. (2013). An evaluation of impacts in “Nanoscience and nanotechnology”: Steps towards standards for citation analysis. Scientometrics, 94(1), 35–55.Google Scholar
  57. Lindner, M. D., Torralba, K. D., & Khan, N. A. (2018). Scientific productivity: An exploratory study of metrics and incentives. PLoS ONE, 13(4), e0195321.Google Scholar
  58. Malesios, C. (2016). Measuring the robustness of the journal h-index with respect to publication and citation values: A Bayesian sensitivity analysis. Journal of Informetrics, 10(3), 719–731.Google Scholar
  59. McGarty, C. (2000). The citation impact factor in social psychology: A bad statistic that encourages bad science? Current Research in Social Psychology, 5(1), 1–16.Google Scholar
  60. McVeigh, M. E., & Mann, S. J. (2009). The journal impact factor denominator: Defining citable (counted) items. JAMA-Journal of the American Medical Association, 302(10), 1107–1109.Google Scholar
  61. Mimouni, M., Ratmansky, M., Sacher, Y., Aharoni, S., & Mimouni-Bloch, A. (2016). Self-citation rate and impact factor in pediatrics. Scientometrics, 108(3), 1455–1460.Google Scholar
  62. Mingers, J., Macri, F., & Petrovici, D. (2012). Using the h-index to measure the quality of journals in the field of business and management. Information Processing and Management, 48(2), 234–241.Google Scholar
  63. Mirnezami, S. R., Beaudry, C., & Lariviere, V. (2016). What determines researchers’ scientific impact? A case study of Quebec researchers. Science and Publicpolicy, 43(2), 262–274.Google Scholar
  64. Olden, J. D. (2007). How do ecological journals stack-up? Ranking of scientific quality according to the h-index. Ecoscience, 14(3), 370–376.Google Scholar
  65. Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research, 33(1), 1–7.Google Scholar
  66. Packalen, M., & Bhattacharya, J. (2017). Neophilia ranking of scientific journals. Scientometrics, 110(1), 43–64.Google Scholar
  67. Polit, D. F., & Northam, S. (2011). Impact factors in nursing journals. Nursing Outlook, 59(1), 18–28.Google Scholar
  68. Prathap, G., Mini, S., & Nishy, P. (2016). Does high impact factor successfully predict future citations? An analysis using Peirce’s measure. Scientometrics, 108(3), 1043–1047.Google Scholar
  69. Prozesky, H., & Boshoff, N. (2012). Bibilometrics as a tool for measuring gender-specific research performance: An example from South African invasion ecology. Scientometrics, 90(2), 383–406.Google Scholar
  70. Racz, A., & Markovic, S. (2018). “Worth(less) papers”-are journal impact factor and number of citations suitable indicators to evaluate quality of scientists? Nova Prisutnost, 16(2), 369–389.Google Scholar
  71. Romero-Torres, M., Acosta-Moreno, L. A., & Tejada-Gomez, M. A. (2013). Use of the h index to rank scientific Latin American journals: Colombia a case study. Revista Espanola de Documentacion Cientifica, 36(1), e003.Google Scholar
  72. Rousseau, R. (2016). Positive correlation between journal production and journal impact factors. Journal of Informetrics, 10(2), 567–568.Google Scholar
  73. Saad, G. (2006). Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics, 69(1), 117–120.Google Scholar
  74. Sangwal, K. (2013). Some citation-related characteristics of scientific journals published in individual countries. Scientometrics, 97(3), 719–741.Google Scholar
  75. Schubert, A., & Glänzel, W. (2007). A systematic analysis of Hirsch-type indices for journals. Journal of Informetrics, 1(3), 179–184.Google Scholar
  76. Sebire, N. J. (2008). H-index and impact factors: Assessing the clinical impact of researchers and specialist journals. Ultrasound in Obstetrics and Gynecology, 32(7), 843–845.Google Scholar
  77. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 498–502.Google Scholar
  78. Shen, Z. S., Yang, L. Y., & Wu, J. S. (2018). Lognormal distribution of citation counts is the reason for the relation between Impact Factors and Citation Success Index. Journal of Informetrics, 12(1), 153–157.Google Scholar
  79. Smith, K. M., Crookes, E., & Crookes, P. A. (2013). Measuring research ‘impact’ for academic promotion: Issues from the literature. Journal of Higher Education Policy and Management, 35(4), 410–420.Google Scholar
  80. Solarino, S. (2012). Impact factor, citation index, h-index: Are researchers still free to choose where and how to publish their results? Annals of Geophysics, 55(3), 473–477.Google Scholar
  81. Sternberg, R. J. (2018). Evaluating merit among scientists. Journal of Applied Research in Memory and Cognition, 7(2), 209–216.MathSciNetGoogle Scholar
  82. Tahamtan, I., Afshar, A. S., & Ahamdzadeh, K. (2016). Factors affecting number of citations: A comprehensive review of the literature. Scientometrics, 107(3), 1195–1225.Google Scholar
  83. Valderrama, P., Escabias, M., Jimenez-Contreras, E., Rodriguez-Archilla, A., & Valderrama, M. J. (2018). Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: Application to the field of Dentistry. Scientometrics, 115(2), 1087–1095.Google Scholar
  84. Van Raan, A. F. J. (2012). Properties of journal impact in relation to bibliometric research group performance indicators. Scientometrics, 92(2), 457–469.Google Scholar
  85. Vanclay, J. K. (2008). Ranking forestry journals using the h-index. Journal of Informetrics, 2(4), 326–334.Google Scholar
  86. Vinkler, P. (2001). An attempt for defining some basic categories of scientometrics and classifying the indicators of evaluative scientometrics. Scientometrics, 50(3), 539–544.Google Scholar
  87. Vinkler, P. (2002). The institutionalization of scientific information. The scientometric model (ISI-S MODEL). Library Trends, 50(3), 553–569.Google Scholar
  88. Vinkler, P. (2004). Characterization of the impact of sets of scientific papers: The garfield (impact) factor. Journal of the American Society for Information Science and Technology, 55(5), 431–435.Google Scholar
  89. Vinkler, P. (2009a). Introducing the Current Contribution Index for characterizing the recent, relevant impact of journals. Scientometrics, 79(2), 409–420.Google Scholar
  90. Vinkler, P. (2009b). The π-index: A new indicator for assessing scientific impact. Journal of Information Science, 35(5), 602–612.Google Scholar
  91. Vinkler, P. (2010). The π v-index: A new indicator to characterize the impact of journals. Scientometrics, 82(3), 461–475.Google Scholar
  92. Vinkler, P. (2012). The Garfield impact factor, one of the fundamental indicators in scientometrics. Scientometrics, 92(2), 471–483.Google Scholar
  93. Vinkler, P. (2017). The size and impact of the elite set of publications in scientometric assessments. Scientometrics, 110(1), 163–177.Google Scholar
  94. Vinluan, L. R. (2012). Research productivity in education and psychology in the Philippines and comparison with ASEAN countries. Scientometrics, 91(1), 277–294.Google Scholar
  95. Xia, J. F., & Smith, M. P. (2018). Alternative journal impact factors in open access publishing. Learned Publishing, 31(4), 403–411.Google Scholar
  96. Yan, E. J., Wu, C. J., & Song, M. (2018). The funding factor: A cross-disciplinary examination of the association between research funding and citation impact. Scientometrics, 115(1), 369–384.Google Scholar
  97. Yang, D. H., Li, X., Sun, X. X., & Wan, J. (2016). Detecting impact factor manipulation with data mining techniques. Scientometrics, 109(3), 1989–2005.Google Scholar
  98. Yeung, A. W. K. (2019). Higher impact factor of neuroimaging journals is associated with larger number of articles published and smaller percentage of uncited articles. Scientometrics, 12, 523.Google Scholar
  99. Yu, G., & Li, Y. J. (2007). Parameter identification of the observed citation distribution. Scientometrics, 71(2), 339–348.Google Scholar
  100. Yu, G., & Li, Y. J. (2010). Identification of referencing and citation processes of scientific journals based on the citation distribution model. Scientometrics, 82(2), 249–261.Google Scholar
  101. Yu, G., Wang, X. H., & Yu, D. R. (2005). The influence of publication delays on impact factors. Scientometrics, 64(2), 235–246.Google Scholar
  102. Yuen, J. (2018). Comparision of impact factor, eigenfactor metrics, and scimago journal rank indicator and h-index for neurosurgical and spinal surgical journals. World Neurosurgery, 119, E328–E337.Google Scholar
  103. Yuret, T. (2018). Author-weighted impact factor and reference return ratio: Can we attain more equality among fields? Scientometrics, 116(3), 2097–2111.Google Scholar
  104. Zhang, L., Thijs, B., & Glänzel, W. (2011). The diffusion of H-related literature. Journal of Informetrics, 5(4), 583–593.Google Scholar
  105. Zou, Y. W., & Laubichler, M. D. (2017). Measuring the contributions of Chinese scholars to the research field of systems biology from 2005 to 2013. Scientometrics, 110(3), 1615–1631.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2019

Authors and Affiliations

  1. 1.College of Information and Computer EngineeringNortheast Forestry UniversityHarbinPeople’s Republic of China
  2. 2.Industrial and Systems EngineeringNational University of SingaporeSingaporeSingapore

Personalised recommendations