Skip to main content
SpringerLink
Log in

Building journal’s long-term impact: using indicators detected from the sustained active articles

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

The Journal’s Impact Factor is an appropriate measure of recent concern rather than an effective measure of long-term impact of journals. This paper is mainly to find indicators that can effectively quantify the long-term impact of journal, with the aim to provide more useful supplementary information for journal evaluation. By examining the correlation between articles’ past citations and their future citations in different time windows, we found that the articles which were referenced in the past years will yield useful information also in the future. The age characteristics of these sustained active articles in journals provide clues for establishing long-term impact metrics for journals. A new indicator: h1-index was proposed to extract the active articles with at least the same number of citations as the h1-index in the statistical year. On this basis, four indicators describing the age characteristics of active articles were proposed to quantify the long-term impact of journals. The experimental results show that these indicators have a high correlation with the journal’s total citations, indicating that it is appropriate for these indicators to express the impact of the journal. Combining the average age of the active articles with the impact factors of journals, we found that some journals with short-term attraction strategies can also build long-term impact. The indicators presented in this paper that describe the long-term impact of journals will be a useful complement to journal quality assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Andrei, T., Teodorescu, D., & Mirica, A. (2016). Beyond the impact factor: Measuring the international visibility of Romanian social sciences journals. Scientometrics, 108(1), 1–20.

    Article  Google Scholar 

  • Antonoyiannakis, M. (2018). Impact factors and the central limit theorem: Why citation average are scale dependent. Journal of Informetrics, 12(4), 1072–1088.

    Article  Google Scholar 

  • Archambault, E., & Larivière, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649.

    Article  Google Scholar 

  • Asnafi, S., Gunderson, T., McDonald, R. J., & Kallmes, D. F. (2017). Association of h-index of editorial board members and impact factor among radiology journals. Academic Radiology, 24(2), 119–123.

    Article  Google Scholar 

  • Bador, P., & Lafouge, T. (2010). Comparative analysis between impact factor and h-index for pharmacology and psychiatry journals. Scientometrics, 84(1), 65–69.

    Article  Google Scholar 

  • Barker, K. L., & Thyer, B. A. (2005). An empirical evaluation of the editorial practices of social work journals: Voices of authors pubished in 2000. Journal of Social Service Research, 32(1), 17–31.

    Article  Google Scholar 

  • Bertoli-Barsotti, L., & Lando, T. (2017a). The h-index as an almost-exact function of some basic statistics. Scientometrics, 113(2), 1209–1228.

    Article  Google Scholar 

  • Bertoli-Barsotti, L., & Lando, T. (2017b). A theoretical model of the relationship between the h-index and other simple citation indicators. Scientometrics, 111(3), 1415–1448.

    Article  Google Scholar 

  • Bogocz, J., Bak, A., & Polanski, J. (2014). No free lunches in nature? An analysis of the regional distribution of the affiliations of nature publications. Scientometrics, 101(1), 547–568.

    Article  Google Scholar 

  • Bornmann, L. (2017). Confidence intervals for Journal Impact Factors. Scientometrics, 111(3), 1869–1871.

    Article  Google Scholar 

  • Bornmann, L., & Williams, R. (2017). Can the journal impact factor be used as a criterion for the selection of junior researchers? A large-scale empirical study based on researcher ID data. Journal of Informetrics, 11(3), 788–799.

    Article  Google Scholar 

  • Braun, T., Glänzel, W., & Schubert, A. (2005). A Hirsch-type index for journals. The Scientist, 19(22), 8.

    Google Scholar 

  • Braun, T., Glänzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169–173.

    Article  Google Scholar 

  • Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: Unintended consequences of journal rank. Frontiers in Human Neuroscience, 7, 291.

    Article  Google Scholar 

  • Cameron, B. F. (2005). Trends in the usage of ISI bibliometric data: Uses, abuses, and implications. Portal: Libraries and the Academy, 5(1), 105–125.

    Article  Google Scholar 

  • Campanario, J. M. (2018). Are leaders really leading? Journals that are first in web of science subject categories in the context of their groups. Scientometrics, 115(1), 111–130.

    Article  Google Scholar 

  • Chi, P. S., & Glänzel, W. (2018). Comparison of citation and usage indicators in research assessment in scientific disciplines and journals. Scientometrics, 116(1), 537–554.

    Article  Google Scholar 

  • Crookes, P. A., Reis, S. L., & Jones, S. C. (2010). The development of a ranking tool for refereed journals in which nursing and midwifery researchers publish their work. Nurse Education Today, 30(5), 420–427.

    Article  Google Scholar 

  • da Silva, J. A. T. (2017). Does China need to rethink its metrics-and citation-based research rewards policies? Scientometrics, 112(3), 1853–1857.

    Article  Google Scholar 

  • de Araujo, C. G. S., & Sardinha, A. (2011). H-index of the citing articles: A contribution to the evaluation of scientific production of experienced researchers. Revista Brasileira de Medicina do Esporte, 17(5), 358–362.

    Article  Google Scholar 

  • Dempsey, J. A. (2009). Impact factor and its role in academic promotion: A statement adopted by the international Respiratory Journal Editors Roundtable. Journal of Applied Physiology, 107(4), 1005.

    Article  Google Scholar 

  • Dorta-González, P., & Dorta-González, M. I. (2013). Impact maturity times and citation time windows: The 2-year maximum journal impact factor. Journal of Informetrics, 7(3), 593–602.

    Article  Google Scholar 

  • Editors. (2006). The impact factor game. It is time to find a better way to assess the scientific literature. PLoS Medicine, 3(6), 707–708.

  • Epstein, W. M. (2004). Confirmational response bias and the quality of the editorial processes among American social work journals. Research on Social Work Practice, 14(6), 450–458.

    Article  Google Scholar 

  • Eyre-Walker, A., & Stoletzki, N. (2013). The assessment of science: The relative merits of post-publication review, the impact factor, and the number of citations. PLoS Biology, 11(10), e1001675.

    Article  Google Scholar 

  • Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., & Karageorgopoulos, D. E. (2008). Comparison of SCImago journal rank indicator with journal impact factor. Faseb Journal, 22(8), 2623–2628.

    Article  Google Scholar 

  • Fiala, J., Mares, J. J., & Sestak, J. (2017). Reflections on how to evaluate the professional value of scientific papers and their corresponding citations. Scientometrics, 112(1), 697–709.

    Article  Google Scholar 

  • Fischer, I., & Steiger, H. J. (2018). Dynamics of journal impact factors and limits to their inflation. Journal of Scholarly Publishing, 50(1), 26–36.

    Article  Google Scholar 

  • Fuyono, I., & Cyranoski, D. (2006). Cash for papers: Putting a premium on publication. Nature, 441(7095), 792.

    Article  Google Scholar 

  • Garfield, E. (1955). Citation indexes to science: A new dimension in documentation through association of ideas. Science, 122, 108–111.

    Article  Google Scholar 

  • Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.

    Article  Google Scholar 

  • Garfield, E. (1996). How can impact factors be improved? British Medical Journal, 313(7054), 411–413.

    Article  Google Scholar 

  • Garfield, E. (2001). Impact factors, and why they won’t go away. Nature, 411(6837), 522.

    Article  Google Scholar 

  • Glänzel, W. (2008). On some new bibliometric applications of statistics related to the h-index. Scientometrics, 77(1), 187–196.

    Article  Google Scholar 

  • Glänzel, W. (2009). The multi-dimensionality of journal impact. Scientometrics, 78(2), 355–374.

    Article  Google Scholar 

  • Glänzel, W. (2010a). The role of the h-index and the characteristic scores and scales in testing the tail properties of scientometric distributions. Scientometrics, 83(3), 697–709.

    Article  Google Scholar 

  • Glänzel, W. (2010b). On reliability and robustness of scientometrics indicators based on stochastic models. An evidence-based opinion paper. Journal of Informetrics, 4(3), 313–319.

    Article  Google Scholar 

  • Glänzel, W. (2011). The application of characteristic scores and scales to the evaluation and ranking of scientific journals. Journal of Information Science, 37(1), 40–48.

    Article  MathSciNet  Google Scholar 

  • Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193.

    Article  Google Scholar 

  • Glänzel, W., & Schubert, A. (2010). Hirsch-type characteristics of the tail of distributions. The genralised h-index. Journal of Informetrics, 4(1), 118–123.

    Article  Google Scholar 

  • Glänzel, W., & Thijs, B. (2018). The role of baseline granularity for benchmarking citation impact. The case of CSS profiles. Scientometrics, 116(1), 521–536.

    Article  Google Scholar 

  • Goncalves, R. R., Kieling, C., Bressan, R. A., Mari, J. J., & Rohde, L. A. (2009). The evaluation of scientific productivity in Brazil: An assessment of the mental health field. Scientometrics, 80(2), 529–537.

    Article  Google Scholar 

  • Ha, T. C., Tan, S. B., & Soo, K. C. (2006). The journal impact factor: Too much of an impact? Annals of the Academy of Medicine, Singapore, 35(12), 911–916.

    Google Scholar 

  • Haghdoost, A., Zare, M., & Bazrafshan, A. (2014). How variable are the journal impact measures? Online Information Review, 38(6), 723–737.

    Article  Google Scholar 

  • Hartley, J. (2012). To cite or not to cite: Author self-citations and the impact factor. Scientometrics, 92(2), 313–317.

    Article  Google Scholar 

  • Harzing, A. W., & van der Wal, R. (2009). A Google Scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41–46.

    Article  Google Scholar 

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.

    Article  MATH  Google Scholar 

  • Hsu, W. C., Tasi, C. F., & Li, J. H. (2015). A hybrid indicator for journal ranking: An example from the field of health care sciences and services. Online Information Review, 39(7), 858–869.

    Article  Google Scholar 

  • Hunt, G. E., Cleary, M., & Walter, G. (2010). Psychiatry and the Hirsch h-index: The relationship between journal impact factors and accrued citations. Harvard Review of Psychiatry, 18(4), 207–219.

    Article  Google Scholar 

  • Iglesias, J. E., & Pecharroman, C. (2007). Scaling the h-index for different scientific ISI fields. Scientometrics, 73(3), 303–320.

    Article  Google Scholar 

  • Jacob, J. H., Lehrl, S., & Henkel, A. W. (2007). Early recognition of high quality researchers of the German psychiatry by worldwide accessible bibliometric indicators. Scientometrics, 73(2), 117–130.

    Article  Google Scholar 

  • Jung, H., Seo, I., Kim, J., & Kim, B. K. (2017). Factors affecting government-funded research quality. Asian Journal of Technology Innovation, 25(3), 447–467.

    Article  Google Scholar 

  • Kumaran, M., & Ha, C. (2017). Knowledge of journal impact factors among nursing faculty: A cross-sectional study. Journal of the Medical Library Association, 105(2), 140–144.

    Google Scholar 

  • Leydesdorff, L. (2008). Caveats for the use of citation indicators in research and journal evaluations. Journal of the American Society for Information Science and Technology, 59(2), 278–287.

    Article  Google Scholar 

  • Leydesdorff, L. (2012). Alternatives to the journal impact factor: 13 and the top-10% (or top-25%) of the most-highly cited papers. Scientometrics, 92(2), 355–365.

    Article  Google Scholar 

  • Leydesdorff, L. (2013). An evaluation of impacts in “Nanoscience and nanotechnology”: Steps towards standards for citation analysis. Scientometrics, 94(1), 35–55.

    Article  Google Scholar 

  • Lindner, M. D., Torralba, K. D., & Khan, N. A. (2018). Scientific productivity: An exploratory study of metrics and incentives. PLoS ONE, 13(4), e0195321.

    Article  Google Scholar 

  • Malesios, C. (2016). Measuring the robustness of the journal h-index with respect to publication and citation values: A Bayesian sensitivity analysis. Journal of Informetrics, 10(3), 719–731.

    Article  Google Scholar 

  • McGarty, C. (2000). The citation impact factor in social psychology: A bad statistic that encourages bad science? Current Research in Social Psychology, 5(1), 1–16.

    Google Scholar 

  • McVeigh, M. E., & Mann, S. J. (2009). The journal impact factor denominator: Defining citable (counted) items. JAMA-Journal of the American Medical Association, 302(10), 1107–1109.

    Article  Google Scholar 

  • Mimouni, M., Ratmansky, M., Sacher, Y., Aharoni, S., & Mimouni-Bloch, A. (2016). Self-citation rate and impact factor in pediatrics. Scientometrics, 108(3), 1455–1460.

    Article  Google Scholar 

  • Mingers, J., Macri, F., & Petrovici, D. (2012). Using the h-index to measure the quality of journals in the field of business and management. Information Processing and Management, 48(2), 234–241.

    Article  Google Scholar 

  • Mirnezami, S. R., Beaudry, C., & Lariviere, V. (2016). What determines researchers’ scientific impact? A case study of Quebec researchers. Science and Publicpolicy, 43(2), 262–274.

    Google Scholar 

  • Olden, J. D. (2007). How do ecological journals stack-up? Ranking of scientific quality according to the h-index. Ecoscience, 14(3), 370–376.

    Article  Google Scholar 

  • Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research, 33(1), 1–7.

    Article  Google Scholar 

  • Packalen, M., & Bhattacharya, J. (2017). Neophilia ranking of scientific journals. Scientometrics, 110(1), 43–64.

    Article  Google Scholar 

  • Polit, D. F., & Northam, S. (2011). Impact factors in nursing journals. Nursing Outlook, 59(1), 18–28.

    Article  Google Scholar 

  • Prathap, G., Mini, S., & Nishy, P. (2016). Does high impact factor successfully predict future citations? An analysis using Peirce’s measure. Scientometrics, 108(3), 1043–1047.

    Google Scholar 

  • Prozesky, H., & Boshoff, N. (2012). Bibilometrics as a tool for measuring gender-specific research performance: An example from South African invasion ecology. Scientometrics, 90(2), 383–406.

    Article  Google Scholar 

  • Racz, A., & Markovic, S. (2018). “Worth(less) papers”-are journal impact factor and number of citations suitable indicators to evaluate quality of scientists? Nova Prisutnost, 16(2), 369–389.

    Article  Google Scholar 

  • Romero-Torres, M., Acosta-Moreno, L. A., & Tejada-Gomez, M. A. (2013). Use of the h index to rank scientific Latin American journals: Colombia a case study. Revista Espanola de Documentacion Cientifica, 36(1), e003.

    Article  Google Scholar 

  • Rousseau, R. (2016). Positive correlation between journal production and journal impact factors. Journal of Informetrics, 10(2), 567–568.

    Article  Google Scholar 

  • Saad, G. (2006). Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics, 69(1), 117–120.

    Article  Google Scholar 

  • Sangwal, K. (2013). Some citation-related characteristics of scientific journals published in individual countries. Scientometrics, 97(3), 719–741.

    Article  Google Scholar 

  • Schubert, A., & Glänzel, W. (2007). A systematic analysis of Hirsch-type indices for journals. Journal of Informetrics, 1(3), 179–184.

    Article  Google Scholar 

  • Sebire, N. J. (2008). H-index and impact factors: Assessing the clinical impact of researchers and specialist journals. Ultrasound in Obstetrics and Gynecology, 32(7), 843–845.

    Article  Google Scholar 

  • Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 498–502.

    Article  Google Scholar 

  • Shen, Z. S., Yang, L. Y., & Wu, J. S. (2018). Lognormal distribution of citation counts is the reason for the relation between Impact Factors and Citation Success Index. Journal of Informetrics, 12(1), 153–157.

    Article  Google Scholar 

  • Smith, K. M., Crookes, E., & Crookes, P. A. (2013). Measuring research ‘impact’ for academic promotion: Issues from the literature. Journal of Higher Education Policy and Management, 35(4), 410–420.

    Article  Google Scholar 

  • Solarino, S. (2012). Impact factor, citation index, h-index: Are researchers still free to choose where and how to publish their results? Annals of Geophysics, 55(3), 473–477.

    Google Scholar 

  • Sternberg, R. J. (2018). Evaluating merit among scientists. Journal of Applied Research in Memory and Cognition, 7(2), 209–216.

    Article  MathSciNet  Google Scholar 

  • Tahamtan, I., Afshar, A. S., & Ahamdzadeh, K. (2016). Factors affecting number of citations: A comprehensive review of the literature. Scientometrics, 107(3), 1195–1225.

    Article  Google Scholar 

  • Valderrama, P., Escabias, M., Jimenez-Contreras, E., Rodriguez-Archilla, A., & Valderrama, M. J. (2018). Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: Application to the field of Dentistry. Scientometrics, 115(2), 1087–1095.

    Article  Google Scholar 

  • Van Raan, A. F. J. (2012). Properties of journal impact in relation to bibliometric research group performance indicators. Scientometrics, 92(2), 457–469.

    Article  Google Scholar 

  • Vanclay, J. K. (2008). Ranking forestry journals using the h-index. Journal of Informetrics, 2(4), 326–334.

    Article  Google Scholar 

  • Vinkler, P. (2001). An attempt for defining some basic categories of scientometrics and classifying the indicators of evaluative scientometrics. Scientometrics, 50(3), 539–544.

    Article  Google Scholar 

  • Vinkler, P. (2002). The institutionalization of scientific information. The scientometric model (ISI-S MODEL). Library Trends, 50(3), 553–569.

    Google Scholar 

  • Vinkler, P. (2004). Characterization of the impact of sets of scientific papers: The garfield (impact) factor. Journal of the American Society for Information Science and Technology, 55(5), 431–435.

    Article  Google Scholar 

  • Vinkler, P. (2009a). Introducing the Current Contribution Index for characterizing the recent, relevant impact of journals. Scientometrics, 79(2), 409–420.

    Article  Google Scholar 

  • Vinkler, P. (2009b). The π-index: A new indicator for assessing scientific impact. Journal of Information Science, 35(5), 602–612.

    Article  Google Scholar 

  • Vinkler, P. (2010). The π v-index: A new indicator to characterize the impact of journals. Scientometrics, 82(3), 461–475.

    Article  Google Scholar 

  • Vinkler, P. (2012). The Garfield impact factor, one of the fundamental indicators in scientometrics. Scientometrics, 92(2), 471–483.

    Article  Google Scholar 

  • Vinkler, P. (2017). The size and impact of the elite set of publications in scientometric assessments. Scientometrics, 110(1), 163–177.

    Article  Google Scholar 

  • Vinluan, L. R. (2012). Research productivity in education and psychology in the Philippines and comparison with ASEAN countries. Scientometrics, 91(1), 277–294.

    Article  Google Scholar 

  • Xia, J. F., & Smith, M. P. (2018). Alternative journal impact factors in open access publishing. Learned Publishing, 31(4), 403–411.

    Article  Google Scholar 

  • Yan, E. J., Wu, C. J., & Song, M. (2018). The funding factor: A cross-disciplinary examination of the association between research funding and citation impact. Scientometrics, 115(1), 369–384.

    Article  Google Scholar 

  • Yang, D. H., Li, X., Sun, X. X., & Wan, J. (2016). Detecting impact factor manipulation with data mining techniques. Scientometrics, 109(3), 1989–2005.

    Article  Google Scholar 

  • Yeung, A. W. K. (2019). Higher impact factor of neuroimaging journals is associated with larger number of articles published and smaller percentage of uncited articles. Scientometrics, 12, 523.

    Google Scholar 

  • Yu, G., & Li, Y. J. (2007). Parameter identification of the observed citation distribution. Scientometrics, 71(2), 339–348.

    Article  Google Scholar 

  • Yu, G., & Li, Y. J. (2010). Identification of referencing and citation processes of scientific journals based on the citation distribution model. Scientometrics, 82(2), 249–261.

    Article  Google Scholar 

  • Yu, G., Wang, X. H., & Yu, D. R. (2005). The influence of publication delays on impact factors. Scientometrics, 64(2), 235–246.

    Article  Google Scholar 

  • Yuen, J. (2018). Comparision of impact factor, eigenfactor metrics, and scimago journal rank indicator and h-index for neurosurgical and spinal surgical journals. World Neurosurgery, 119, E328–E337.

    Article  Google Scholar 

  • Yuret, T. (2018). Author-weighted impact factor and reference return ratio: Can we attain more equality among fields? Scientometrics, 116(3), 2097–2111.

    Article  Google Scholar 

  • Zhang, L., Thijs, B., & Glänzel, W. (2011). The diffusion of H-related literature. Journal of Informetrics, 5(4), 583–593.

    Article  Google Scholar 

  • Zou, Y. W., & Laubichler, M. D. (2017). Measuring the contributions of Chinese scholars to the research field of systems biology from 2005 to 2013. Scientometrics, 110(3), 1615–1631.

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant No. 71473034), the Heilongjiang Provincial Natural Science Foundation of China (Grant No. LH2019G001), and the financial assistance from Postdoctoral Scientific Research Developmental Fund of Heilongjiang Province (Grant No. LBH-Q16003).

Author information

Authors and Affiliations

  1. College of Information and Computer Engineering, Northeast Forestry University, Harbin, 150040, People’s Republic of China

    Mingyang Wang, Shijia Jiao & Guangsheng Chen

  2. Industrial and Systems Engineering, National University of Singapore, Singapore, 117576, Singapore

    Kah-Hin Chai

Authors
  1. Mingyang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shijia Jiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kah-Hin Chai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangsheng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guangsheng Chen.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 773 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, M., Jiao, S., Chai, KH. et al. Building journal’s long-term impact: using indicators detected from the sustained active articles. Scientometrics 121, 261–283 (2019). https://doi.org/10.1007/s11192-019-03196-8

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-019-03196-8

Keywords

Search

Navigation