Properties of journal impact in relation to bibliometric research group performance indicators
In this paper we present a compilation of journal impact properties in relation to other bibliometric indicators as found in our earlier studies together with new results. We argue that journal impact, even calculated in a sufficiently advanced way, becomes important in evaluation practices based on bibliometric analysis only at an aggregate level. In the relation between average journal impact and actual citation impact of groups, the influence of research performance is substantial. Top-performance as well as lower performance groups publish in more or less the same range of journal impact values, but top-performance groups are, on average, more successful in the entire range of journal impact. We find that for the high field citation-density groups a larger size implies a lower average journal impact. For groups in the low field citation-density regions however a larger size implies a considerably higher average journal impact. Finally, we found that top-performance groups have relatively less self-citations than the lower performance groups and this fraction is decreasing with journal impact.
KeywordsImpact factor Journal impact Bibliometric analysis Research group performance
The discussion on the meaning of journal impact for both the assessment of the standing of journals as well as its use in evaluation practices regularly flares up. A striking example is the discussion in Nature (initiated by the paper of Lawrence 2003) in which researchers, referring to the work of Seglen (1992, 1994), fulminate against the supposed dominant role of journal status and journal impact factors in the present-day life of a scientist. The most important finding of Seglen was the poor correlation between the impact of publications and journal impact at the level of individual publications. Seglen therefore concluded that the use of journal impact as an indicator for research performance evaluation is inappropriate. We stress that even when peer-review based assessment of journal status would completely replace citation-based (bibliometric) measures of journal impact, the above discussed poor correlation would remain. This is due to the skew distribution of citations over individual publications in any journal and even in any entity, regardless of whatever peer-review based journal standing indicator.
However, Seglen also found that aggregating publications in classes of journal impact yielded a high correlation between the average number of citations per publication and journal impact. We found (van Raan 2006a) that aggregating publications in a more institutional way, namely the publications of research groups, also shows a significant correlation between the average number of citations per publication of research groups, and the average journal impact of these groups.
In Seglen’s work the calculation of journal impact was based on the ISI1 journal impact factor. This measure has important disadvantages for bibliometric studies (Moed and Van Leeuwen 1995, 1996; Vanclay 2011). In our work we use the more sophisticated journal impact indicators developed with a long standing experience by our institute (CWTS). We discussed in recent papers statistical properties of the relations between journal impact and other bibliometric indicators (van Raan 2006a, b, 2008a, b). Together with further results we present in this paper a compilation of the findings of the above papers in order to have a concise overview of the most relevant properties of journal impact. In this paper we argue that journal impact, even calculated in a sufficiently advanced way, becomes important in evaluation practices based on bibliometric analysis only at an aggregate level.
CWTS standard bibliometric indicators
Number of publications (P) in WoS-covered journals of a specific entity in a given time period
Number of citations without self-citations (C) received by P during the entire period
Average number of citations per publication without self-citations (CPP)
Percentage of publications not cited (in the given time period), Pnc
Average journal impact for each journal used by the specific entity (JCS, journal citation score), without self-citations; as almost always a set of journals is used, we calculate the weighted average JCSm; for the calculation of JCSm the same publication and citation counting procedure, time windows, and article types are used as in the case of CPP
Taking all journals of a field, we calculate the average field-based impact as an international reference level for the specific entity (FCS, field citation score), without self-citations. In the case of more than one field (as almost always) we use the weighted average FCSm; for the calculation of FCSm the same publication and citation counting procedure, time windows, and article types are used as in the case of CPP
Comparison of the actually received impact of the specific entity with the world-wide average based on JCSm as a standard, without self-citations, indicator CPP/JCSm
Comparison of the actually received impact of the specific entity with the world-wide average based on FCSm as a standard, without self-citations, indicator CPP/FCSm
Ratio JCSm/FCSm indicating whether the impact of a journal is above (ratio >1) or below (ratio <1) the field average
Percentage of self-citations sc
The structure of our paper is as follows. We present in the next section results on four main topics: the distribution of journal impact over individual publications (the lowest aggregation level) and over research groups (the next higher aggregation level); the relation between journal impact and actual citation impact of groups and the influence of research performance; the group-size dependence of journal impact in relation to field-specific citation density; and the relation between journal impact and self-citation. In the last section we summarize our main conclusions.
Distribution of journal impact for individual publications and for groups
JCS values are calculated with the same citation window as used for the citation counting of the publication; this window is at least 4 years (for the ISI impact factor the window is only 2 years);
If the publication is a review, a letter of a normal paper, the JCS values are calculated for only review, letters or normal papers in the journal, thus our JCS takes into account document type.
We clearly see that the distribution function changes from skew to almost normal if group average journal impact factors are field-normalized.
Journal impact and actual citation impact of groups
Thus, by the ‘phase transition’ to a higher aggregation level a significant relation is found between average impact of the publications of a group (the actual impact) and the average impact of journals of a group.
Thus, the important observation is that top-performance groups are, on average, more successful in the entire range of journal impact. In other words, they perform better in the lower-impact journals as well as in the higher-impact journals. This nicely confirms Seglen’s findings as discussed above. However, we also observe that the top-20 % groups have a slight preference for the higher-impact journals. Another interesting finding is the difference in power law behaviour between the top-20 % and the bottom-20 % groups: CPP increases with JCSm for the top-20 % groups somewhat less stronger (exponent 0.90) than for the bottom-20 % groups (exponent 1.02).4
The coefficients of the power law equations provide a quantitative measure of the extent to which the top-20 % groups have a higher average number of citations per publication (CPP) for the same journal impact (JCS) values as compared to lower performance (bottom-20 %) groups. The ratio of the coefficients is 2.95. Thus, the top-groups perform in terms of citations per publications (CPP), a factor of about 3 better than the lower performance groups in the same journals. 5 Also this finding is in agreement with Seglen’s work, he finds a factor between 1.5 and 3.5.
For the bottom-20 % groups we observe that the better performing groups tend to publish in the higher-impact journals, but the significance of this relation is low.
Size dependence of journal impact in relation to field citation density
Here we make a distinction between research groups active in fields that belong to the top-25 % of the field citation density (FCSm)—in most cases these are the more basic research oriented fields and those groups active in fields with the lowest 25 % of the field citation density—mostly the more applied research oriented fields. We stress that this division does not imply any difference in performance, as explained earlier. For both subsets we calculated the JCSm values as a function of size. We find that for the high field citation-density groups a larger size implies a lower average JCSm value. This implies for research groups operating in high field citation-density regions that a larger number of publications will lead to a somewhat lower average journal citation impact. Thus, ‘expanding in size’ may take place within the same field citation-density region, but it will generally include publications in journals with a lower impact.
In contrast, for groups in the low field citation-density regions a larger size implies a considerably higher average JCSm value. Thus, for groups operating in low field citation-density regions a larger number of publications can be seen as an ‘expansion’ to journals with a higher impact. In our previous work (van Raan 2008a) we presented a ‘landscape’ model to explain these observations and their quantitative properties.
Journal impact and self-citation
We conclude that the fraction of self-citations tend to decrease with journal impact and with performance. The significance however is not very high.
Summary of the main conclusions
We argued that the ISI journal impact factor is unsuitable for the use in bibliometric studies in general and particularly for evaluation studies. The journal impact indicator developed by the CWTS has considerably better properties. We showed that there is a remarkable ‘phase transition’ in the meaning of journal impact when going from the lowest aggregation level—individual publications—to a higher aggregation level—research groups, about two orders of magnitude larger. For individual publications even a more sophisticated journal impact is still an inappropriate measure predictor of the actual impact of a publication, whereas for research group the average journal impact correlates well with the actual impact of a group.
The distribution of journal impact over individual publications follows an exponential function with high significance and this is a nice example that not all relations of bibliometric measures follow power laws. In the relation between average journal impact and actual citation impact of groups, the influence of research performance is substantial. Top-performance as well as lower performance groups publish in more or less the same range of journal impact values, but top-performance groups are, on average, more successful in the entire range of journal impact. In other words, top-groups perform better in the lower impact as well as in the higher-impact journals: in terms of citations per publications they perform a factor 3 better than the lower performance groups in the same journals. We also observe that the top-20 % groups have a slight preference for the higher-impact journals and they do not publish in the journals with the lowest impact. The average number of citations increases with average journal impact for the top-20 % groups somewhat less strong than for the bottom-20 % groups.
We find that for the high field citation-density groups a larger size implies a lower average journal impact: for research groups operating in high field citation-density regions a larger number of publications will lead to a somewhat lower average journal citation impact. For groups in the low field citation-density regions however a larger size implies a considerably higher average journal impact. Finally, we found that top-performance groups have relatively less self-citations than the lower performance groups and this fraction is decreasing with journal impact.
As long as they exist, journals do and will play an important role in the assessment of the quality of research, regardless whether the assessment is based on peer review only, on bibliometric analysis, or a combination of both. Therefore it is of crucial importance to know the properties of journal impact in relation to other bibliometric indicators. Hopefully, the results reported in this paper will stimulate the careful use of journal impact measures.
The former Institute for Scientific Information (ISI) in Philadelphia was the producer of the Science Citation Index and related indexes. Now Thomson Scientific is the producer of the citation indexes combined in the Web of Science (WoS) data system.
β is a constant factor which can be determined empirically from the plot (β = 4,497.61).
By randomly removing 10 groups in the set of research groups and recalculating the correlation functions, we estimate that the uncertainty in the power law exponents is about ±0.04.
For top-10% and bottom-10% the ratio is 3.45, for top-50% and bottom-50% we find 1.87.
This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.
- van Raan, A. F. J. (2004). Measuring science. Capita selecta of current main issues. In H. F. Moed, W. Glänzel, & U. Schmoch (Eds.), Handbook of quantitative science and technology research (pp. 19–50). Dordrecht: Kluwer Academic Publishers.Google Scholar
- Vanclay, J. K. (2011). Impact factor: outdated artefact or stepping-stone to journal certification? Scientometrics. doi: 10.1007/s11192-011-0561-0.