On the use of journal classification in social sciences and humanities: evidence from an Italian database

Abstract

In social sciences and humanities, a two-tier journal classification is currently used in Italy in the context of the National Habilitation programme; peer review is also available for a large number of articles published in the same journals, in the framework of the last national evaluation exercise (VQR 2011–2014). We take advantage of these combined two rich datasets in order to check if journals classified as top class by scientific experts show higher impact and if articles published in those journals receive higher marks in peer-reviewed evaluation exercises with respect to other journals. Our main result is that ANVUR classification offers on average a reliable proxy for the quality of journals, as measured by journal indicators and by the assessments of independent experts evaluating individual articles published in those journals. While peer review is still to be considered as the main method for evaluation in Humanities and Social sciences, our analysis supports the view that journal classification can be a useful tool to support peer review even in SSH.

This is a preview of subscription content, log in to check access.

Fig. 1

Source: Authors elaboration on ANVUR data

Fig. 2

Source: Authors elaboration on ANVUR data

Fig. 3

Source: Authors elaboration on ANVUR data

Notes

  1. 1.

    For a more complete discussion on the various methods of journal rating and ranking, see Ferrara and Bonaccorsi (2016).

  2. 2.

    In Italy, each professor is bound to a specific academic discipline, which corresponds to a scientific field of study. Academic disciplines are organized in main groups, defined as area of research. In SSH there are 78 academic disciplines grouped in 6 areas of research; the list is available at http://attiministeriali.miur.it/media/265754/allegato_a.pdf.

  3. 3.

    See also ANVUR (2017b), Tables 2 and 6.

  4. 4.

    VQR and the National Scientific Habilitation are two distinct procedures that are not bind by the same rules. So it is possible that an article submitted for evaluation in VQR is not published in the ANVUR list of scientific journals: it may be the case, for instance, for articles published in recently-founded journals that are not yet classified by ANVUR, or for scientific articles which are indeed published in journals that are considered as catering for a professional rather than an academic public, and as such are not admitted in the ANVUR list.

  5. 5.

    We prefer to use the SJR indicator with respect to possible alternatives like the impact factor or the h index because SJR allows to consider the importance or prestige of the journals from which citations come from.

  6. 6.

    As stated above, the list of scientific journals also includes A-Class journals; in the remainder of the paper, we compare A-Class journals only with the scientific journals that are not also comprised in the A-Class list.

  7. 7.

    An ideal logistic model has 100% sensitivity (or the proportion of truly positive observations which is classified as such by the model) and 100% specificity (or the proportion of truly negative observations which is classified as such by the model). ROC curve is a plot of the values of sensitivity against one minus specificity. A good discrimination is considered when the value under the curve is almost 0.80.

References

  1. Ancaiani, A., et al. (2015). Evaluating scientific research in Italy: The 2004–10 research evaluation exercise. Research Evaluation,24(3), 242–255.

    Article  Google Scholar 

  2. Anfossi, A., Ciolfi, A., Costa, F., Parisi, G., & Benedetto, S. (2016). Large-scale assessment of research outputs through a weighted combination of bibliometric indicators. Scientometrics,107(2), 671–683.

    Article  Google Scholar 

  3. ANVUR. (2017a). Regolamento per la classificazione delle riviste nelle aree non bibliometriche. http://www.anvur.it/wp-content/uploads/2017/10/RegolamClassificazRiviste.pdf.

  4. ANVUR. (2017b). Evaluation of research quality 201114, ANVUR final report. http://www.anvur.it/wp-content/uploads/2017/06/VQR2011-2014_Final%20Report.pdf.

  5. Bonaccorsi, A., Ferrara, A., & Malgarini, M. (2018). Journal ratings as predictors of article quality in arts, humanities and social sciences: An analysis based on the Italian research evaluation exercise. In A. Bonaccorsi (Ed.), The evaluation of research in social sciences and humanities. Berlin: Springer.

    Google Scholar 

  6. Cicero, T., & Malgarini, M. (2019). Robustness of journal classifications in SSH: An empirical analysis from Italy. In 17th International conference on scientometrics and informetrics (ISSI 2019). 2–5 September 2019, Rome, Italy.

  7. Ferrara, A., & Bonaccorsi, A. (2016). How robust is journal ratings in humanities and social sciences? Evidence from a large-scale, multi-method exercise. Research Evaluation,25(3), 279–291.

    Article  Google Scholar 

  8. Franssen, T., & Wouters, P. (2019). Science and its significant other: Representing the humanities in bibliometric scholarship. Journal of the Association for Information Science and Technology,70, 1124–1137. https://doi.org/10.1002/asi.24206.

    Article  Google Scholar 

  9. Hix, S. (2004). A global ranking of political science departments. Political Studies,2, 293–313.

    Article  Google Scholar 

  10. Hogler, R., & Gross, M. A. (2009). Journal rankings and academic research: Two discourses about the quality of faculty work. Management Communication Quarterly,23(1), 107–126.

    Article  Google Scholar 

  11. Kalaitzidakis, P., Mamuneas, T. P., & Stengos, T. (1999). European economics: An analysis based on publications in the core journals. European Economic Review,43, 1150–1168.

    Article  Google Scholar 

  12. Lavik, G. A. V., & Sivertsen, G. (2017). ERIH PLUS making the SSH visible, searchable and available. Procedia Computer Science,106, 61–65.

    Article  Google Scholar 

  13. Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in Peer Review. Advances in Information Science,64(1), 2–17.

    Google Scholar 

  14. Moed, H. F. (2008). Research assessment in social sciences and humanities. In ECOOM Colloquium Antwerp. December 12–13.

  15. Pontille, D., & Torny, D. (2010). The controversial policies of journal ratings: Evaluating social sciences and humanities. Research Evaluation,19(5), 347–360.

    Article  Google Scholar 

  16. Sivertsen G. (2014). Scholarly publication patterns in the social sciences and humanities and their coverage in Scopus and Web of Science. In Proceedings of the science and technology indicators conference (pp. 598–604).

  17. Walker, J., Fenton, E., Salandra, R., & Salter, A. (2018). What influences business academics’ use of the Association of Business Schools’ (ABS) list? Evidence from a survey of UK academics. British Journal of Management. https://doi.org/10.1111/1467-8551.12294.

    Article  Google Scholar 

  18. Williams, R. (2012). Using the margins command to estimate and interpret adjusted predictions and marginal effects. The Stata Journal,12(2), 308–331.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Marco Malgarini.

Appendices

Appendix 1: Statistical test (test t) between average SJR 2018 of A-Class journals and no A-Class journals in the 78 academic disciplines of interest

SC NO A-Class A-Class test t p value
0.580 0.919 2.844 0.002 0.580
0.580 0.919 2.844 0.002 0.580
0.580 0.919 2.844 0.002 0.580
0.580 0.919 2.844 0.002 0.580
0.580 0.919 2.844 0.002 0.580
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.472 0.534 0.825 0.205 0.472
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.491 0.494 0.045 0.482 0.491
0.491 0.494 0.045 0.482 0.491
0.491 0.494 0.045 0.482 0.491
0.492 0.493 0.015 0.494 0.492
0.491 0.494 0.045 0.482 0.491
0.492 0.493 0.015 0.494 0.492
0.492 0.493 0.015 0.494 0.492
0.821 0.700 − 0.647 0.741 0.821
0.830 0.572 − 1.613 0.946 0.830
0.834 0.560 − 1.885 0.970 0.834
0.818 0.762 − 0.267 0.605 0.818
0.804 1.355 2.232 0.013 0.804
0.771 1.617 5.335 0.000 0.771
0.763 1.201 4.036 0.000 0.763
0.763 1.192 3.961 0.000 0.763
0.763 1.199 4.003 0.000 0.763
0.763 1.192 3.958 0.000 0.763
0.763 1.199 4.003 0.000 0.763
0.780 1.171 3.192 0.001 0.780
0.759 1.284 4.581 0.000 0.759
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.424 0.836 2.716 0.003 0.424
0.889 2.922 21.078 0.000 0.889
0.889 2.922 21.078 0.000 0.889
0.889 2.922 21.078 0.000 0.889
0.889 2.922 21.078 0.000 0.889
0.889 2.922 21.078 0.000 0.889
0.868 2.820 21.202 0.000 0.868
0.868 2.820 21.202 0.000 0.868
0.868 2.820 21.202 0.000 0.868
0.868 2.820 21.202 0.000 0.868
0.868 2.820 21.202 0.000 0.868
0.875 2.604 19.279 0.000 0.875
0.898 2.964 20.939 0.000 0.898
0.898 2.964 20.939 0.000 0.898
0.898 2.964 20.939 0.000 0.898
0.908 3.112 21.481 0.000 0.908
0.819 1.575 2.596 0.005 0.819
0.624 1.458 9.396 0.000 0.624
0.818 2.337 3.794 0.000 0.818
0.838 0.766 − 0.476 0.683 0.838
0.727 1.267 5.439 0.000 0.727
0.728 1.251 5.327 0.000 0.728
0.727 1.267 5.447 0.000 0.727
0.728 1.267 5.430 0.000 0.728

Appendix 2: Statistical test (test t) between VQR average scores of A-Class journals and no A-Class journals in the 78 academic disciplines of interest

SC NO A-Class A-Class test t p value
08/C1 0.298 0.618 3.742 0.000
08/D1 0.313 0.338 0.214 0.416
08/E1 0.109 0.451 2.588 0.007
08/E2 0.515 0.637 1.161 0.125
08/F1 0.248 0.525 2.782 0.003
10/A1 0.524 0.724 2.541 0.006
10/B1 0.395 0.614 2.170 0.017
10/C1 0.682 0.633 − 0.497 0.690
10/D1 0.667 0.646 − 0.166 0.566
10/D2 0.417 0.755 3.040 0.002
10/D3 0.423 0.766 2.686 0.005
10/D4 0.654 0.821 1.347 0.092
10/E1 0.667 0.694 0.194 0.423
10/F1 0.609 0.601 − 0.072 0.528
10/F2 0.400 0.477 0.354 0.363
10/F3 0.667 0.762 0.730 0.234
10/F4 0.000 0.575 2.346 0.014
10/G1 0.600 0.778 2.119 0.018
10/H1 0.594 0.633 0.363 0.359
10/I1 0.324 0.541 2.166 0.017
10/L1 0.586 0.757 2.593 0.005
10/M1 0.401 0.624 2.089 0.020
10/M2 0.643 0.747 0.720 0.238
10/N1 0.675 0.702 0.239 0.406
10/N3 0.500 0.935 4.211 0.000
11/A1 0.273 0.645 2.885 0.003
11/A2 0.572 0.718 1.277 0.103
11/A3 0.243 0.629 4.715 0.000
11/A4 0.821 0.774 − 0.542 0.705
11/A5 0.571 0.644 0.586 0.280
11/B1 0.514 0.737 1.928 0.029
11/C1 0.732 0.769 0.317 0.376
11/C2 0.353 0.811 5.235 0.000
11/C3 0.414 0.607 1.535 0.065
11/C4 0.605 0.663 0.558 0.289
11/C5 0.596 0.732 1.451 0.075
11/D1 0.545 0.846 3.475 0.000
11/D2 0.588 0.654 0.744 0.229
12/A1 0.302 0.459 1.937 0.028
12/B1 0.329 0.427 0.854 0.198
12/B2 0.107 0.531 3.516 0.001
12/C1 0.452 0.513 0.480 0.317
12/C2 0.250 0.608 1.409 0.090
12/D1 0.190 0.508 4.490 0.000
12/D2 0.120 0.484 2.476 0.011
12/E1 0.461 0.639 1.456 0.075
12/E2 0.448 0.566 1.201 0.117
12/E3 0.375 0.490 0.838 0.203
12/E4 0.444 0.458 0.072 0.471
12/F1 0.167 0.433 1.929 0.031
12/G1 0.220 0.467 2.068 0.022
12/G2 0.417 0.678 1.560 0.065
12/H1 0.335 0.669 2.607 0.007
12/H2 0.692 0.893 1.585 0.062
12/H3 0.240 0.582 3.404 0.001
13/A1 0.306 0.938 16.940 0.000
13/A2 0.333 0.955 13.556 0.000
13/A3 0.278 0.965 11.581 0.000
13/A4 0.386 0.947 8.088 0.000
13/A5 0.533 0.991 5.548 0.000
13/B1 0.225 0.986 16.351 0.000
13/B2 0.295 0.971 15.535 0.000
13/B3 0.325 1.000 10.902 0.000
13/B4 0.291 1.000 8.962 0.000
13/B5 0.500 1.000 2.192 0.017
13/C1 0.225 0.917 6.546 0.000
13/D1 0.453 0.930 8.213 0.000
13/D2 0.324 0.907 8.069 0.000
13/D3 0.218 0.914 8.673 0.000
13/D4 0.408 0.975 9.906 0.000
14/A1 0.474 0.893 2.871 0.003
14/A2 0.414 0.703 3.234 0.001
14/B1 0.535 0.821 1.758 0.043
14/B2 0.594 0.614 0.149 0.441
14/C1 0.303 0.645 4.706 0.000
14/C2 0.403 0.438 0.408 0.342
14/C3 0.333 0.566 1.575 0.062
14/D1 0.367 0.617 2.578 0.006

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cicero, T., Malgarini, M. On the use of journal classification in social sciences and humanities: evidence from an Italian database. Scientometrics (2020). https://doi.org/10.1007/s11192-020-03581-8

Download citation

Keywords

  • Journal classification
  • Social sciences and humanities
  • Peer review
  • Italy

Mathematics Subject Classification

  • 62
  • 68
  • 91

JEL Classification

  • C12
  • I23