Skip to main content
SpringerLink
Log in

On the use of journal classification in social sciences and humanities: evidence from an Italian database

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

In social sciences and humanities, a two-tier journal classification is currently used in Italy in the context of the National Habilitation programme; peer review is also available for a large number of articles published in the same journals, in the framework of the last national evaluation exercise (VQR 2011–2014). We take advantage of these combined two rich datasets in order to check if journals classified as top class by scientific experts show higher impact and if articles published in those journals receive higher marks in peer-reviewed evaluation exercises with respect to other journals. Our main result is that ANVUR classification offers on average a reliable proxy for the quality of journals, as measured by journal indicators and by the assessments of independent experts evaluating individual articles published in those journals. While peer review is still to be considered as the main method for evaluation in Humanities and Social sciences, our analysis supports the view that journal classification can be a useful tool to support peer review even in SSH.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Source: Authors elaboration on ANVUR data

Fig. 2

Source: Authors elaboration on ANVUR data

Fig. 3

Source: Authors elaboration on ANVUR data

Similar content being viewed by others

Notes

  1. For a more complete discussion on the various methods of journal rating and ranking, see Ferrara and Bonaccorsi (2016).

  2. In Italy, each professor is bound to a specific academic discipline, which corresponds to a scientific field of study. Academic disciplines are organized in main groups, defined as area of research. In SSH there are 78 academic disciplines grouped in 6 areas of research; the list is available at http://attiministeriali.miur.it/media/265754/allegato_a.pdf.

  3. See also ANVUR (2017b), Tables 2 and 6.

  4. VQR and the National Scientific Habilitation are two distinct procedures that are not bind by the same rules. So it is possible that an article submitted for evaluation in VQR is not published in the ANVUR list of scientific journals: it may be the case, for instance, for articles published in recently-founded journals that are not yet classified by ANVUR, or for scientific articles which are indeed published in journals that are considered as catering for a professional rather than an academic public, and as such are not admitted in the ANVUR list.

  5. We prefer to use the SJR indicator with respect to possible alternatives like the impact factor or the h index because SJR allows to consider the importance or prestige of the journals from which citations come from.

  6. As stated above, the list of scientific journals also includes A-Class journals; in the remainder of the paper, we compare A-Class journals only with the scientific journals that are not also comprised in the A-Class list.

  7. An ideal logistic model has 100% sensitivity (or the proportion of truly positive observations which is classified as such by the model) and 100% specificity (or the proportion of truly negative observations which is classified as such by the model). ROC curve is a plot of the values of sensitivity against one minus specificity. A good discrimination is considered when the value under the curve is almost 0.80.

References

  • Ancaiani, A., et al. (2015). Evaluating scientific research in Italy: The 2004–10 research evaluation exercise. Research Evaluation, 24(3), 242–255.

    Article  Google Scholar 

  • Anfossi, A., Ciolfi, A., Costa, F., Parisi, G., & Benedetto, S. (2016). Large-scale assessment of research outputs through a weighted combination of bibliometric indicators. Scientometrics, 107(2), 671–683.

    Article  Google Scholar 

  • ANVUR. (2017a). Regolamento per la classificazione delle riviste nelle aree non bibliometriche. http://www.anvur.it/wp-content/uploads/2017/10/RegolamClassificazRiviste.pdf.

  • ANVUR. (2017b). Evaluation of research quality 2011–14, ANVUR final report. http://www.anvur.it/wp-content/uploads/2017/06/VQR2011-2014_Final%20Report.pdf.

  • Bonaccorsi, A., Ferrara, A., & Malgarini, M. (2018). Journal ratings as predictors of article quality in arts, humanities and social sciences: An analysis based on the Italian research evaluation exercise. In A. Bonaccorsi (Ed.), The evaluation of research in social sciences and humanities. Berlin: Springer.

    Chapter  Google Scholar 

  • Cicero, T., & Malgarini, M. (2019). Robustness of journal classifications in SSH: An empirical analysis from Italy. In 17th International conference on scientometrics and informetrics (ISSI 2019). 2–5 September 2019, Rome, Italy.

  • Ferrara, A., & Bonaccorsi, A. (2016). How robust is journal ratings in humanities and social sciences? Evidence from a large-scale, multi-method exercise. Research Evaluation, 25(3), 279–291.

    Article  Google Scholar 

  • Franssen, T., & Wouters, P. (2019). Science and its significant other: Representing the humanities in bibliometric scholarship. Journal of the Association for Information Science and Technology, 70, 1124–1137. https://doi.org/10.1002/asi.24206.

    Article  Google Scholar 

  • Hix, S. (2004). A global ranking of political science departments. Political Studies, 2, 293–313.

    Article  Google Scholar 

  • Hogler, R., & Gross, M. A. (2009). Journal rankings and academic research: Two discourses about the quality of faculty work. Management Communication Quarterly, 23(1), 107–126.

    Article  Google Scholar 

  • Kalaitzidakis, P., Mamuneas, T. P., & Stengos, T. (1999). European economics: An analysis based on publications in the core journals. European Economic Review, 43, 1150–1168.

    Article  Google Scholar 

  • Lavik, G. A. V., & Sivertsen, G. (2017). ERIH PLUS making the SSH visible, searchable and available. Procedia Computer Science, 106, 61–65.

    Article  Google Scholar 

  • Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in Peer Review. Advances in Information Science, 64(1), 2–17.

    Google Scholar 

  • Moed, H. F. (2008). Research assessment in social sciences and humanities. In ECOOM Colloquium Antwerp. December 12–13.

  • Pontille, D., & Torny, D. (2010). The controversial policies of journal ratings: Evaluating social sciences and humanities. Research Evaluation, 19(5), 347–360.

    Article  Google Scholar 

  • Sivertsen G. (2014). Scholarly publication patterns in the social sciences and humanities and their coverage in Scopus and Web of Science. In Proceedings of the science and technology indicators conference (pp. 598–604).

  • Walker, J., Fenton, E., Salandra, R., & Salter, A. (2018). What influences business academics’ use of the Association of Business Schools’ (ABS) list? Evidence from a survey of UK academics. British Journal of Management. https://doi.org/10.1111/1467-8551.12294.

    Article  Google Scholar 

  • Williams, R. (2012). Using the margins command to estimate and interpret adjusted predictions and marginal effects. The Stata Journal, 12(2), 308–331.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Agency for the Evaluations of Universities and Research Institutes (ANVUR), 00153, Rome, Italy

    Tindaro Cicero & Marco Malgarini

Authors
  1. Tindaro Cicero
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marco Malgarini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marco Malgarini.

Appendices

Appendix 1: Statistical test (test t) between average SJR 2018 of A-Class journals and no A-Class journals in the 78 academic disciplines of interest

SC

NO A-Class

A-Class

test t

p value

0.580

0.919

2.844

0.002

0.580

0.580

0.919

2.844

0.002

0.580

0.580

0.919

2.844

0.002

0.580

0.580

0.919

2.844

0.002

0.580

0.580

0.919

2.844

0.002

0.580

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.472

0.534

0.825

0.205

0.472

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.491

0.494

0.045

0.482

0.491

0.491

0.494

0.045

0.482

0.491

0.491

0.494

0.045

0.482

0.491

0.492

0.493

0.015

0.494

0.492

0.491

0.494

0.045

0.482

0.491

0.492

0.493

0.015

0.494

0.492

0.492

0.493

0.015

0.494

0.492

0.821

0.700

− 0.647

0.741

0.821

0.830

0.572

− 1.613

0.946

0.830

0.834

0.560

− 1.885

0.970

0.834

0.818

0.762

− 0.267

0.605

0.818

0.804

1.355

2.232

0.013

0.804

0.771

1.617

5.335

0.000

0.771

0.763

1.201

4.036

0.000

0.763

0.763

1.192

3.961

0.000

0.763

0.763

1.199

4.003

0.000

0.763

0.763

1.192

3.958

0.000

0.763

0.763

1.199

4.003

0.000

0.763

0.780

1.171

3.192

0.001

0.780

0.759

1.284

4.581

0.000

0.759

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.424

0.836

2.716

0.003

0.424

0.889

2.922

21.078

0.000

0.889

0.889

2.922

21.078

0.000

0.889

0.889

2.922

21.078

0.000

0.889

0.889

2.922

21.078

0.000

0.889

0.889

2.922

21.078

0.000

0.889

0.868

2.820

21.202

0.000

0.868

0.868

2.820

21.202

0.000

0.868

0.868

2.820

21.202

0.000

0.868

0.868

2.820

21.202

0.000

0.868

0.868

2.820

21.202

0.000

0.868

0.875

2.604

19.279

0.000

0.875

0.898

2.964

20.939

0.000

0.898

0.898

2.964

20.939

0.000

0.898

0.898

2.964

20.939

0.000

0.898

0.908

3.112

21.481

0.000

0.908

0.819

1.575

2.596

0.005

0.819

0.624

1.458

9.396

0.000

0.624

0.818

2.337

3.794

0.000

0.818

0.838

0.766

− 0.476

0.683

0.838

0.727

1.267

5.439

0.000

0.727

0.728

1.251

5.327

0.000

0.728

0.727

1.267

5.447

0.000

0.727

0.728

1.267

5.430

0.000

0.728

Appendix 2: Statistical test (test t) between VQR average scores of A-Class journals and no A-Class journals in the 78 academic disciplines of interest

SC

NO A-Class

A-Class

test t

p value

08/C1

0.298

0.618

3.742

0.000

08/D1

0.313

0.338

0.214

0.416

08/E1

0.109

0.451

2.588

0.007

08/E2

0.515

0.637

1.161

0.125

08/F1

0.248

0.525

2.782

0.003

10/A1

0.524

0.724

2.541

0.006

10/B1

0.395

0.614

2.170

0.017

10/C1

0.682

0.633

− 0.497

0.690

10/D1

0.667

0.646

− 0.166

0.566

10/D2

0.417

0.755

3.040

0.002

10/D3

0.423

0.766

2.686

0.005

10/D4

0.654

0.821

1.347

0.092

10/E1

0.667

0.694

0.194

0.423

10/F1

0.609

0.601

− 0.072

0.528

10/F2

0.400

0.477

0.354

0.363

10/F3

0.667

0.762

0.730

0.234

10/F4

0.000

0.575

2.346

0.014

10/G1

0.600

0.778

2.119

0.018

10/H1

0.594

0.633

0.363

0.359

10/I1

0.324

0.541

2.166

0.017

10/L1

0.586

0.757

2.593

0.005

10/M1

0.401

0.624

2.089

0.020

10/M2

0.643

0.747

0.720

0.238

10/N1

0.675

0.702

0.239

0.406

10/N3

0.500

0.935

4.211

0.000

11/A1

0.273

0.645

2.885

0.003

11/A2

0.572

0.718

1.277

0.103

11/A3

0.243

0.629

4.715

0.000

11/A4

0.821

0.774

− 0.542

0.705

11/A5

0.571

0.644

0.586

0.280

11/B1

0.514

0.737

1.928

0.029

11/C1

0.732

0.769

0.317

0.376

11/C2

0.353

0.811

5.235

0.000

11/C3

0.414

0.607

1.535

0.065

11/C4

0.605

0.663

0.558

0.289

11/C5

0.596

0.732

1.451

0.075

11/D1

0.545

0.846

3.475

0.000

11/D2

0.588

0.654

0.744

0.229

12/A1

0.302

0.459

1.937

0.028

12/B1

0.329

0.427

0.854

0.198

12/B2

0.107

0.531

3.516

0.001

12/C1

0.452

0.513

0.480

0.317

12/C2

0.250

0.608

1.409

0.090

12/D1

0.190

0.508

4.490

0.000

12/D2

0.120

0.484

2.476

0.011

12/E1

0.461

0.639

1.456

0.075

12/E2

0.448

0.566

1.201

0.117

12/E3

0.375

0.490

0.838

0.203

12/E4

0.444

0.458

0.072

0.471

12/F1

0.167

0.433

1.929

0.031

12/G1

0.220

0.467

2.068

0.022

12/G2

0.417

0.678

1.560

0.065

12/H1

0.335

0.669

2.607

0.007

12/H2

0.692

0.893

1.585

0.062

12/H3

0.240

0.582

3.404

0.001

13/A1

0.306

0.938

16.940

0.000

13/A2

0.333

0.955

13.556

0.000

13/A3

0.278

0.965

11.581

0.000

13/A4

0.386

0.947

8.088

0.000

13/A5

0.533

0.991

5.548

0.000

13/B1

0.225

0.986

16.351

0.000

13/B2

0.295

0.971

15.535

0.000

13/B3

0.325

1.000

10.902

0.000

13/B4

0.291

1.000

8.962

0.000

13/B5

0.500

1.000

2.192

0.017

13/C1

0.225

0.917

6.546

0.000

13/D1

0.453

0.930

8.213

0.000

13/D2

0.324

0.907

8.069

0.000

13/D3

0.218

0.914

8.673

0.000

13/D4

0.408

0.975

9.906

0.000

14/A1

0.474

0.893

2.871

0.003

14/A2

0.414

0.703

3.234

0.001

14/B1

0.535

0.821

1.758

0.043

14/B2

0.594

0.614

0.149

0.441

14/C1

0.303

0.645

4.706

0.000

14/C2

0.403

0.438

0.408

0.342

14/C3

0.333

0.566

1.575

0.062

14/D1

0.367

0.617

2.578

0.006

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cicero, T., Malgarini, M. On the use of journal classification in social sciences and humanities: evidence from an Italian database. Scientometrics 125, 1689–1708 (2020). https://doi.org/10.1007/s11192-020-03581-8

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-020-03581-8

Keywords

Mathematics Subject Classification

JEL Classification

Search

Navigation