, Volume 90, Issue 2, pp 429–443 | Cite as

Comparison of Brazilian researchers in clinical medicine: are criteria for ranking well-adjusted?

  • Eduardo A. Oliveira
  • Enrico A. Colosimo
  • Daniella R. Martelli
  • Isabel G. Quirino
  • Maria Christina L. Oliveira
  • Leonardo S. Lima
  • Ana Cristina Simões e Silva
  • Hercílio Martelli-Júnior


Quantifying the relative performance of individual scholars has become an integral part of decision-making in research policy. The objective of the present study was to evaluate if the scholarship rank of Brazilian Council for Scientific and Technological Development (CNPq) researchers in Medicine is consistent with their scientific productivity. The Lattes curricula of 411 researchers (2006–2008) were included in the study. Scholarship category was the variable of interest. Other variables analyzed were: time since receiving the doctorate, teaching activity (undergraduate, master’s and doctoral students), number of articles published, and number of papers indexed by the Institute for Scientific Information (ISI) and Scopus databases. Additional performance indicators included were: citations, h-index, and m-index. There was a significant difference among scholarship categories regarding number of papers per year, considering the entire scientific career (P < 0.001) or the last 5 years (P < 0.001). There was no significant difference among scholarship categories regarding the number of citations per article in the ISI (Thomson Reuters) database (P = 0.23). There was a significant difference in h-index among scholarship categories in both databases, i.e. (P < 0.001) and Scopus (P < 0.001). Regarding the m-index, there was a significant difference among categories only in the ISI database (P = 0.012). According to our findings, a better instrument for qualitative and quantitative indicators is needed to identify researchers with outstanding scientific output.


Scientific publication indicators Clinical medicine Health postgraduate programs Health sciences H index Scopus 

JEL Classification




This study was partially supported by CNPq (National Council for Scientific and Technological Development) and FAPEMIG (Research Support Foundation of Minas Gerais). Eduardo A. Oliveira and Ana Cristina Simões e Silva were the recipients of CNPq scholarships Category 2 in the area of Medicine. Hercílio Martelli-Júnior and Enrico A. Colosimo were the recipients of CNPq scholarships Category 2 in the areas of Dentistry and Mathematics, respectively.


  1. Abramo, G., D’Angelo, C. A., & Costa, F. (2011a). National research assessment exercises: The effects of changing the rules of the game during the game. Scientometrics, 88, 229–238.CrossRefGoogle Scholar
  2. Abramo, G., D’Angelo, C. A., & Costa, F. (2011b). Research productivity: Are higher academic ranks more productive than lower ones? Scientometrics. doi: 10.1007/s11192-011-0426-6.
  3. Allen, L., Jones, C., Dolby, K., Lynn, D., & Walport, M. (2009). Looking for landmarks: The role of expert review and bibliometric analysis in evaluating scientific publication outputs. PLoS One, 4(6), e5910. doi: 10.1371/journal.pone.0005910.CrossRefGoogle Scholar
  4. Arruda, D., Bezerra, F., Neris, V. A., Toro, P. R., & Wainer, J. (2009). Brazilian computer science research: Gender and regional distributions. Scientometrics, 79(3), 651–665.CrossRefGoogle Scholar
  5. Barata, R. B., & Goldbaum, M. (2003). A profile of researchers in public health with productivity grants from the Brazilian National Research Council (CNPq). Cad Saude Publica, 19(6), 1863–1876.CrossRefGoogle Scholar
  6. Boell, S. K., & Wilson, C. S. (2010). Journal impact factors for evaluating scientific performance: Use of h-like indicators. Scientometrics, 82, 613–626.CrossRefGoogle Scholar
  7. Bornmann, L., & Daniel, H.-D. (2007). What do we know about the h index? Journal of the American Society for Information Science, 58, 1381–1385.CrossRefGoogle Scholar
  8. Bornmann, L., Mutz, R., & Daniel, H.-D. (2008a). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. Journal of the American Society for Information Science & Technology, 59(5), 830–837.CrossRefGoogle Scholar
  9. Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H.-D. (2008b). Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8, 93–102.CrossRefGoogle Scholar
  10. Bornmann, L., Wallon, G., & Ledin, A. (2008c). Is the h index related to (standard) bibliometric measures and to the assessments by peers? An investigation of the h index by using molecular life sciences data. Research Evaluation, 17(2), 149–156.CrossRefGoogle Scholar
  11. Browman, H. I., & Stergiou, K. I. (2008). Factors and indices are one thing, deciding who is scholarly, why they are scholarly, and the relative value of their scholarship is something else entirely. Ethics in Science and Environmental Politics, 8, 1–3.CrossRefGoogle Scholar
  12. Burrell, Q. L. (2007a). Hirsch index or Hirsch rate? Some thoughts arising from Liang’s data. Scientometrics, 73(1), 19–28.CrossRefGoogle Scholar
  13. Burrell, Q. L. (2007b). Hirsch’s h-index: A stochastic model. Journal of Informetrics, 1(1), 16–25.CrossRefGoogle Scholar
  14. Cavalcante, R. A., Barbosa, D. R., Bonan, P. R. F., Pires, M. B. O., & Martelli-Junior, H. (2008). Perfil dos pesquisadores da área de odontologia no Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq). Revista Brasileira de Epidemiologia, 11(1), 106–113.CrossRefGoogle Scholar
  15. CNPq, Ministério de Ciência e Tecnologia. (2010). Critérios de Julgamento—CA-MD. Accessed 4th Aug 2010.
  16. Franceschini, F., & Maisano, D. (2011). Proposals for evaluating the regularity of a scientist’s research output. Scientometrics, 88, 279–295.CrossRefGoogle Scholar
  17. Geuna, A., & Martin, B. R. (2003). University research evaluation and funding: An international comparison. Minerva, 41, 277–304.CrossRefGoogle Scholar
  18. Glanzel, W. (2006). On the h-index: A mathematical approach to a new measure of publication activity and citation impact. Scientometrics, 67, 315–321.CrossRefGoogle Scholar
  19. Glänzel, W., & Schubert, A. (2003). A new classification scheme of science fields and subfields designed for scientometric evaluation purposes. Scientometrics, 56, 357–367.CrossRefGoogle Scholar
  20. Haeffner-Cavaillon, N., & Graillot-Gak, C. (2009). The use of bibliometric indicators to help peer-review assessment. Archivum Immunologiae et Therapiae Experimentalis (Warsz), 57(1), 33–38.CrossRefGoogle Scholar
  21. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.CrossRefGoogle Scholar
  22. Jeang, K. T. (2008). H-index, mentoring-index, highly-cited and highly-accessed: How to evaluate scientists? Retrovirology, 5, 106.CrossRefGoogle Scholar
  23. Lane, J. (2010). Let’s make science metrics more scientific. Nature, 464(7288), 488–489.CrossRefGoogle Scholar
  24. Lehmann, S., Jackson, A. D., & Lautrup, B. E. (2006). Measures for measures. Nature, 444(7122), 1003–1004.CrossRefGoogle Scholar
  25. Leite, P., Mugnaini, R., & Leta, J. (2011). A new indicator for international visibility: Exploring Brazilian scientific community. Scientometrics, 88, 311–319.CrossRefGoogle Scholar
  26. Martelli-Junior, H., Martelli, D. R., Quirino, I. G., Oliveira, M. C., Lima, L. S., & Oliveira, E. A. (2010). CNPq researchers in medicine: A comparative study of research areas. Revista Da Associacao Medica Brasileira, 56(4), 478–483.CrossRefGoogle Scholar
  27. Mendes, P. H. C., Martelli, D. R., Souza, W. P., Filho, S. Q., & Martelli Junior, H. (2010). Perfil dos pesquisadores bolsistas de produtividade científica na medicina no CNPq, Brasil. Revista Brasileira de Educação Médica (in press).Google Scholar
  28. Moed, H. F. (2009). New developments in the use of citation analysis in research evaluation. Arch Immunol Ther Exp (Warsz), 57(1), 13–18. doi: 10.1007/s00005-009-0001-5.CrossRefGoogle Scholar
  29. Montgomery, D. C. (2008). Design and analysis of experiments (Vol. 1, 7th ed.). Boston: Wiley.Google Scholar
  30. Oliveira, E. A., Pecoits-Filho, R., Quirino, I. G., Oliveira, M. C. L., Martelli, D. R., Lima, L. S., et al. (2011a). Perfil e produção científica dos pesquisadores do CNPq nas áreas de Nefrologia e Urologia. Jornal Brasileiro de Nefrologia, 33, 31–37.CrossRefGoogle Scholar
  31. Oliveira, E. A., Ribeiro, A. L. P., Quirino, I. G., Oliveira, M. C., Martelli, D. R., Lima, L. S., et al. (2011b). Perfil e produção científica dos pesquisadores do Conselho Nacional de Desenvolvimento Científico e Tecnológico na área de Cardiologia. Arquivos Brasileiro de Cardiologia. doi: 10.1590/S0066-782X2011005000086
  32. Panaretos, J., & Malesios, C. (2009). Assessing scientific research performance and impact with single indices. Scientometrics, 81(3), 635–670.CrossRefGoogle Scholar
  33. Petherick, A. (2010). High hopes for Brazilian science. Nature, 465(7299), 674–675.CrossRefGoogle Scholar
  34. Podlubny, I. (2005). Comparison of scientific impact expressed by the number of citations in different fields of science. Scientometrics, 64, 95–99.CrossRefGoogle Scholar
  35. Regalado, A. (2010). Science in Brazil. Brazilian science: Riding a gusher. Science, 330(6009), 1306–1312. doi: 10.1126/science.330.6009.1306.Google Scholar
  36. Santos, N. C. F., Candido, L. F. O., & Kuppens, C. L. (2010). Produtividade em pesquisa do CNPq: Análise do perfil dos pesquisadores da química. Quimica Nova, 33(2), 489–495.CrossRefGoogle Scholar
  37. Santos, S. M. C., Lima, L. S., Martelli, D. R. B., & Martelli Junior, H. (2009). Perfil dos pesquisadores da Saúde Coletiva no Conselho Nacional de Desenvolvimento Científico e Tecnológico. Physis Revista de Saúde Coletiva, 19(3), 761–775.CrossRefGoogle Scholar
  38. Shibayama, S. (2011). Distribution of academic research funds: A case of Japanese national research grant. Scientometrics, 88, 43–60.CrossRefGoogle Scholar
  39. van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67, 491–502.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2011

Authors and Affiliations

  • Eduardo A. Oliveira
    • 1
    • 4
  • Enrico A. Colosimo
    • 2
  • Daniella R. Martelli
    • 3
  • Isabel G. Quirino
    • 1
  • Maria Christina L. Oliveira
    • 1
  • Leonardo S. Lima
    • 3
  • Ana Cristina Simões e Silva
    • 1
  • Hercílio Martelli-Júnior
    • 3
  1. 1.Department of PediatricsUniversidade Federal de Minas Gerais Belo HorizonteBrazil
  2. 2.Department of StatisticsUniversidade Federal de Minas GeraisBelo HorizonteBrazil
  3. 3.Programa de Pós-Graduação em Ciências da SaúdeUniversidade Estadual de Montes ClarosMontes ClarosBrazil
  4. 4.Belo HorizonteBrazil

Personalised recommendations