, Volume 90, Issue 2, pp 383–406 | Cite as

Bibliometrics as a tool for measuring gender-specific research performance: an example from South African invasion ecology



Citations to published work are gaining increasing prominence in evaluations of the research performance of scientists. Considering the importance accorded to gender issues in South African science, it is surprising that (to our knowledge) no research has as yet ascertained the extent of sex differences in citations to the published work of scientists in this country. Our literature study shows that studies that have been conducted elsewhere tend to neglect in their analyses important gender-related and other factors, such as the sex composition of multi-authored papers and the extent of foreign co-authorship. Against this background, we illustrate the difficulties inherent in measuring the quality aspect of sex-specific research performance by means of an analysis of a dataset of articles (n = 229) that were published between 1990 and 2002 in the field of invasion ecology and in journals included in the Thomson Reuters Web of Science. Each article has at least one South African author address. The results indicate that foreign co-authorship is a better correlate of high citations than the sex of South African authors, and this is true irrespective of whether the annual citation rate or window period is used, whether or not self-citations are excluded, and whether or not the number of authors is controlled for by calculating fractional counts. The paper highlights these and other considerations that are relevant for future gender-focused bibliometric research, both in South Africa and beyond.


Bibliometrics Gender Research performance Citations Invasion ecology 



The authors thank CREST for the use of SA Knowledgebase, and the former database manager, Derick van Niekerk, for running the keyword searches. Acknowledgements are due to the Director (Steven Chown) and Deputy Director (David Richardson) of the CIB for their assistance in identifying the sex of some of the authors in the dataset, and in identifying relevant keywords. David Richardson is also thanked for highlighting the need to control for the time available for accumulation of citations, and his and Dan Simberloff’s insightful comments on an earlier draft of this paper, as well as the comments of two anonymous reviewers, are much appreciated. Research assistance for this project was funded from the first author’s CIB core team member grant. Finally, we are highly indebted to Charline Mouton, Marion van Dorssen and Christopher Mechnig for their assistance in collecting data for this project.


  1. Aksnes, D. W., Rorstad, K., Piro, F., & Sivertsen, G. (2011). Are female researchers less cited? A large-scale study of Norwegian scientists. Journal of the American Society for Information Science and Technology, 62(4), 628–636.CrossRefGoogle Scholar
  2. Beaver, D. B. (1986). Collaboration and teamwork in physics. Czechoslovak Journal of Physics, 1, 14–18.CrossRefGoogle Scholar
  3. Bordons, M., Morillo, F., Fernández, M. T., & Gómez, I. (2003). One step further in the production of bibliometric indicators at the micro level: Differences by gender and professional category of scientists. Scientometrics, 57(2), 159–173.CrossRefGoogle Scholar
  4. Bornmann, L. (2010). Towards an ideal method of measuring research performance: Some comments to the Opthof and Leydesdorff (2010) paper. Journal of Informetrics, 4, 441–443.CrossRefGoogle Scholar
  5. Borrego, Á., Barrios, M., Villarroya, A., & Ollé, C. (2010). Scientific output and impact of postdoctoral scientists: A gender perspective. Scientometrics, 83, 93–101.CrossRefGoogle Scholar
  6. Borsuk, R. M., Budden, A. E., Leimu, R., Aarssen, L. W., & Lortie, C. J. (2009). The influence of author gender, national language and number of authors on citation rate in ecology. The Open Ecology Journal, 2, 25–28.Google Scholar
  7. Boshoff, N. (2005). The representation of women academics in higher education in South Africa: Progress in the pipeline? South African Journal of Higher Education, 19(2), 359–377.Google Scholar
  8. Cole, J. R. (1979). Fair science: Women in the scientific community. New York: Free Press.Google Scholar
  9. Cole, J. R., & Cole, S. (1973). Social stratification in science. Chicago: University of Chicago Press.Google Scholar
  10. Cole, J. R., & Zuckerman, H. (1984). The productivity puzzle: Persistence and change in patterns of publication of men and women scientists. Advances in Motivation and Achievement, 2, 217–258.Google Scholar
  11. Corley, E. A., & Sabharwal, M. (2010). Scholarly collaboration and productivity patterns in public administration: Analysing recent trends. Public Administration, 88(3), 627–648.CrossRefGoogle Scholar
  12. Craig, I. D., Plume, A. M., McVeigh, M. E., Pringle, J., & Amin, M. (2007). Do open access articles have greater citation impact? A critical review of the literature. Journal of Informetrics, 1(3), 239–248.CrossRefGoogle Scholar
  13. Creamer, E. G. (1998). Assessing faculty publication productivity: Issues of equity. ASHE-ERIC Higher Education Report (Vol. 26, No. 2). Washington, DC: Association for the Study of Higher Education.Google Scholar
  14. Cronin, B., & Overfelt, K. (1994). Citation-based auditing of academic performance. Journal of the American Society for Information Science, 45(2), 61–72.CrossRefGoogle Scholar
  15. Davenport, E., & Snyder, H. (1995). Who cites women? Whom do women cite? An exploration of gender and scholarly citation in sociology. Journal of Documentation, 51(4), 404–410.CrossRefGoogle Scholar
  16. Davis, D. E., & Astin, H. S. (1987). Reputational standing in academe. The Journal of Higher Education, 58, 261–275.CrossRefGoogle Scholar
  17. Demetrulias, D. M. (1986). Gender differences in co-authorship. Journal of Educational Equity and Leadership, 6(2), 119–127.Google Scholar
  18. Drake, J., Mooney, H. A., Di Castri, F., Groves, R., Kruger, F. J., Rejmánek, M., et al. (Eds.). (1989). Biological invasions: A global perspective. Chichester: Wiley.Google Scholar
  19. Ferber, M. A. (1986). Citations: Are they an objective measure of scholarly merit? Signs: Journal of Women in Culture and Society, 11(2), 381–389.CrossRefGoogle Scholar
  20. Flores, L. Y., Rooney, S. C., Heppner, P. P., Browne, L. D., & Wei, M.-F. (1999). Trend analyses of major contributions in The Counseling Psychologist cited from 1986 to 1996: Impact and implications. The Counseling Psychologist, 27(1), 73–95.CrossRefGoogle Scholar
  21. Fox, M. F. (1991). Gender, environmental milieu, and productivity in science. In H. Zuckerman, J. R. Cole, & J. T. Bruer (Eds.), The outer circle: Women in the scientific community (pp. 188–204). New York: Norton.Google Scholar
  22. García-Pérez, M. A. (2009). The Hirsch h index in a non-mainstream area: Methodology of the behavioral sciences in Spain. The Spanish Journal of Psychology, 12(2), 833–849.Google Scholar
  23. Garfield, E. (1981). The 1,000 contemporary scientists most-cited 1965–1978 (Part I: The basic list and introduction). Current Contents, 41, 5–14. [Reprinted in: Garfield, E. (1981–1982). Essays of an information scientist, 5, 269–278].Google Scholar
  24. Gonzalez-Brambila, C., & Veloso, F. M. (2007). The determinants of research output and impact: A study of Mexican researchers. Research Policy, 36, 1035–1051.CrossRefGoogle Scholar
  25. Hammel, E. A., Mason, C., Prater, A., & Lundy, R. (1995). Gender and the academic career in North American Anthropology: Differentiating intramarket from extramarket bias. Current Anthropology, 36(2), 366–380.CrossRefGoogle Scholar
  26. Haslam, N., Ban, L., Kaufmann, L., Loughnan, S., Peters, K., Whelan, J., et al. (2008). What makes an article influential? Predicting impact in social and personality psychology. Scientometrics, 76(1), 169–185.CrossRefGoogle Scholar
  27. Helmreich, R. L., Spence, J. T., Beane, W. E., Lucker, G. W., & Matthews, K. A. (1980). Making it in academic psychology: Personality correlates of attainment. Journal of Personality and Social Psychology, 39(5), 896–908.CrossRefGoogle Scholar
  28. Housri, N., Cheung, M. C., Koniaris, L. G., & Zimmers, T. A. (2008). Scientific impact of women in academic surgery. Journal of Surgical Research, 148, 13–16.CrossRefGoogle Scholar
  29. Hunter, L. A., & Leahey, E. (2010). Parenting and research productivity: New evidence and methods. Social Studies of Science, 40(3), 433–451.CrossRefGoogle Scholar
  30. Hutson, S. R. (2002). Gendered citation practices in American Antiquity and other archaeology journals. American Antiquity, 67(2), 331–342.CrossRefGoogle Scholar
  31. Hutson, S. R. (2006). Self-citation in archaeology: Age, gender, prestige, and the self. Journal of Archaeological Method and Theory, 13(1), 1–19.CrossRefGoogle Scholar
  32. Irvine, J., & Martin, B. R. (1986). Women in radio astronomy—Shooting stars? In J. Harding (Ed.), Perspectives on gender and science (pp. 80–103). London: The Falmer Press.Google Scholar
  33. Kelly, C. D., & Jennions, M. D. (2006). The h index and career assessment by numbers. Trends in Ecology & Evolution, 21(4), 167–170.CrossRefGoogle Scholar
  34. Kolpin, V. W., & Singell, L. D., Jr. (1996). The gender composition and scholarly performance of economics departments: A test for employment discrimination. Industrial and Labor Relations Review, 49(3), 408–423.CrossRefGoogle Scholar
  35. Krampen, G. (2008). The evaluation of university departments and their scientists: Some general considerations with reference to exemplary bibliometric publication and citation analyses for a Department of Psychology. Scientometrics, 76(1), 3–21.CrossRefGoogle Scholar
  36. Leahey, E. (2007). Not by productivity alone: How visibility and specialization contribute to academic earnings. American Sociological Review, 72(4), 533–561.CrossRefGoogle Scholar
  37. Leimu, R., & Koricheva, J. (2005). What determines the citation frequency of ecological papers? Trends in Ecology & Evolution, 20(1), 28–32.CrossRefGoogle Scholar
  38. Leta, J., & Lewison, G. (2003). The contribution of women in Brazilian science: A case study in astronomy, immunology and oceanography. Scientometrics, 57(3), 339–353.CrossRefGoogle Scholar
  39. Lewison, G. (2001). The quantity and quality of female researchers: A bibliometric study of Iceland. Scientometrics, 52(1), 29–43.CrossRefGoogle Scholar
  40. Lindsey, D. (1980). Production and citation measures in the sociology of science: The problem of multiple authorship. Social Studies of Science, 10(2), 145–162.CrossRefGoogle Scholar
  41. Long, J. S. (1992). Measures of sex differences in scientific productivity. Social Forces, 71(1), 159–178.Google Scholar
  42. Lutz, C. (1990). The erasure of women’s writing in sociocultural anthropology. American Ethnologist, 17, 611–627.CrossRefGoogle Scholar
  43. MacRoberts, M. H., & MacRoberts, B. R. (1987). Problems of citation analysis: a critical review. Journal of the American Society for Information Science, 40, 342–349.CrossRefGoogle Scholar
  44. Mauleón, E., & Bordons, M. (2006). Productivity, impact and publication habits by gender in the area of materials science. Scientometrics, 66(1), 199–218.CrossRefGoogle Scholar
  45. Mauleón, E., Bordons, M., & Oppenheim, C. (2008). The effect of gender on research staff success in life sciences in the Spanish National Research Council. Research Evaluation, 17(3), 213–225.CrossRefGoogle Scholar
  46. Mcelhinny, B., Hols, M., Holtzkener, J., Unger, S., & Hicks, C. (2003). Gender, publication and citation in sociolinguistics and linguistic anthropology: The construction of a scholarly canon. Language in Society, 32, 299–328.CrossRefGoogle Scholar
  47. Montpetit, E., Blais, A., & Foucault, M. (2008). What does it take for a Canadian political scientist to be cited? Social Science Quarterly, 89(3), 802–816.CrossRefGoogle Scholar
  48. National Advisory Council on Innovation (NACI). (2008). Creating the future South African national system of innovation: Gender, race and SET sector issues. Report by the Science, Engineering and Technology for Women (SET4W), NACI Permanent Sub-Committee. Accessed 9 March 2011.
  49. Over, R., & Moore, D. (1980). Research productivity and impact of men and women in psychology departments of Australian universities, 1975–1977. Australian Psychologist, 15(3), 413–418.CrossRefGoogle Scholar
  50. Peñas, C. S., & Willet, P. (2006). Gender differences in publication and citation counts in librarianship and information science research. Journal of Information Science, 32(5), 480–485.CrossRefGoogle Scholar
  51. Persell, C. H. (1983). Gender, rewards and research in education. Psychology of Women Quarterly, 8(1), 33–47.CrossRefGoogle Scholar
  52. Powell, A., Hassan, T. M., Dainty, A. R. J., & Carter, C. (2009). Exploring gender differences in construction research: A European perspective. Construction Management and Economics, 27, 803–807.CrossRefGoogle Scholar
  53. Prozesky, H. E. (2006). Gender differences in the journal publication productivity of South African academic authors. South African Review of Sociology, 37(2), 87–112.Google Scholar
  54. Pyšek, P., Richardson, D. M., & Jarošik, V. (2006). Who cites who in the invasion zoo: Insights from an analysis of the most highly cited papers in invasion ecology. Preslia, 78, 437–468.Google Scholar
  55. Pyšek, P., Richardson, D. M., Pergl, J., Jarošík, V., Sixtová, Z., & Weber, E. (2008). Geographical and taxonomical biases in invasion ecology. Trends in Ecology & Evolution, 23, 237–244.CrossRefGoogle Scholar
  56. Qiu, H., & Chen, Y.-F. (2009). Bibliometric analysis of biological invasions research during the period of 1991 to 2007. Scientometrics, 81(3), 601–610.CrossRefGoogle Scholar
  57. Reese-Evans, L. (2010). Gender and citation in two LIS e-journals: A bibliometric analysis of LIBRES and Information Research. Library and Information Science Research Electronic Journal, 20(1), 1–19.Google Scholar
  58. Reskin, B. F. (1978). Scientific productivity, sex and location in the institution of science. American Journal of Sociology, 83(5), 1235–1243.CrossRefGoogle Scholar
  59. Richardson, D. M., & Van Wilgen, B. W. (2004). Invasive alien plants in South Africa: How well do we understand the ecological impacts? South African Journal of Science, 100(Jan/Feb), 45–52.Google Scholar
  60. Rousseau, R. (1992). Letter to the editor: Why I am not cited or why are multi-authored papers more cited than others? Journal of Documentation, 48, 79–80.CrossRefGoogle Scholar
  61. Sandström, U. (2009). Combining curriculum vitae and bibliometric analysis: Mobility, gender and research performance. Research Evaluation, 18(2), 135–142.CrossRefGoogle Scholar
  62. Sonnert, G. (1995). What makes a good scientist? Determinants of peer evaluation among biologists. Social Studies of Science, 25(1), 35–55.CrossRefGoogle Scholar
  63. Sonnert, G., & Holton, G. J. (1995). Gender differences in science careers: The Project Access Study. New Brunswick, NJ: Rutgers University Press.Google Scholar
  64. Sooryamoorthy, R. (2009). Collaboration and publication: How collaborative are scientists in South Africa? Scientometrics, 80(2), 419–439.CrossRefGoogle Scholar
  65. Stack, S. (1994). An analysis of the impacts of books and journal articles. International Review of Modern Sociology, 24(2), 119–125.Google Scholar
  66. Symonds, M. R. E., Gemmell, N. J., Braisher, T. L., Gorringe, K. L., & Elgar, M. A. (2006). Gender differences in publication output: Towards an unbiased metric of research performance. PLoS ONE, 1(2), e127.CrossRefGoogle Scholar
  67. Teghtsoonian, M. (1974). Distribution of sex by authors and editors of psychological journals, 1970–1972: Are there enough women editors? American Psychologist, 29, 262–269.CrossRefGoogle Scholar
  68. Toutkoushian, R. K. (1994). Using citations to measure sex discrimination in faculty scales. The Review of Higher Education, 18(1), 61–82.Google Scholar
  69. Tregenza, T. (2002). Gender bias in the refereeing process? Trends in Ecology & Evolution, 17(8), 349–350.CrossRefGoogle Scholar
  70. Trifunac, M. D. (2006). A note on publication and citation rates of female academics in earthquake engineering. Soil Dynamics and Earthquake Engineering, 26, 1063–1075.CrossRefGoogle Scholar
  71. Trimble, V. (1985). Some notes on patterns in citations of papers by American astronomers. Quarterly Journal of the Royal Astronomical Society, 26, 40–50.Google Scholar
  72. Trimble, V. (1993). Patterns in citations of papers by American astronomers. Quarterly Journal of the Royal Astronomical Society, 34, 235–250.Google Scholar
  73. United Nations Educational, Scientific and Cultural Organization (UNESCO). (2007). Science, technology and gender: An international report (executive summary). Science and technology for development series. Paris: UNESCO Publishing. Accessed 9 March 2011.
  74. Van Raan, A. F. J. (1998). The influence of international collaboration on the impact of research results. Scientometrics, 42(3), 423–428.CrossRefGoogle Scholar
  75. Walters, G. D. (2006). Predicting subsequent citations to articles published in twelve crime-psychology journals: Author impact versus journal impact. Scientometrics, 69(3), 499–510.CrossRefMathSciNetGoogle Scholar
  76. Walters, C. G., Fry, E. H., & Chaisson, B. D. (1990). Women scholars: Closing the publication gap. Research in Higher Education, 31(4), 355–367.CrossRefGoogle Scholar
  77. Ward, K. B., Gast, J., & Grant, L. (1992). Visibility and dissemination of women’s and men’s sociological scholarship. Social Problems, 39(3), 291–298.CrossRefGoogle Scholar
  78. Ward, K. B., & Grant, L. (1991). Gender and publishing in sociology. Gender & Society, 5(2), 207–223.CrossRefGoogle Scholar
  79. Ward, K. B., & Grant, L. (1996). Gender and academic publishing. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. xi, pp. 172–212). New York: Agathon Press.Google Scholar
  80. Webster, B. M. (2001). Polish women in science: A bibliometric analysis of Polish science and its publications, 1980–1999. Research Evaluation, 10(3), 185–194.CrossRefMathSciNetGoogle Scholar
  81. Wennerås, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387, 341–343.CrossRefGoogle Scholar
  82. Williamson, M. (1996). Biological invasions. London: Chapman and Hall.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2011

Authors and Affiliations

  1. 1.Department of Sociology and Social Anthropology and DST-NRF Centre of Excellence for Invasion BiologyStellenbosch UniversityMatielandSouth Africa
  2. 2.Centre for Research on Evaluation, Science and TechnologyStellenbosch UniversityMatielandSouth Africa

Personalised recommendations