Advertisement

Higher Education

, Volume 62, Issue 2, pp 199–212 | Cite as

Researchers as evaluators: tasks, tensions and politics

  • Liv Langfeldt
  • Svein Kyvik
Article

Abstract

Researchers undertake a number of different research evaluation tasks, taking up a substantial part of their research time—estimated to about one work month per year for a professor. This paper addresses the various evaluator roles and tasks researchers take on, and the tensions they involve. How the research evaluator role may conflict with the researcher role and with societal expectations is discussed, as well as the intrinsic tensions in peer review; including expertise vs. impartiality, evaluators as neutral judges vs. exercise of power and influence, divergent peer assessments vs. the need for unanimous conclusions in peer panels, peer review vs. increase in quantitative indicators, and accountability to society vs. peer review as preserving the autonomy of science. The examination of these tensions provides insight in the political aspects of peer review, and a basis for discussing an agenda for future studies on the role of peer evaluators. Major future challenges for peer review concern how to meet demands for transparency and public accountability, and maintain academic autonomy.

Keywords

Peer review Scientific quality Evaluation spiral Academic autonomy Public accountability 

References

  1. AAUW. (2004). Tenure denied. Cases of sex discrimination in academia. Washington, DC: American Association of University Women Education Foundation and American Association of University Women Legal Advocacy Fund.Google Scholar
  2. Aksnes, D. W., & Taxt, R. E. (2004). Peer review and bibliometric indicators: A comparative study at a Norwegian university. Research Evaluation, 13(1), 33–41.CrossRefGoogle Scholar
  3. Bornman, L., & Daniel, H. D. (2008). Reliability, fairness, and predictive validity of the peer review process for the selection of research fellowship recipients of the Boehringer Ingelheim fonds. In B. M. Kehm (Ed.), Hochschule im Wandel. Die Universität als Forschungsgegenstand. Festschrift für Ulrich Teichler (pp. 365–376). Frankfurt am Main: Campus.Google Scholar
  4. Campanario, J. M. (1998a). Peer review for journals as it stands today—part 1. Science Communication, 19(3), 181–211.Google Scholar
  5. Campanario, J. M. (1998b). Peer review for journals as it stands today—part 2. Science Communication, 19(4), 277–306.Google Scholar
  6. Ceci, S. J., & Peters, D. P. (1982). Peer review: A study of reliability. Change, 14(6), 44–48.Google Scholar
  7. Chubin, D. E., & Hackett, E. J. (1990). Peerless science. Peer review and US science policy. New York: State University of New York Press.Google Scholar
  8. Cole, S. (1983). The hierarchy of the sciences? American Journal of Sociology, 89(1), 111–139.CrossRefGoogle Scholar
  9. Friedman, R. M. (2001). The politics of excellence. Behind the nobel prize in science. New York: Times Books.Google Scholar
  10. Hackett, E. J. (1997). Peer review in science and science policy. In M. S. Frankel & J. Cave (Eds.), Evaluating science and scientists (pp. 51–60). Budapest: Central European University Press.Google Scholar
  11. Hansen, H. F. (2009). Research evaluation: Methods, practice, and experience. Copenhagen: Danish Agency for Science, Technology and Innovation. Research: Analysis and Evaluation.Google Scholar
  12. Hearn, J. C., & Anderson, M. S. (2002). Conflict in academic departments: An analysis of disputes over faculty promotion and tenure. Research in Higher Education, 43(5), 503–529.CrossRefGoogle Scholar
  13. Henkel, M. (2007). Can academic autonomy survive in the knowledge society? A perspective from Britain. Higher education Research & Development, 26(1), 87–99.CrossRefGoogle Scholar
  14. Horrobin, D. F. (1982). Peer review: A philosophically faulty concept which is proving disastrous for science. Behavioral and Brain Sciences, 5(2), 217–218.CrossRefGoogle Scholar
  15. Irvine, J., & Martin, B. (1984). Foresight in science: Picking the winners. London: Pinter.Google Scholar
  16. Jasanoff, S. (1990). The fifth branch, science advisers as policymakers. Cambridge, MA: Harvard University press.Google Scholar
  17. Jasanoff, S. (2003). Technologies of humility: Citizen participation in governing science. Minerva, 41(3), 223–244.CrossRefGoogle Scholar
  18. Johnston, S. (1997). Examining the examiners: An analysis of examiners’ reports on doctoral theses. Studies in Higher Education, 22(3), 333–347.CrossRefGoogle Scholar
  19. Kyvik, S., & Langfeldt, L. (2004). Tidkrevende bedømmelsesarbeid. Forskningspolitikk, 27(2), 15.Google Scholar
  20. Lamont, M. (2009). How professors think: Inside the curious world of academic judgment. Cambridge, MA: Harvard University Press.Google Scholar
  21. Langfeldt, L. (2001). The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science, 31(6), 820–841.CrossRefGoogle Scholar
  22. Langfeldt, L. (2002). Decision-making in expert panels evaluating research. Constraints, processes and bias. Oslo: Unipub. Accessed September 30, 2010, from http://www.statsvitenskap.uio.no/fag/polit/disputas/fulltxt/langfeldt.pdf.
  23. Martin, B. (2000). Research grants: Problems and options. Australian Universities’ Review, 43(2), 17–22. Accessed September 23, 2010, from http://www.bmartin.cc/pubs/00aur.html.
  24. Morley, L., Leonard, D., & David, M. (2002). Variations in vivas: Quality and equality in British PhD assessments. Studies in Higher Education, 27(3), 263–273.CrossRefGoogle Scholar
  25. Neave, G. (1998). The evaluative state reconsidered. European Journal of Education, 33(3), 265–284.Google Scholar
  26. Nowotny, H., Scott, P., & Gibbons, M. (2003). ‘Mode 2’ revisited: The new production of knowledge—Introduction. Minerva, 41(3), 179–194.CrossRefGoogle Scholar
  27. Pöschl, U., & Koop, T. (2008). Interactive open access publishing and collaborative peer review for improved scientific communication and quality assurance. Information Services & Use, 28(2), 105–107.Google Scholar
  28. Ravetz, J. R. (1971). Scientific knowledge and its social problems. Oxford: Clarendon Press.Google Scholar
  29. Roy, R. (1984). Alternatives to review by peers: A contribution to the theory of scientific choice. Minerva, 22(3–4), 316–328.CrossRefGoogle Scholar
  30. Scott, A. (2007). Peer review and the relevance of science. Futures, 39(7), 827–845.CrossRefGoogle Scholar
  31. Sörlin, S. (2007). Funding diversity: Performance-based funding regimes as drivers of differentiation in higher education systems. Higher Education Policy, 20(4), 413–440.CrossRefGoogle Scholar
  32. Speck, B. W. (1993). Publication peer review. An annotated bibliography. Westport/London: Greenwood Press.Google Scholar
  33. Tinkler, P., & Jackson, C. (2000). Examining the doctorate: Institutional policy and the PhD examination. Studies in Higher Education, 25(2), 167–180.CrossRefGoogle Scholar
  34. Travis, G. D. L., & Collins, H. M. (1991). New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology & Human Values, 16(3), 322–341.Google Scholar
  35. van den Besselaar, P., & Leydesdorf, L. (2009). Past performance, peer review and project selection: A case study in the social and behavioral sciences. Research Evaluation, 18(4), 272–288.Google Scholar
  36. Van der Meulen, B. (1998). Science policies as principal-agent games. Institutionalization and path dependency in the relation between government and science. Research Policy, 27(4), 397–414.CrossRefGoogle Scholar
  37. Weller, A. C. (2001). Editorial peer review. Its strengths and weaknesses. Medford, NJ: Information Today (ASIS&T Monograph Series).Google Scholar
  38. Woodhouse, E., & Sarewitz, D. (2007). Science policies for reducing societal inequities. Science and Public Policy, 34(3), 139–150.CrossRefGoogle Scholar
  39. Ziman, J. M. (1994). Prometheus bound: Science in a dynamic steady state. Cambridge: Cambridge University Press.Google Scholar
  40. Zuckerman, H., & Merton, R. K. (1971). Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva, 9(1), 66–100.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  1. 1.NIFU STEP Norwegian Institute for Studies in Innovation, Research and EducationOsloNorway

Personalised recommendations