The Dedisciplining of Peer Review
The demand for greater public accountability is changing the nature of ex ante peer review at public science agencies worldwide. Based on a four year research project, this essay examines these changes through an analysis of the process of grant proposal review at two US public science agencies, the National Institutes of Health (NIH) and the National Science Foundation (NSF). Weaving historical and conceptual narratives with analytical accounts, we describe the ways in which these two agencies struggle with the question of incorporating considerations of societal impact into the process of peer review. We use this comparative analysis to draw two main conclusions. First, evaluation of broader societal impacts is not different in kind from evaluation of intellectual merit. Second, the scientific community may actually bolster its autonomy by taking a broader range of considerations into its peer review processes.
KeywordsPeer review Disciplinarity Societal impacts Scientific autonomy Interdisciplinarity National Science Foundation National Institutes of Health
- Allen, Janice, 2010. ‘The NIH Scientific Review Process.’ CAPR Digital Repository, at http://csid-capr.unt.edu/fedora/repository/capr:1063
- Ben-David, Joseph. 1984. The Scientist’s Role in Society. Chicago: University of Chicago Press.Google Scholar
- Benos, Dale, et al, 2007. “The Ups and Downs of Peer Review.” Advances in Physiology Education, vol. 31, pp. 145–152.Google Scholar
- Beecher, Henry, 1966. “Ethics and Clinical Research.” New England Journal of Medicine, vol. 274, pp. 1354–60.Google Scholar
- Blanpied, William. 1999. “Science and Public Policy: The Steelman Report and the Politics of Post-World War II Science Policy.” AAAS Science and Technology Policy Yearbook, available at: http://www.aaas.org/spp/yearbook/chap29.htm.
- Bozeman, Barry, and Craig Boardman. 2009. ‘Broad Impacts and Narrow Perspectives: Passing the Buck on Science and Social Impacts.’ Special Issue, Social Epistemology: US National Science Foundation’s Broader Impacts Criterion. Vo. 23, No. 3–4, Dec. 2009, pp. 183–198.Google Scholar
- Briggle, Adam. 2005. “Institutional Review Boards,” in Encyclopedia of Science, Technology, and Ethics, Carl Mitcham (ed.), 4 vols. Farmington Hills, MI: Macmillan Reference, 2005, vol. 2, pp. 1024–26.Google Scholar
- Briggle, Adam. 2010. A Rich Bioethics: Public Policy, Biotechnology, and the Kass Council. Notre Dame, IN: University of Notre Dame Press.Google Scholar
- Carmichael, Mary, and Sharon Begley. 2010. “Desperately Seeking Cures.” Newsweek, May 15, 2010.Google Scholar
- Echegoyen, Luis. 2008. Dear Colleague Letter on Broader Impacts Proposal Requirements; at http://www.delta.wisc.edu/events/Workshops/workshop_docs/2008%20workshop%20docs/Dear_Colleague_NSF_Letter.pdf
- Fitzpatrick, Kathleen. 2010. ‘Peer-to-Peer Review and the Future of Scholarly Authority.’ Social Epistemology, vol. 24, no. 3, pp. 161–180.Google Scholar
- Frodeman, Robert, and J. Britt Holbrook. 2007. “Science’s Social Effects.” Issues in Science and Technology, Vol. 23, Issue 3 (Spring 2007), pp 28–30: http://www.issues.org/23.3/p_frodeman.html.
- Frodeman, Robert, and J. Britt Holbrook. 2011. “NSF’s Struggle to Articulate Societal Relevance.” Science, July 8, 2011.Google Scholar
- Frodeman, Robert, and J. Britt Holbrook, under review. “A Comparative Assessment of Metrics for Societal Impact.” Research Evaluation.Google Scholar
- Gieryn, Thomas. 1999. Cultural Boundaries of Science: Credibility on the Line. Chicago: University of Chicago Press.Google Scholar
- Guston, David. 2000. Between Politics and Science: Assuring the Integrity and Productivity of Research. Cambridge: Cambridge University Press.Google Scholar
- Holbrook, J. Britt. 2005. “Assessing the Science – Society Relation: The Case of the U.S. National Science Foundation’s Second Merit Review Criterion,” in Technology in Society, vol. 27, Issue 4, November 2005, pp. 437–451.Google Scholar
- Holbrook, J. Britt. 2010. “Peer Review,” in The Oxford Handbook of Interdisciplinarity, Robert Frodeman, (ed.). Oxford: Oxford University Press, 2010, pp. 321–32.Google Scholar
- Holbrook, J. Britt, and Robert Frodeman. 2011. “Peer Review and Societal Impacts: A Comparison of the US National Science Foundation and the European Commission’s Framework Programmes.” Research Evaluation, Volume 20, Number 3, September 2011, 239–246(8)Google Scholar
- Hollander, unpublished. “Ethics at the National Science Foundation (NSF): A Brief History of Bureaucratic Struggle.” Available at http://csi.ensmp.fr.
- Kaiser, Jocelyn. 2011. “Collins Sparks Furor with Proposed NIH Reshuffling.” Science, vol. 331, no. 6016 (28 January), p. 386.Google Scholar
- Kington, 2007, at http://enhancing-peer-review.nih.gov/meetings/102207-summary.html.
- Kuhn, Thomas, 1970. “Logic of Discovery or Psychology of Research?” in Criticism and the Growth of Knowledge, edited by I. Lakatos and A. Musgrave (London: Cambridge University Press) pp. 1–23.Google Scholar
- Latour, Bruno. 1993. We Have Never Been Modern. Harvard University Press.Google Scholar
- Lightborne, James. 2010. Public Presentation, CAPR Midterm Workshop, Washington, DC, April 22, 2010.Google Scholar
- Mandel, Richard. 1996. A Half Century of Peer Review, 1946–1996. Bethesda, Md.: Division of Research Grants, National Institutes of Health.Google Scholar
- NAPA, 2001. National Academy of Public Administration Report on Implementation of new Grant Proposal Guide Requirements Related to the Broader Impacts Criterion. Available at http://www.nsf.gov/publications/pub_summ.jsp?ods_key=iin127.
- National Academies. 2010. ‘Federal Demonstration Partnership: STAR metrics working Group’. Available at http://sites.nationalacademies.org/PGA/fdp/PGA_057189
- Peer Review Process. 2011. NIH website, at http://grants.nih.gov/grants/peer_review_process.htm.
- Pielke, Jr., Roger A., and R. Byerly. 1998. ‘Beyond basic and applied’. Physics Today 51 (2) 42–46, Times Cited: 17, issn: 0031–9228.Google Scholar
- Reisch, George. 2005. How the Cold War Transformed Philosophy of Science: To the Icy Slopes of Logic. Cambridge University Press: Cambridge.Google Scholar
- Roberts, Melanie. 2009. “Realizing Societal Benefit from Academic Research: Analysis of the National Science Foundation’s Broader Impacts Criterion.” Special Issue, Social Epistemology: US National Science Foundation’s Broader Impacts Criterion. Vo. 23, No. 3-4, Dec. 2009, pp. 199–219.Google Scholar
- Rossiter, Margaret W. 1984. ‘The History and Philosophy of Science Program at the National Science Foundation.’ Isis, vol. 75, no. 1, pp. 95–104.Google Scholar
- Rothenberg, Marc. 2010. ‘Making Judgments about Grant Proposals: A Brief History of the Merit Review Criteria at the National Science Foundation.’ Technology and Innovation, Vol. 12, pp. 189–195.Google Scholar
- Sarewitz, Daniel. 1996. Frontiers of Illusion: Science, Technology, and the Politics of Progress. Temple University Press.Google Scholar
- Sarewitz, Daniel. 2000. “Science and Environmental Policy: An Excess of Objectivity,” in Earth Matters: The Earth Sciences, Philosophy, and the Claims of Community, edited by Robert Frodeman (Upper Saddle River, NJ: Prentice Hall) pp. 79–98.Google Scholar
- Sarewitz, Daniel. April 08, 2003. “Does Science Policy Exist, and If So, Does it Matter?: Some Observations on the U.S. R&D Budget.” Discussion paper. Columbia University’s Earth Institute’s Science, Technology and Global Development Seminar, New York, NY.Google Scholar
- Schienke, Erich, et al. 2009. “The Role of the National Science Foundation Broader Impacts Criterion in Enhancing Research Ethics Pedagogy.” Special Issue, Social Epistemology: US National Science Foundation’s Broader Impacts Criterion. Vo. 23, No. 3–4, Dec. 2009, pp. 317–336.Google Scholar
- Shapin, Steven, and Simon Schaffer. 1985. Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life. Princeton, NJ: Princeton University Press.Google Scholar
- Spier, Ray. 2002. “The History of the Peer-review Process,” TRENDS in Biotechnology, vol. 20, no. 8, pp. 357–358.Google Scholar