Evaluating Federal R&D in the United States

  • Ron Kostoff
Chapter

Abstract

This chapter describes the practice of federal research impact evaluation. Evaluation of research impact is described for three cases: Research selection, where the work has not yet been performed; Research review, where work and results are ongoing; and ex-post research assessment, where research has been completed and results can be tracked. Qualitative methods (such as peer review), semi-quantitative methods (such as historiographic tracings of accomplishments and critical scientific events), and quantitative methods (such as cost-benefit analysis and bibliometrics) are described. While peer review in its broadest sense is the most widely used method in research selection, review, and ex-post assessment, it has its deficiencies, and there is no single method which provides a complete impact evaluation.

Keywords

Research Impact Research Selection Research Advisory Committee Task Area Research Impact Assessment 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Bibliography

  1. AFOSR, “Recent Research Accomplishments of the Air Force Office of Scientific Research”, AFOSR, 1989.Google Scholar
  2. Averch, H., “Policy Uses of ‘Evaluation of Research’ Literature”, OTA Contractor Report, July 1990.Google Scholar
  3. Averch, H., “The Practice of Research Evaluation in the United States,”, Research Evaluation, 1: 2, 1991.Google Scholar
  4. Cozzens, S. E., “Expert Review in Evaluating Programs”, Science and Public Policy, 14: 2, April 1987.Google Scholar
  5. DOD, “Project Hindsight”, Office of the Director of Defense Research and Engineering, DTIC Report No. AD495905, October 1969.Google Scholar
  6. DOE, “An Assessment of the Basic Energy Sciences Program”, Office of Energy Research, Office of Program Analysis, Report No. DOE/ER-0123, March 1982.Google Scholar
  7. DOE, “Health and Environmental Research: Summary of Accomplishments”, Office of Energy Research, Office of Program Analysis, Report No. DOE/ER0194, May 1983Google Scholar
  8. DOE, “Health and Environmental Research: Summary of Accomplishments”, Office of Energy Research, Office of Program Analysis, Report No. DOE/ER0275, August 1986.Google Scholar
  9. DOE, “Multiprogram Laboratory Appraisals”, DOE ORDER 5000. 2A, September 1988.Google Scholar
  10. DOE, “Procedures for Peer Review Assessments”, Office of Energy Research, Office of Program Analysis, Report No. DOE/ER-0491P, April 1991.Google Scholar
  11. Frazier, S. P., “University Funding: Information on the Role of Peer Review at NSF and NIH”, U.S. General Accounting Office Report No. GAO/RCED-8787FS, March 1987.Google Scholar
  12. Griliches, Z., “Research Costs and Social Returns: Hybrid Corn and Related Innovations,” Journal of Political Economy, 66, 1958.Google Scholar
  13. Hall, D., and Nauda, A., “An Interactive Approach for Selecting IRD Projects,” IEEE Transactions on Engineering Management, 37: 2, May 1990.CrossRefGoogle Scholar
  14. IITRI, “Technology in Retrospect and Critical Events in Science”, Illinois Institute of Technology Research Institute Report, December, 1968.Google Scholar
  15. Irvine, J. and Martin, B. R., “Foresight in Science: Picking the Winners,” Frances Pinter, London, 1984.Google Scholar
  16. Johnston, R., “Project Selection Mechanisms: International Comparisons”, OTA Contractor Report, July 1990.Google Scholar
  17. Kerpelman, L. C., and Fitzsimmons, S. J. J., “Methods for the Strategic Evaluation of Research Programs: The State-of-the Art”, and “Annotated Bibliography”, NSF Contract No. PRA 8400688, Abt Associates Inc., 1985.Google Scholar
  18. Kostoff, R. N., “A Cost/ Benefit Analysis of Commercial Fusion-Fission Hybrid Reactor Development”, Journal of Fusion Energy, 3: 2, 1983.CrossRefGoogle Scholar
  19. Kostoff, R. N., “Evaluation of Proposed and Existing Accelerated Research Programs by the Office of Naval Research”, IEEE Transactions of Engineering Management, 35: 4, November 1988.CrossRefGoogle Scholar
  20. Kostoff, R. N. and Stanford, L. B., “Program Funding Profiles under Budgetary Constraints”, Research Evaluation, 1: 1, Apr. 1991a.Google Scholar
  21. Kostoff, R. N., “A Quantitative Approach to Determining the Impact of Research”, Presented at Twenty-Second Annual Pittsburgh Conference on Modeling and Simulation, May 2–3, 1991b.Google Scholar
  22. Kostoff, R. N., “Research Impact Assessment”, Presented at Third International Conference on Management of Technology, February 17–21, 1992.Google Scholar
  23. Logsdon, J. M., and Rubin, C. B., “An Overview of Federal Research Evaluation Activities”, Program of Policy Studies in Science and Technology Report, George Washington Univ., April 1985.Google Scholar
  24. Luukonen-Gronow, T., “Scientific Research Evaluation: A Review of Methods and Various Contexts of Their Application,” RD Management, 17: 3, 1987.Google Scholar
  25. Martin, B. R. et al, “Recent trends in the Output and Impact of British Science”, Science and Public Policy, 17:1, Feb., 1990.Google Scholar
  26. Narin, F., “Bibliometric Techniques in the Evaluation of Research Programs”, Science and Public Policy, 14: 2, April 1987b.Google Scholar
  27. Narin, F., “The Impact of Different Modes of Research Funding”, in: Evered, David and Harnett, Sara, Eds., The Evaluation of Scientific Research, John Wiley and Sons, Chichester, UK, 1989.Google Scholar
  28. NIST, “Annual Report, 1990,” Visiting Committee on Advanced Technology, January 1991a.Google Scholar
  29. NIST, “An Assessment of the National Institute of Standards and Technology Programs: FY 1990,” Board on Assessment of NIST Programs, National Research Council, National Academy Press, 1991b.Google Scholar
  30. NSF, “Science and Engineering Indicators - 1989”, National Science Board Report NSB 89–1, GPO, Wash., D.C., 1989.Google Scholar
  31. ONR, “1989 ONR Significant Accomplishments: Office of Naval Research Sponsored Programs”, ONR, January 1989.Google Scholar
  32. Ormala, E., “Nordic Experiences of the Evaluation of Technical Research and Development”, Research Policy, 18, 1989.Google Scholar
  33. OTA, “Research Funding as an Investment: Can We Measure the Returns”, U. S. Congress, Office of Technology Assessment, OTA-TM-SET-36 ( Wash., DC: U. S. GPO, April 1986 ).Google Scholar
  34. OTA, “Federally Funded Research: Decisions for a Decade”, U. S. Congress, Office of Technology Assessment, OTA-SET-490 ( Wash., DC: U. S. GPO, May 1991 ).Google Scholar
  35. Salasin, J. et al, “The Evaluation of Federal Research Programs”, MITRE Technical Report MTR-80W123, June 1980.Google Scholar
  36. Wirt, J.G. et al, “RD Management: Methods Used by Federal Agencies”, Rand Report No. R-1156-HEW, January 1974.Google Scholar

Copyright information

© Springer Science+Business Media New York 1993

Authors and Affiliations

  • Ron Kostoff
    • 1
  1. 1.United States Office of Naval ResearchUSA

Personalised recommendations