Knowledge in Society

, Volume 1, Issue 4, pp 40–55 | Cite as

Exploring the values underlying evaluation of research: A social judgment analysis

  • Joseph W. Whorton
  • James A. Feldt
  • Delmer D. Dunn


This study uses social judgment analysis in a mailed questionnaire design to examine the judgment process used to evaluate research by editorial board members of various journals in one field of inquiry, public administration. The study examined the way board members valued criteria commonly used to evaluate research, examined the internal consistency with which they applied these values, and tested for consistency among board members in the relative weights they placed upon these criteria. The respondents evaluated 30 hypothetical research products which had been randomly scored on the criteria of clarity and organization, linkage to previous research, design and methods, contributions to knowledge, and utility of findings. The study found that respondents applied criteria consistently in evaluating research. The highest weighted criteria were contributions to knowledge and utility of findings and the least weighted was linkage to previous research. The weights seem to contravene the commonly held norms for conducting resarch by placing less weight on design and methods. Finally, the study suggests that the apparent inconsistency common to manuscript review is not caused by lack of agreement on the relative importance of the criteria, but rather by a lack of agreement on what constitutes good design or a contribution to knowledge.


Board Member Public Administration Editorial Board Member Judgment Process Judgment Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Bowen, D., Perloff, R., & Jacoby, J. (1972). Improving manuscript evaluation procedures.American Psychologist, 27, 221–225.CrossRefGoogle Scholar
  2. Brackbill, Y., & Korten, F. (1970). Journal reviewing practices: Author's and APA members' suggestions for revision.American Psychologist, 25, 937–940.CrossRefGoogle Scholar
  3. Cicchetti, D. (1980). Reliability of reviews for the American Psychologist: A biostatistical assessment of the data.American Psychologist, 35, 300–303.CrossRefGoogle Scholar
  4. Crane, D. (1967). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals.The American Sociologist, 2, 195–201.Google Scholar
  5. Dillman, D. (1978). Mail and telephone surveys: The total design method. New York: John Wiley.Google Scholar
  6. Dunn, W. (1982). Reforms as arguments.Knowledge, 3, 293–326.Google Scholar
  7. Dunn, W., Holzner, B., & Zaltman, G. (1985). Knowledge of Education (pp. 2831–2839). New York: Pergamon Press.Google Scholar
  8. Executive Decision Services (1986). Policy PC reference manual (version 2.0). Albany, NY: Executive Decision Services.Google Scholar
  9. Einhorn, H., & Hogarth, R. (1981). Behavioral decision theory: Processes of judgment and choice. In M. Rosenzweig & L. Porter (Eds.),Annual Review of Psychology (Vol. 32). Palo Alto, CA: Annual Reviews.Google Scholar
  10. Fitts, P., & Posner, M. (1967).Human performance. Belmont, CA: Brooks/Cole.Google Scholar
  11. Gottfredson, S. (1978). Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments.American Psychologist, 33, 920–934.CrossRefGoogle Scholar
  12. Gove, W. (1979). The review process and its consequences in the major sociology journals.Contemporary Sociology, 8, 799–804.CrossRefGoogle Scholar
  13. Hammond, K., McClelland, G., & Mumpower, J. (1980).Human judgment and decision making: Theories, methods and procedures. New York: Praeger.Google Scholar
  14. Hammond, K., Rohrbaugh, J., Mumpower, J. & Adelman, L. (1977). Social judgment theory: Applications in policy formation, In M. Kaplan & S. Schwartz (Eds.),Human judgment and decision processes in applied settings. New York: Academic Press.Google Scholar
  15. Hammond, K., Stewart, T. Brehmer, B., & Steinmann, D. (1975). Social judgment theory. In M. Kaplan & S. Schwartz (Eds.),Human judgment and decision processes. New York: Academic Press.Google Scholar
  16. Hogarth, R. (1980).Judgment and choice: The psychology of decision. New York: John Wiley.Google Scholar
  17. Holzner, B. (1982). Guest editor's introduction.Knowledge: Creation, Diffusion, Utilization, 4: 3–6.Google Scholar
  18. Manohey, M. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system.Cognitive Therapy and Research, 1, 161–175.CrossRefGoogle Scholar
  19. McReynolds, P. (1971). Reliability of ratings of research papers.American Psychologist, 26, 400–401.CrossRefGoogle Scholar
  20. Merton, R. (1973). Paradigm for the sociology of knowledge. In R. Merton,The Sociology of Science (pp. 7–40). Chicago: University of Chicago Press.Google Scholar
  21. Rohrbaugh, J., & Wher, P. (1978). Judgment analysis in policy formation: A new method for improving public participation.Public Opinion Quarterly, 42, 521–532.CrossRefGoogle Scholar
  22. Scarpello, V. (March 1988). Reducing error in the scholarly paper review process. Paper presented at meeting of the Institute for Behavioral Research, University of Georgia, Athens.Google Scholar
  23. Scott, W. (1974). Inter-referee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology.American Psychologist, 29, 698–702.CrossRefGoogle Scholar
  24. Simon, H. (1947). Administrative behavior: A study of decision-making processes in administrative organization. New York: Macmillan.Google Scholar
  25. Simon, H. (1957).Models of man: Social and rational. New York: John Wiley.Google Scholar
  26. Slovic, P. (1974). Limitations on the mind of man: Implications for decision making in the nuclear age.Oregon Research Institute Research Bulletin, 11, 17.Google Scholar
  27. Slovic, P., Fischoff, B., & Lichtenstein, S. (1977). Behavioral decision theory, In M. Rosenweig & L. Porter (Eds.),Annual Review of Psychology (Vol. 28). Palo Alto, CA: Annual Reviews.Google Scholar
  28. Wolff, W. (1973). Publication problems in psychology and an explicit evaluation schema for manuscripts.American Psychologist, 28, 257–261.CrossRefGoogle Scholar
  29. Wolff, W. (1970). A study of criteria for journal manuscripts.American Psychologist, 25, 636–639.CrossRefGoogle Scholar

Copyright information

© Winter 1989

Authors and Affiliations

  • Joseph W. Whorton
    • 1
  • James A. Feldt
    • 2
  • Delmer D. Dunn
    • 1
  1. 1.Institute of Community and Area DevelopmentUniversity of GeorgiaAthens
  2. 2.Institute of Community and Area Development's Center for Creative Decision Marking at the University of GeorgiaAthens

Personalised recommendations