Skip to main content
SpringerLink
Log in

A Critique of Validity Generalization

  • Chapter
Policy Issues in Employment Testing

Part of the book series: Evaluation in Education and Human Services ((EEHS,volume 35))

  • 72 Accesses

Abstract

Validity generalization refers to whether validity coefficients obtained in prior studies in organizational environments may be used to forecast the value of a validity coefficient in a new but similar organizational environment. Validity coefficients obtained in prior situations are “generalizable” if evidence suggests that they may be employed to forecast the magnitude of a validity coefficient in a new situation. Of perhaps greater importance is the belief that a validation analysis may not be required in the new situation if evidence for validity generalization is extensive. Conversely, a “local” or “situationally specific” validation analysis in the new situation would be required if evidence for validity generalization is unavailable or if the existing evidence suggests a lack of generalization.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 16.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Algera, J. A., P. G. W. Jansen, R. A. Roe, and P. Vijn. 1984. Validity generalization: Some critical remarks on the Schmidt-Hunter procedure. Journal of Occupational Psychology 57:197–210.

    Article  Google Scholar 

  • American Educational Research Association (AERA), American Psychological Association, and National Council on Measurement in Education. 1985. Standards for educational and psychological testing. Washington, DC: American Psychological Association. (Known as APA Standards).

    Google Scholar 

  • Bernardin H. J., and R. W. Beatty. 1984. Performance appraisal: Assessing human behavior at work. Boston, MA: Kent Publishing Company.

    Google Scholar 

  • Blum M. L., and J. C. Naylor. 1968. Industrial psychology: Its theoretical and social foundation. New York: Harper and Row.

    Google Scholar 

  • Bowers, K. S. 1973. Situationism in psychology: An analysis and a critique. Psychological Review 80:307–336.

    Article  Google Scholar 

  • Brogden H. E., and E. K. Taylor. 1950. A theory and classification of criterion bias. Educational and Psychological Measurement 10:159–186.

    Article  Google Scholar 

  • Brown, S. H. 1981. Validity generalization and situational moderation in the life insurance industry. Journal of Applied Psychology 66:664–670.

    Article  Google Scholar 

  • Burke, M. J. 1984. Validity generalization: A review and critique of the correlation model. Personnel Psychology 37:93–113.

    Article  Google Scholar 

  • Burke M. J., N. S. Raju, and K. Pearlman. 1986. An empirical comparison of the results of five validity generalization procedures. Journal of Applied Psychology 71:349–353.

    Article  Google Scholar 

  • Callender J. C., and H. G. Osburn. 1980. Development and test of a new model for validity generalization. Journal of Applied Psychology 65:543–558.

    Article  Google Scholar 

  • Callender J. C., and H. G. Osburn. 1981. Testing the constancy of validity with computer generated sampling distributions of the multiplicative model variance estimate: Result for petroleum industry validation research. Journal of Applied Psychology 66:274–281.

    Article  Google Scholar 

  • Callender J. C., and H. G. Osburn. 1982. Another view of progress in validity generalization: Reply to Schmidt, Hunter, and Pearlman. Journal of Applied Psychology 67:846–852.

    Article  Google Scholar 

  • Callender J. C., and H. G. Osburn. 1988. Unbiased estimation of sampling variance of correlations. Journal of Applied Psychology 73:321–415.

    Article  Google Scholar 

  • Callender J. C., H. G. Osburn, J. M. Greener, and S. Ashworth. 1982. The multiplicative validity generalization model: Accuracy of estimates as a function of sample size and mean, variance, and shape of the distribution of true validities. Journal of Applied Psychology 67:859–867.

    Article  Google Scholar 

  • Cronbach L. J., and R. E. Snow. 1977. Aptitudes and instructional methods. New York: Irvington

    Google Scholar 

  • Dunnette M. D., R. L. Rosse, J. S. Houston, L. M. Hough, J. Toquam, S. Lammlein, K. W. King, M. J. Bosshart, and M. Keyes. 1982. Development and validation of an industry-wide electric power plant operator selection system. Minneapolis, MN: Personnel Decisions Research Institute.

    Google Scholar 

  • Ekehammer, B. 1974. Interactionism in personality from a historical perspective. Psychological Bulletin 81:1025–1048.

    Google Scholar 

  • Endler N. S., and D. Magnusson. 1976. Toward an interactional psychology of personality. Psychological Bulletin 83:956–974.

    Article  Google Scholar 

  • Ezekiel M., and K. A. Fox. 1959. Methods of correlation and regression analysis. New York: Wiley

    Google Scholar 

  • Fisher, R. A. 1954. Statistical methods for research workers. Edinburgh: Oliver and Boyd.

    Google Scholar 

  • Ghiselli, E. E. 1966. The validity of occupational aptitude tests. New York: Wiley.

    Google Scholar 

  • Ghiselli, E. E. 1973. The validity of aptitude tests in personnel selection. Personnel Psychology 26:461–477.

    Article  Google Scholar 

  • Ghosh, B. K. 1966. Asymptotic expansions for the moments of the distribution of sample correlation coefficient. Biometrika 53:258–262.

    Article  Google Scholar 

  • Guion, R. M. 1965. Personal testing. New York: McGraw Hill.

    Google Scholar 

  • Guion, R. M. 1976. Recruiting, selection, and job placement. In Handbook of industrial and organizational psychology, M. D. Dunnette, ed. pp. 777–828. Chicago: Rand McNally.

    Google Scholar 

  • Gutenberg R. L., R. D. Arvey, H. G. Osburn, and P. R. Jeanneret. 1983. Moderating effects of decision-making/information-processing job dimensions on test validities. Journal of Applied Psychology 68:602–608.

    Article  Google Scholar 

  • Hayduk, L. A. 1987. Structural equation modeling with Lisrel. Baltimore: John Hopkins University Press.

    Google Scholar 

  • Hedges L. V., and I. Olkin. 1985. Statistical methods for meta-analysis. New York: Academic Press, Inc

    Google Scholar 

  • Herman, J. B. 1973. Are situational contingencies limiting job attitude-job performance relationships? Organizational Behavior and Human Performance 10:208–224.

    Article  Google Scholar 

  • Hotelling, H. 1953. New light on the correlation coefficient and its transforms. Journal of the Royal Statistical Society (Series B) 15:193–232.

    Google Scholar 

  • Hunter J. E., and R. F. Hunter. 1984. Validity and utility of alternative predictors of job performance. Psychological Bulletin 96:72–98.

    Article  Google Scholar 

  • Hunter J. E., F. L. Schmidt, and G. B. Jackson. 1982a. Advanced meta-analysis: Quantitative methods for cumulating research findings across studies. Beverly Hills, CA: Sage.

    Google Scholar 

  • Hunter J. E., F. L. Schmidt, and K. Pearlman. 1982b. The history and accuracy of validity generalization equations: A response to the Callender and Osburn reply. Journal of Applied Psychology 67:853–858.

    Article  Google Scholar 

  • James, L. R. 1973. Criterion models and construct validity for criteria. Psychological Bulletin 80:75–83.

    Article  Google Scholar 

  • James L. R., and A. P. Jones. 1976. Organizational structure: A review of structural dimensions and their conceptual relationships with individual attitudes and behavior. Organizational Behavior and Human Performance 16:74–113.

    Article  Google Scholar 

  • James L. R., S. A. Mulaik, and J. M. Brett. 1982. Causal analysis: Assumptions, models, and data. Beverly Hills: Sage.

    Google Scholar 

  • James L. R., R. G. Demaree, and G. Wolf. 1984. Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology 69:85–98.

    Article  Google Scholar 

  • James L. R., R. G. Demaree, and S. A. Mulaik. 1986. A note on validity generalization procedures. Journal of Applied Psychology 71:440–450.

    Article  Google Scholar 

  • James L. R., R. G. Demaree, S. A. Mulaik, and M. D. Mumford. 1988. Validity generalization: A rejoinder to Schmidt, Hunter, and Raju 1988. Journal of Applied Psychology 73:673–678.

    Article  Google Scholar 

  • Jöreskog K. G., and D. Sörbom. 1979. Advances in factor analysis and structural equation models. Cambridge, MA: Abt Books

    Google Scholar 

  • Kemery E. R., K. W. Mossholder, and L. Roth. 1987. The power of the Schmidt and Hunter additive model of validity generalization. Journal of Applied Psychology 72:30–37.

    Article  Google Scholar 

  • Kendall M. G., and A. Stuart. 1969. The advanced theory of statistics: Volume 1. London, England: Griffin.

    Google Scholar 

  • Kendall M. G., and A. Stuart. 1973. The advanced theory of statistics: Volume 2. London, England: Griffin

    Google Scholar 

  • Lent R. H., H. A. Aurbach, and L. S. Levin. 1971. Research design and validity assessment. Personnel Psychology 24:247–274.

    Article  Google Scholar 

  • Lewin, K. 1938. The conceptual representation of the measurement of psychological forces. Durham, NC: Duke University Press.

    Book  Google Scholar 

  • Lichtman C. M., and R. G. Hunt. 1971. Personality and organization theory: A review of some conceptual literature. Psychological Bulletin 76:211–294.

    Article  Google Scholar 

  • Linn R. L., and S. B. Dunbar. 1982. Validity generalization and predictive bias. Paper presented at the Fourth Johns Hopkins University National Symposium on Educational Research, Washington, DC, November 1982.

    Google Scholar 

  • Lord, F. M. 1960. An empirical study of the normality and independence of errors of measurement in test scores. Psychometrika 25:91–104.

    Article  Google Scholar 

  • Lord F. M., and M. R. Novick. 1968. Statistical theories of mental tests scores. Reading, MA: Addison-Wesley

    Google Scholar 

  • Magnusson, D. 1981. Problems in environmental analysis: An introduction. In Toward a psychology of situations: An interactional perspective, D. Magnusson, ed. Hillsdale NJ: Lawrence Erlbaum.

    Google Scholar 

  • Marascuilo, L. A. 1971. Statistical methods for behavioral science research. New York: McGraw-Hill.

    Google Scholar 

  • Millsap, R. E. 1988. Sampling variance in attenuated correlation coefficients: A Monte Carlo study. Journal of Applied Psychology 73:316–319.

    Article  Google Scholar 

  • Muirhead, R. J. 1982. Aspects of multivariate statistical theory. New York: Wiley.

    Book  Google Scholar 

  • O’Connor E. J., L. H. Peters, A. Pooyan, J. Weekley, B. Frank, and B. Erenkrantz. 1984. Situational constraint effects on performance, affective reactions, and turnover: A field replication and extension. Journal of Applied Psychology 69:663–672.

    Article  Google Scholar 

  • Osburn H. G., J. C. Callender, J. M. Greener, and S. Ashworth. 1983. Statistical power of tests of the situational specificity hypothesis in validity generalization studies: A cautionary note. Journal of Applied Psychology 68:115–122.

    Article  Google Scholar 

  • Paese P. W., and F. S. Switzer III. 1988. Validity generalization and hypothetical reliability distributions: A test of the Schmidt-Hunter procedure. Journal of Applied Psychology 73:267–274.

    Article  Google Scholar 

  • Pearlman, K. 1980. Job families: A review and discussion of their implications for personnel selection. Psychological Bulletin 87:1–28.

    Article  Google Scholar 

  • Pearlman K., F. L. Schmidt, and J. E. Hunter. 1980. Validity generalization results for tests used to predict training success and job proficiency in clerical occupations. Journal of Applied Psychology 65:373–406.

    Article  Google Scholar 

  • Pedhazur, E. J. 1982. Multiple regression in behavioral research: Explanation and prediction. New York: Holt, Rinehart, and Winston.

    Google Scholar 

  • Pervin, L. 1968. Performance and satisfaction as a function of individual environment fit. Psychological Bulletin 69:56–58.

    Article  Google Scholar 

  • Pervin, L. A. 1985. Personality: Current controversies, issues, and directions. Annual Review of Psychology 36:83–114.

    Article  Google Scholar 

  • Peters L. H., and E. J. O’Connor. 1980. Situational constraints and work outcomes: The influence of a frequently overlooked construct. Academy of Management Review 5:391–397.

    Google Scholar 

  • Peterson N. G., R. L. Rosse, and J. S. Houston, J. S. 1982. The job effectiveness prediction system, Technical Report #4, Validity Analysis. Atlanta, GA: Life Office Management Association.

    Google Scholar 

  • Raju N. S., and M. J. Burke. 1983. Two new procedures for studying validity generalization. Journal of Applied Psychology 68:382–395.

    Article  Google Scholar 

  • Raju N. S., R. Fralicx, and S. Steinhaus, S. 1986. Covariance and regression slope models for studying validity generalization. Applied Psychological Measurement 10:195–211.

    Article  Google Scholar 

  • Rasmussen J. L., and B. T. Loher. 1988. Appropriate critical percentages for the Schmidt and Hunter meta-analysis procedure: Comparative evaluation of Type I error rate and power. Journal of Applied Psychology 73:683–687.

    Article  Google Scholar 

  • Sackett P. R., M. M. Harris, and J. M. Orr. 1986. On seeking moderator variables in the meta-analysis of correlational data: A Monte Carlo investigation of statistical power and resistance to Type I error. Journal of Applied Psychology 71:302–310.

    Article  Google Scholar 

  • Schmidt F. L., and J. E. Hunter. 1977. Development of a general solution to the problem of validity generalization. Journal of Applied Psychology 62:529–540.

    Article  Google Scholar 

  • Schmidt F. L., and J. E. Hunter. 1978. Moderator research and the law of small numbers. Personnel Psychology 31:215–232.

    Article  Google Scholar 

  • Schmidt F. L., and J. E. Hunter. 1980. The future of criterion-related validity. Personnel Psychology 33:41–60.

    Article  Google Scholar 

  • Schmidt F. L., and J. E. Hunter. 1981. Employment testing: Old theories and new research findings. American Psychologist 36:1128–1137.

    Article  Google Scholar 

  • Schmidt F. L., and J. E. Hunter. 1984. A within setting empirical test of the situational specificity hypothesis in personnel selection. Personnel Psychology 37:317–326.

    Article  Google Scholar 

  • Schmidt F. L., J. E. Hunter, and V. W. Urry. 1976. Statistical power in criterion-related validity studies. Journal of Applied Psychology 61:473–485.

    Article  Google Scholar 

  • Schmidt F. L., J. E. Hunter, K. Pearlman, and G. S. Shane. 1979. Further tests of the Schmidt-Hunter Bayesian validity generalization procedure. Personnel Psychology 32:257–281.

    Article  Google Scholar 

  • Schmidt F. L., I. Gast-Rosenberg, and J. E. Hunter. 1980. Validity generalization results for computer programmers. Journal of Applied Psychology 65:643–661.

    Article  Google Scholar 

  • Schmidt F. L., J. E. Hunter, and K. Pearlman. 1981. Task differences and validity of aptitude tests in selection: A red herring. Journal of Applied Psychology 66:166–185.

    Article  Google Scholar 

  • Schmidt F. L., J. E. Hunter, and K. Pearlman. 1982. Progress in validity generalization: Comments on Callender and Osburn and further developments. Journal of Applied Psychology 67:835–845.

    Article  Google Scholar 

  • Schmidt F. L., J. E. Hunter, K. Pearlman, H. R. Hirsch, P. R. Sackett, N. Schmidt, M. L. Tenopyr, J. Kehoe, and S. Zedeck. 1985a. Forty questions about validity generalization and meta-analysis with commentaries. Personnel Psychology 38:697–798.

    Article  Google Scholar 

  • Schmidt F. L., B. P. Ocasio, J. M. Hillery, and J. E. Hunter. 1985b. Further within-setting empirical tests of the situational specificity hypothesis in personnel selection. Personnel Psychology 38:509–524.

    Article  Google Scholar 

  • Schmidt F. L., J. E. Hunter, and A. N. Outerbridge. 1986. Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance. Journal of Applied Psychology 71:432–439.

    Article  Google Scholar 

  • Schmidt F. L., J. E. Hunter, and N. S. Raju. 1988. Validity generalization and situational specificity: A second look at the 75% rule and the Fisher’s z transformation. Journal of Applied Psychology 75:665–672.

    Article  Google Scholar 

  • Schmitt N., R. Z. Gooding, R. A. Noe, and M. Kirsch. 1984. Meta-analysis of validity studies published between 1964 and 1982 and the investigation of study characteristics. Personnel Psychology 37:407–422.

    Article  Google Scholar 

  • Schneider, B. 1975. Organizational climates: An essay. Personnel Psychology 28:447–479.

    Article  Google Scholar 

  • Schneider, B. 1978. Person-situation selection: A review of some ability-situation interaction research. Personnel Psychology 31:281–297.

    Article  Google Scholar 

  • Silver N. C., and W. P. Dunlap. 1987. Averaging correlation coefficients: Should Fisher’s z transformation be used? Journal of Applied Psychology 72:146–148

    Article  Google Scholar 

  • Spector P. E., and E. L. Levine. 1987. Meta-analysis for integrating study outcomes: A Monte Carlo study of its susceptibility to Type I and Type II errors. Journal of Applied Psychology 72:3–9.

    Article  Google Scholar 

  • Terbog J. R., T. W. Lee, F. J. Smith, G. A. Davis, and M. S. Turbin. 1982. Extension of the Schmidt and Hunter validity generalization procedure to the prediction of absenteeism behavior from knowledge of job satisfaction and organizational commitment. Journal of Applied Psychology 67:440–449.

    Article  Google Scholar 

  • Thomas, H. 1988. What is the interpretation of the validity generalization estimate S2p = S2r-S2e? Journal of Applied Psychology 73:679–682.

    Article  Google Scholar 

  • Tukey, J. W. 1964. Causation, regression, and path analysis. In Statistics and mathematics in biology, O. Kempthorne, T. A. Bancroft, J. W. Gowen, and J. L. Lush, eds. New York: Hofner.

    Google Scholar 

Download references

Authors
  1. Lawrence R. James
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert G. Demaree
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stanley A. Mulaik
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Harvard University, USA

    Linda C. Wing

  2. University of California at Berkeley, USA

    Bernard R. Gifford

Rights and permissions

Reprints and permissions

Copyright information

© 1994 Springer Science+Business Media New York

About this chapter

Cite this chapter

James, L.R., Demaree, R.G., Mulaik, S.A. (1994). A Critique of Validity Generalization. In: Wing, L.C., Gifford, B.R. (eds) Policy Issues in Employment Testing. Evaluation in Education and Human Services, vol 35. Springer, Dordrecht. https://doi.org/10.1007/978-94-011-2202-3_2

Download citation

  • DOI: https://doi.org/10.1007/978-94-011-2202-3_2

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-94-010-4976-4

  • Online ISBN: 978-94-011-2202-3

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation