Skip to main content
SpringerLink
Log in

Validity Evidence in the Journal of Educational Psychology: Documenting Current Practice and a Comparison with Earlier Practice

  • Chapter
  • First Online:
Validity and Validation in Social, Behavioral, and Health Sciences

Part of the book series: Social Indicators Research Series ((SINS,volume 54))

Abstract

This study examined validity evidence presented in articles published in the Journal of Educational Psychology from 2000 through 2010 with the aim of documenting current validation practice. In addition, this study examined validity evidence presented in articles published from 1950 through 1960 within the same journal to provide an understanding of whether and how validation practice has changed over time. Several categories were used to assess validation practice including the articles’ characterization of validity, sources of evidence provided, justification for and types of comparison variables used, and number of sources presented. The results showed that current validation practice appears to conform to contemporary validity theory in several ways. At the same time, however, there were many instances where current practice was based on outdated theory and practice. Furthermore, in several cases validation practice appears to be very similar to practice conducted half a century ago.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1985). Standards for education and psychological testing. Washington, DC: American Psychological Association.

    Google Scholar 

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

    Google Scholar 

  • American Psychological Association. (1954). Technical recommendations for psychological tests and diagnostic techniques. Psychological Bulletin, 51, 201–238.

    Article  Google Scholar 

  • American Psychological Association, Committee on Test Standards. (1952). Technical recommendations for psychological tests and diagnostic techniques: A preliminary proposal. American Psychologist, 7, 461–465.

    Article  Google Scholar 

  • American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (1966). Standards for educational and psychological tests and manuals. Washington, DC: American Psychological Association.

    Google Scholar 

  • American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (1974). Standards for educational and psychological tests. Washington, DC: American Psychological Association.

    Google Scholar 

  • Bandura, A. (1982). Self-efficacy mechanism in human agency. American Psychologist, 37, 122–147.

    Article  Google Scholar 

  • Bandura, A. (1993). Perceived self-efficacy in cognitive development and functioning. Educational Psychologist, 28, 117–148.

    Article  Google Scholar 

  • Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061–1071. doi:10.1037/0033-295X.111.4.1061.

    Article  Google Scholar 

  • Brenner, C. A., Perry, N. E., & Collie, R. J. (2012, September). Student teachers’ developing practices that promote self-regulated learning: Linking efficacy and utility beliefs to effectiveness. In N. Perry & B. Kramarski (Co-chairs), Metacognition and self-regulation in developing professionals. Symposium presented at the biennial meeting of the European Association for Research on Learning and Instruction, Milan, Italy.

    Google Scholar 

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by multitrait–multimethod matrix. Psychological Bulletin, 56, 81–105.

    Article  Google Scholar 

  • Caprara, G. V., Barbaranelli, C., Steca, P., & Malone, P. S. (2006). Teachers’ self-efficacy beliefs as determinants of job satisfaction and students’ academic achievement: A study at the school level. Journal of School Psychology, 44, 473–490. doi:10.1016/j.jsp.2006.09.001.

    Article  Google Scholar 

  • Cizek, G. J., Rosenberg, S. L., & Koons, H. H. (2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68, 397–412. doi:10.1177/0013164407310130.

    Article  Google Scholar 

  • Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation. Design and analysis issues for field settings. Chicago: Rand McNally.

    Google Scholar 

  • Cronbach, L. J. (1949). Essentials of psychological testing. New York: Harper.

    Google Scholar 

  • Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.

    Article  Google Scholar 

  • Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2011). Investigating the substantive aspect of construct validity for the satisfaction with life scale adapted for children: A focus on cognitive processes. Social Indicators Research, 100, 37–60. doi:10.1007/s11205-010-9603-x.

    Article  Google Scholar 

  • Geisinger, K. F. (1992). The metamorphosis of test validation. Educational Psychologist, 27, 197–222.

    Article  Google Scholar 

  • Goodwin, L. D., & Leech, N. L. (2003). The meaning of validity in the new standards for educational and psychological testing: Implications for measurement courses. Measurement and Evaluation in Counseling and Development, 36(3), 181–191.

    Google Scholar 

  • Hubley, A. M., & Zumbo, B. D. (1996). A dialectic on validity: Where we have been and where we are going. Journal of General Psychology, 123, 207.

    Article  Google Scholar 

  • Hubley, A. M., & Zumbo, B. D. (2011). Validity and the consequences of test interpretation and use. Social Indicators Research: An International Interdisciplinary Journal for Quality of Life Measurement, 103, 219–230. doi: http://dx.doi.org/10.1007/s11205-011-9843-4.

  • Jonson, J. L., & Plake, B. S. (1998). A historical comparison of validity standards and validity practices. Educational and Psychological Measurement, 58, 736–753. doi:10.1177/0013164498058005002.

    Article  Google Scholar 

  • Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38, 319–342.

    Article  Google Scholar 

  • Loevinger, J. (1957). Objective tests as instruments of psychological theory. Psychological Reports, 3, 635–694.

    Article  Google Scholar 

  • Messick, S. (1988). The once and future issues of validity: Assessing the meaning and consequences of measurement. In H. Wainer & H. Braun (Eds.), Test validity (pp. 33–45). Hillsdale: Erlbaum.

    Google Scholar 

  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: Macmillan.

    Google Scholar 

  • Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749. doi:10.1037/0003-066X.50.9.741.

    Article  Google Scholar 

  • Rulon, P. J. (1946). On the validity of educational tests. Harvard Educational Review, 16, 296.

    Google Scholar 

  • Shepard, L. A. (1993). Evaluating test validity. In L. Darling-Hammond (Ed.), Review of research in education (Vol. 19, pp. 405–450). Washington, DC: AERA.

    Google Scholar 

  • Shepard, L. A. (1997). The centrality of test use and consequences for test validity. Educational Measurement: Issues and Practice, 16, 5–8, 13, 24.

    Google Scholar 

  • Sireci, S. G. (1998). The construct of content validity. Social Indicators Research, 45, 83–117.

    Article  Google Scholar 

  • Sireci, S. G. (2009). Packing and unpacking sources of validity evidence: History repeats itself again. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions, and applications (pp. 19–37). Charlotte: Information Age Publishing.

    Google Scholar 

  • Spies, R. A., & Plake, B. S. (Eds.). (2005). The sixteenth mental measurements yearbook. Lincoln: Buros Institute of Mental Measurements.

    Google Scholar 

  • Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics (Psychometrics, Vol. 26, pp. 45–79). Amsterdam: Elsevier Science.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Education, University of New South Wales, Sydney, NSW, 2052, Australia

    Rebecca J. Collie

  2. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counseling Psychology, and Special Education, The University of British Columbia, 2125 Main Mall, Vancouver, BC, V6T 1Z4, Canada

    Bruno D. Zumbo Ph.D.

Authors
  1. Rebecca J. Collie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruno D. Zumbo Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Rebecca J. Collie or Bruno D. Zumbo Ph.D. .

Editor information

Editors and Affiliations

  1. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counseling, Psychology, and Special Education, The University of British Columbia, Vancouver, British Columbia, Canada

    Bruno D. Zumbo

  2. Measurement, Evaluation, and Research Methodology (MERM) Program Department of Educational and Counseling, Psychology, and Special Education, The University of British Columbia, Vancouver, British Columbia, Canada

    Eric K.H. Chan

Appendix

Appendix

References of Articles Published Between 1950 and 1960

Ausubel, D. P., Schiff, H. M., & Zeleny, M. P. (1954). Validity of teachers’ ratings of adolescents’ adjustment and aspirations. Journal of Educational Psychology, 45, 394–406. doi:10.1037/h0056091.

Brown, W. F., & Holtzman, W. H. (1955). A study-attitudes questionnaire for predicting academic success. Journal of Educational Psychology, 46, 75–84. doi:10.1037/h0039970.

Chappell, T. L. (1955). Note on the validity of the army general classification test as a predictor of academic achievement. Journal of Educational Psychology, 46, 53–55. doi:10.1037/h0044316.

Doppelt, J. E., & Wesman, A. G. (1952). The differential aptitude tests as predictors of achievement test scores. Journal of Educational Psychology, 43, 210–217. doi:10.1037/h0060030.

French, J. W. (1958). Validation of new item types against four-year academic criteria. Journal of Educational Psychology, 49, 67–76. doi:10.1037/h0046064.

French, J. W. (1959). The relationship of home and school experiences to scores on achievement tests. Journal of Educational Psychology, 50, 75–82. doi:10.1037/h0047991.

Gage, N. L. (1957). Logical versus empirical scoring keys: The case of the MTAI. Journal of Educational Psychology, 48, 213–216. doi:10.1037/h0047795.

Holland, J. L. (1959). The prediction of college grades from the California psychological inventory and the scholastic aptitude test. Journal of Educational Psychology, 50, 135–142. doi:10.1037/h0041909.

Lodge, W. J. (1951). A validity study of personality questionnaires at the upper elementary grade level. Journal of Educational Psychology, 42, 21–30. doi:10.1037/h0061737.

Lorge, I., & Diamond, L. K. (1954). Validity of an objective examination for the placement of foreign students in English courses. Journal of Educational Psychology, 45, 208–214. doi:10.1037/h0053872.

Neidt, C. O., & Merrill, W. R. (1951). Relative effectiveness of two types of response to items of a scale on attitudes toward education. Journal of Educational Psychology, 42, 432–436. doi:10.1037/h0056066.

Papavassiliou, I. T. (1953). The validity of the goodenough draw-A-man test in Greece. Journal of Educational Psychology, 44, 244–248. doi:10.1037/h0057111.

Schultz, D. G. (1954). Item validity and response change under two different testing conditions. Journal of Educational Psychology, 45, 36–43. doi:10.1037/h0059845.

Scott, O., & Brinkley, S. G. (1960). Attitude changes of student teachers and the validity of the Minnesota Teacher Attitude Inventory. Journal of Educational Psychology, 51, 76–81. doi:10.1037/h0040593.

Uhlinger, C. A., & Stephens, M. A. (1960). Relation of achievement motivation to academic achievement in students of superior ability. Journal of Educational Psychology, 51, 259–266. doi:10.1037/h0041083.

Woodcock, R. W. (1958). An experimental prognostic test for remedial readers. Journal of Educational Psychology, 49, 23–27. doi:10.1037/h0042526.

References of Articles Published Between 2000 and 2010

Bong, M. (2009). Age-related differences in achievement goal differentiation. Journal of Educational Psychology, 101, 879–896. doi:10.1037/a0015945.

Branum-Martin, L., Mehta, P. D., Fletcher, J. M., Carlson, C. D., Ortiz, A., Carlo, M., & Francis, D. J. (2006). Bilingual phonological awareness: Multilevel construct validation among Spanish-speaking kindergarteners in transitional bilingual education classrooms. Journal of Educational Psychology, 98, 170–181. doi:10.1037/0022-0663.98.1.170.

Brockway, J. H., Carlson, K. A., Jones, S. K., & Bryant, F. B. (2002). Development and validation of a scale for measuring cynical attitudes toward college. Journal of Educational Psychology, 94, 210–224. doi:10.1037/0022-0663.94.1.210.

Cho, K., Schunn, C. D., & Wilson, R. W. (2006). Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. Journal of Educational Psychology, 98, 891–901. doi:10.1037/0022-0663.98.4.891; 10.1037/0022-0663.98.4.891.supp (Supplemental).

Chowning, K., & Campbell, N. J. (2009). Development and validation of a measure of academic entitlement: Individual differences in students’ externalized responsibility and entitled expectations. Journal of Educational Psychology, 101, 982–997. doi:10.1037/a0016351.

Craven, R. G., Marsh, H. W., Debus, R. L., & Jayasinghe, U. (2001). Diffusion effects: Control group contamination threats to the validity of teacher-administered interventions. Journal of Educational Psychology, 93, 639–645. doi:10.1037/0022-0663.93.3.639.

d’Ailly, H. (2003). Children’s autonomy and perceived control in learning: A model of motivation and achievement in Taiwan. Journal of Educational Psychology, 95, 84–96. doi:10.1037/0022-0663.95.1.84.

Edwards, W. R., & Schleicher, D. J. (2004). On selecting psychology graduate students: Validity evidence for a test of tacit knowledge. Journal of Educational Psychology, 96, 592–602. doi:10.1037/0022-0663.96.3.592.

Elliot, A. J., & Murayama, K. (2008). On the measurement of achievement goals: Critique, illustration, and application. Journal of Educational Psychology, 100, 613–628. doi:10.1037/0022-0663.100.3.613.

Gathercole, S. E., & Pickering, S. J. (2000). Assessment of working memory in six- and seven-year-old children. Journal of Educational Psychology, 92, 377–390. doi:10.1037/0022-0663.92.2.377.

Greene, J. A., Torney-Purta, J., & Azevedo, R. (2010). Empirical evidence regarding relations among a model of epistemic and ontological cognition, academic performance, and educational level. Journal of Educational Psychology, 102, 234–255. doi:10.1037/a0017998.

Grigorenko, E. L., Jarvin, L., Diffley, R., Goodyear, J., Shanahan, E. J., & Sternberg, R. J. (2009). Are SSATS and GPA enough? A theory-based approach to predicting academic success in secondary school. Journal of Educational Psychology, 101, 964–981. doi:10.1037/a0015906.

Janosz, M., Le Blanc, M., Boulerice, B., & Tremblay, R. E. (2000). Predicting different types of school dropouts: A typological approach with two longitudinal samples. Journal of Educational Psychology, 92, 171–190. doi:10.1037/0022-0663.92.1.171.

Kardash, C. M., & Wallace, M. L. (2001). The perceptions of science classes survey: What undergraduate science reform efforts really need to address. Journal of Educational Psychology, 93, 199–210. doi:10.1037/0022-0663.93.1.199.

Legault, L., Green-Demers, I., & Pelletier, L. (2006). Why do high school students lack motivation in the classroom? Toward an understanding of academic amotivation and the role of social support. Journal of Educational Psychology, 98, 567–582. doi:10.1037/0022-0663.98.3.567.

Mokhtari, K., & Reichard, C. A. (2002). Assessing students’ metacognitive awareness of reading strategies. Journal of Educational Psychology, 94, 249–259. doi:10.1037/0022-0663.94.2.249.

Naglieri, J. A., & Rojahn, J. (2004). Construct validity of the PASS theory and CAS: Correlations with achievement. Journal of Educational Psychology, 96, 174–181. doi:10.1037/0022-0663.96.1.174.

Robbins, S. B., Allen, J., Casillas, A., Peterson, C. H., & Le, H. (2006). Unraveling the differential effects of motivational and skills, social, and self-management measures from traditional predictors of college outcomes. Journal of Educational Psychology, 98, 598–616. doi:10.1037/0022-0663.98.3.598.

Watkins, M. W., & Coffey, D. Y. (2004). Reading motivation: Multidimensional and indeterminate. Journal of Educational Psychology, 96, 110–118. doi:10.1037/0022-0663.96.1.110.

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Collie, R.J., Zumbo, B.D. (2014). Validity Evidence in the Journal of Educational Psychology: Documenting Current Practice and a Comparison with Earlier Practice. In: Zumbo, B., Chan, E. (eds) Validity and Validation in Social, Behavioral, and Health Sciences. Social Indicators Research Series, vol 54. Springer, Cham. https://doi.org/10.1007/978-3-319-07794-9_7

Download citation

Publish with us

Policies and ethics

Search

Navigation