Skip to main content
SpringerLink
Log in
Book cover

Validity and Validation in Social, Behavioral, and Health Sciences pp 193–213Cite as

Synthesis of Validation Practices in Two Assessment Journals: Psychological Assessment and the European Journal of Psychological Assessment

  • Chapter
  • First Online:

Part of the book series: Social Indicators Research Series ((SINS,volume 54))

Abstract

The objective of this study was to conduct a research synthesis of validation practices based on the Standards for Educational and Psychological Testing (AERA, APA, and NCME, Standards for educational and psychological testing. American Educational Research Association, Washington, DC, 1999) in a random selection of 50 articles published in 2011 or 2012 in two premier assessment journals, Psychological Assessment and the European Journal of Psychological Assessment. Commonalities and differences in findings between the journals were identified. Reliability evidence was reported in similarly high percentages of studies from both journals, with internal consistency being the most commonly reported type of evidence. Internal structure and relations to other variables sources of validity evidence were strongly favored in our sample of studies, with little to no evidence presented related to test content, response processes, or consequences of testing. Weaknesses in validation practice and reporting are identified and recommendations for improvement are provided.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    To be referred to henceforth as The Standards.

  2. 2.

    A reference list of the 50 articles randomly selected from PA and EJPA may be requested from the first author.

  3. 3.

    Percentages in this section do not sum to 100 % as some studies may have included multiple language versions of a measure, used more than one sample, or collected data from more than one country.

References

  • American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Barry, A. E., Chaney, B. H., Piazza-Gardner, A. K., & Chavarria, E. A. (2014). Validity and reliability reporting practices in the field of health education and behavior: a review of seven journals. Health Education & Behavior, 41, 12–18.

    Article  Google Scholar 

  • Chuma, J., & Mahadun, P. (2011). Predicting the development of schizophrenia in high-risk populations: Systematic review of the predictive validity of prodromal criteria. The British Journal of Psychiatry, 199, 361–366. doi:10.1192/bjp.bp.110.086868.

    Article  Google Scholar 

  • Cizek, G. J., Rosenberg, S. L., & Koons, H. H. (2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68, 397–412. doi:10.1177/0013164407310130.

    Article  Google Scholar 

  • Cizek, G. J., Bowen, D., & Church, K. (2010). Sources of validity evidence for educational and psychological tests: A follow-up study. Educational and Psychological Measurement, 70, 732–743. doi:10.1177/0013164410379323.

    Article  Google Scholar 

  • Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston.

    Google Scholar 

  • de Vet, H. W., AdÒr, H. J., Terwee, C. B., & Pouwer, F. (2005). Are factor analytical techniques used appropriately in the validation of health status questionnaires? A systematic review on the quality of factor analysis of the SF-36.Quality of Life Research, 14, 1203–1218. doi:10.1007/s11136-004-5742-3.

    Article  Google Scholar 

  • Green, C. E., Chen, C. E., Helms, J. E., & Henze, K. T. (2011). Recent reliability reporting practices in psychological assessment: Recognizing the people behind the data. Psychological Assessment, 23, 656–669. doi:10.1037/a0023089.

    Article  Google Scholar 

  • Helmerhorst, H. F., Brage, S., Warren, J., Besson, H., & Ekelund, U. (2012). A systematic review of reliability and objective criterion-related validity of physical activity questionnaires. The International Journal of Behavioral Nutrition and Physical Activity, 9, 103. doi:10.1186/1479-5868-9-103.

    Article  Google Scholar 

  • Helms, J. E., Henze, K. T., Sass, T. L., & Mifsud, V. A. (2006). Treating Cronbach’s alpha reliability coefficients as data in counseling research. The Counseling Psychologist, 34, 630–660. doi:10.1177/0011000006288308.

    Article  Google Scholar 

  • Hogan, T. P., & Agnello, J. (2004). An empirical study of reporting practices concerning measurement validity. Educational and Psychological Measurement, 64, 802–812. doi:10.1177/0013164404264120.

    Article  Google Scholar 

  • Hogan, T. P., Benjamin, A., & Brezinski, K. L. (2000). Reliability methods: A note on the frequency of use of various types. Educational and Psychological Measurement, 60, 523–531. doi:10.1177/00131640021970691.

    Article  Google Scholar 

  • Hubley, A. M. (2014). Discriminant validity. In A. C. Michalos (Ed.), Encyclopedia of quality of life and well-being research (pp. 1664–1667). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Hubley, A. M., & Zumbo, B. D. (1996). A dialectic on validity: Where we have been and where we are going. Journal of General Psychology, 123, 207–215.

    Article  Google Scholar 

  • Hubley, A. M., & Zumbo, B. D. (2011). Validity and the consequences of test interpretation and use. Social Indicators Research, 103, 219–230. doi:10.1007/s11205-011-9843.4.

    Article  Google Scholar 

  • Hubley, A. M., & Zumbo, B. D. (2013). Psychometric characteristics of assessment procedures: An overview. In K. F. Geisinger (Ed.), APA handbook of testing and assessment in psychology (Vol. 1, pp. 3–19). Washington, DC: American Psychological Association Press.

    Google Scholar 

  • Jonson, J. L., & Plake, B. S. (1998). A historical comparison of validity standards and validity practices. Educational and Psychological Measurement, 58, 736–753. doi:10.1177/0013164498058005002.

    Article  Google Scholar 

  • Meier, S. T., & Davis, S. R. (1990). Trends in reporting psychometric properties of scales used in counseling psychology research. Journal of Counseling Psychology, 37, 113–115. doi:10.1037/0022-0167.37.1.113.

    Article  Google Scholar 

  • Mihura, J. L., Meyer, G. J., Dumitrascu, N., & Bombel, G. (2013). The validity of individual Rorschach variables: Systematic reviews and meta-analyses of the comprehensive system. Psychological Bulletin, 139, 548–605. doi:10.1037/a0029406.

    Article  Google Scholar 

  • Qualls, A. L., & Moss, A. D. (1996). The degree of congruence between test standards and test documentation within journal publications. Educational and Psychological Measurement, 56, 209–214. doi:10.1177/0013164496056002002.

    Article  Google Scholar 

  • Schroeder, R. W., Twumasi-Ankrah, P., Baade, L. E., & Marshall, P. S. (2012). Reliable digit span: A systematic review and cross-validation study. Assessment, 19, 21–30. doi:10.1177/1073191111428764.

    Article  Google Scholar 

  • Singh, J. P., Desmarais, S. L., & Van Dorn, R. A. (2013). Measurement of predictive validity in violence risk assessment studies: A second‐order systematic review. Behavioral Sciences & the Law, 31, 55–73. doi:10.1002/bsl.2053.

    Article  Google Scholar 

  • Slaney, K. L., Tkatchouk, M., Gabriel, S. M., & Maraun, M. D. (2009). Psychometric assessment and reporting practices: Incongruence between theory and practice. Journal of Psychoeducational Assessment, 27, 465–476.

    Article  Google Scholar 

  • Slaney, K. L., Tkatchouk, M., Gabriel, S. M., Ferguson, L. P., Knudsen, J. R. S., & Legere, J. C. (2010). A review of psychometric assessment and reporting practices: An examination of measurement-oriented versus non-measurement-oriented domains. Canadian Journal of School Psychology, 25, 246–259. doi:10.1177/0829573510375549.

    Article  Google Scholar 

  • Thompson, B., & Snyder, P. A. (1998). Statistical significance and reliability analyses in recent Journal of Counseling & Development research articles. Journal of Counseling & Development, 76, 436–441.

    Article  Google Scholar 

  • Traub, R. E. (1994). Reliability for the social sciences. London: Sage.

    Google Scholar 

  • Vacha-Haase, T., Ness, C., Nilsson, J., & Reetz, D. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. The Journal of Experimental Education, 67, 335–341. doi:10.1080/00220979909598487.

    Article  Google Scholar 

  • Vacha-Haase, T., Kogan, L. R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509–522. doi:10.1177/00131640021970682.

    Article  Google Scholar 

  • Vacha-Haase, T., Henson, R. K., & Caruso, J. C. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562–569. doi:10.1177/0013164402062004002.

    Article  Google Scholar 

  • Van Remoortel, H., Giavedoni, S., Raste, Y., Burtin, C., Louvaris, Z., Gimeno-Santos, E., et al. (2012). Validity of activity monitors in health and chronic disease: A systematic review. The International Journal of Behavioral Nutrition and Physical Activity, 9, 84. doi:10.1186/1479-5868-9-84.

    Article  Google Scholar 

  • Whittington, D. (1998). How well do researchers report their measures? An evaluation of measurement in published educational research. Educational and Psychological Measurement, 58, 21–37. doi:10.1177/0013164498058001003.

    Article  Google Scholar 

  • Wilkinson, L., & The APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.

    Article  Google Scholar 

  • Willson, V. L. (1980). Research techniques in AERJ articles: 1969 to 1978. Educational Researcher, 9, 5–10. doi:10.2307/1175221.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Educational and Counseling Psychology, and Special Education, The University of British Columbia, 2125 Main Mall, Vancouver, BC, V6T 1Z4, Canada

    Anita M. Hubley, Sophie Ma Zhu & Ayumi Sasaki

  2. Centre for Health Evaluation and Outcome Sciences, St. Paul’s Hospital, The University of British Columbia, Vancouver, BC, Canada

    Anne M. Gadermann

Authors
  1. Anita M. Hubley
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sophie Ma Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ayumi Sasaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anne M. Gadermann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anita M. Hubley .

Editor information

Editors and Affiliations

  1. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counseling, Psychology, and Special Education, The University of British Columbia, Vancouver, British Columbia, Canada

    Bruno D. Zumbo

  2. Measurement, Evaluation, and Research Methodology (MERM) Program Department of Educational and Counseling, Psychology, and Special Education, The University of British Columbia, Vancouver, British Columbia, Canada

    Eric K.H. Chan

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Hubley, A.M., Zhu, S.M., Sasaki, A., Gadermann, A.M. (2014). Synthesis of Validation Practices in Two Assessment Journals: Psychological Assessment and the European Journal of Psychological Assessment . In: Zumbo, B., Chan, E. (eds) Validity and Validation in Social, Behavioral, and Health Sciences. Social Indicators Research Series, vol 54. Springer, Cham. https://doi.org/10.1007/978-3-319-07794-9_11

Download citation

Publish with us

Policies and ethics

Search

Navigation