Abstract
Validation is an argument for the interpretation of test scores. It is often the case that the validators do not formally consider how the various interactions that take place during the process of test item development, for example with the subject-matter experts (SMEs), may be influencing the results obtained. . . Every effort should be undertaken to more fully understand the impact the validator has on the validation process through the documentation of qualitative evidence regarding how the transactions between and among the validators and SMEs influence the data collection effort upon which interpretations of test scores are made.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
American Educational Research Association, American Psychological Association, National Council on Measurement in Education: Standards for educational and psychological testing. American Educational Research Association, Washington (1999)
American Psychological Association, American Educational Research Association, National Council on Measurement in Education: Standards for educational and psychological testing. American Educational Research Association, Washington (1985)
Bhola D.S., Impara J.C., Buckendahl C.W.: Aligning tests with states’ content standards: Methods and issues. Educational Measurement: Issues and Practice 22(3), 21–29 (2003)
Campbell D.T., Fiske D.W.: Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol. Bull. 56, 81–105 (1959)
Chapelle C.A., Enright M.K., Jamieson J.: Does an argument-based approach to validity make a difference?. Educational Measurement: Issues and Practice 29(1), 3–13 (2010)
Creswell J.W.: Research design: Qualitative, quantitative, and mixed methods approaches. 2nd edn. Sage, Thousand Oaks, CA (2003)
Cronbach L.J.: Test validation. In: Thorndike, R.L. (eds) Educational measurement, pp. 443–507. American Council on Education, Washington (1971)
Denzin, N.K., Lincoln, Y.S.: Introduction. In: Denzin, N.K., Lincoln, Y.S. (eds.) The SAGE handbook of qualitative research, 3rd edn., pp. 1–32. Sage, Thousand Oaks (2005)
Embretson S.E.: Construct validity: A universal system or just another test evaluation procedure?. Educ. res. 36, 449–455 (2007)
Fletcher, J.D., Tobias, S.: Cognitive readiness to deal with the unexpected and metacognition. Paper presented at the 33rd annual Eastern Educational Research Association, Savannah (2010)
Garfinkel H.: Studies in ethnomethodology. Prentice Hall, New York (1967)
Goldstein, J., Behuniak, P.: Exploring an argument-based approach to validation with the CMT/CAPT skills checklist. Paper presented at the annual meeting of the Northeastern Educational Research Association, Rocky Hill (October, 2009)
Gulliksen H.: Perspective on measurement. Appl. Psychol. Meas. 10(2), 109–132 (1986)
Habermas J.: Knowledge and human interests. Beacon Press, Boston (1968)
Hambleton R.K.: Test score validity and standard setting methods. In: Berk, R.A. (eds) Criterion-referenced measurement: The state of the art, pp. 80–123. Johns Hopkins University Press, Baltimore (1980)
Hambleton R.K.: Validating the test scores. In: Berk, R.A. (eds) A guide to criterion-referenced test—construction, pp. 199–230. Johns Hopkins University Press, Baltimore (1984)
Holland P.W., Thayer D.T.: Differential item functioning and the Mantel-Haenszel procedure. In: Wainer, H., Braun, H.I. (eds) Test validity, pp. 129–145. Erlbaum, Hillsdale (1988)
House E.R.: The logic of evaluative argument. CSE Monograph Series in Evaluation, No. 7. University of California Center for the Study of Evaluation, Los Angeles (1977)
House E.R.: Evaluating with validity. Sage, Beverly Hills (1980)
Kane, M.: Validation. In: Brennan, R. (ed.) Educational measurement, 4th edn., pp. 17–64. American Council on Education, Praeger Westport (2006)
Kane M.: Content-related validity evidence in test development. In: Downing, S.M., Haladyna, T.M. (eds) Handbook of test development, pp. 131–153. Lawrence Erlbaum Associates, Mahwah (2006)
Kane M.T.: Terminology, emphasis, and utility in validation. Educ. Res. 37(2), 76–82 (2008)
Krishnan, K.: Equipped for change: Development and implementation of a case statement at an urban community college foundation. Unpublished doctoral dissertation. Rowan University, New Jersey (2010)
Lindquist E.F.: The theory of test construction. In: Hawkes, H.E., Mann, C.R. (eds) The construction and use of achievement examinations, pp. 17–106. Houghton-Mifflin, Boston (1936)
Linn R.L.: Issues of validity in measurement for competency-based programs. In: Bunda, M., Sanders, J. (eds) Practices and problems in competency-based measurement, pp. 108–123. National Council on Measurement in Education, Washington (1979)
Linn R.L.: Issues of validity for criterion-referenced measures. Appl. Psychol. Meas. 4, 547–561 (1980)
Lissitz R.W., Samuelsen K.: A suggested change in terminology and emphasis regarding validity and education. Educ. Res. 36, 437–448 (2007)
Lord F.M.: Applications of item response theory to practical testing problems. Erlbaum, Hillsdale (1980)
Messick S.: Test validity and the ethics of assessment. Am. Psychol. 55, 1012–1027 (1980)
Messick S.: The once and future issues of test validity: Assessing the meaning and consequences of measurement. In: Wainer, H., Braun, H. (eds) Test validity, pp. 33–45. Erlbaum, Hillsdale (1988)
Messick S.: Validity. In: Linn, R.L. (eds) Educational measurement, pp. 13–103. American Council on Education and Macmillan, New York, NY (1989)
Miles M.M., Huberman A.M.: Qualitative data analysis: An expanded sourcebook. Sage, Thousand Oaks (1994)
Mislevy, R.J.: Cognitive psychology and educational assessment. In: Brennan, R.L. (ed.) Educational measurement, 4th edn., pp. 257–305. American Council on Education/Praeger, Westport (2006)
Mislevy R.J., Moss P., Gee J.: On qualitative and quantitative reasoning in validity. In: Ercikan, K., Roth, W-M. (eds) Generalizing from educational research: Beyond qualitative and quantitative polarization., pp. 67–100. Taylor and Francis, London, United Kingdom (2009)
Mislevy R.J., Riconscente M.M.: Evidence-centered assessment design. In: Downing, S.M., Haladyna, T.M. (eds) Handbook of test development, pp. 61–90. Erlbaum, Mahwah (2006)
Moss P.A., Phillips D.C., Erickson F.D., Floden R.E., Lather P.A., Schneider B.L.: Learning from our differences: A dialogue across perspectives on quality in educational research. Educ. Res. 38, 501–517 (2009)
Polin, L., Baker, E.L.: Qualitative analysis of test item attributes for domain-referenced content validity judgments. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco (1979)
Rist, R.C.: Overview on the relations among educational research paradigms: From disdain to détente. Anthropology and Education Quarterly 42-49 (1977)
Rovinelli R.J., Hambleton R.K.: On the use of content specialists in the assessment of criterion-referenced test item validity. Tijdschrift voor Onderwijsresearch 2, 49–60 (1977)
Secolsky C.: Using examinee judgments for detecting ambiguity on teacher-made criterion-referenced tests. J. Educ. Meas. 20(1), 51–63 (1983)
Shepard L.A.: Evaluating test validity. Review of Research in Education 19, 405–450 (1993)
Stake R.E.: Qualitative research: Studying how things work. Guilford Press, New York, NY (2010)
Webb E.J., Campbell D.T., Schwartz R.D., Sechrest L.: Unobtrusive measures: Nonreactive research in the social sciences. Rand McNally, Chicago, IL (1966)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Secolsky, C., Wentland, E. & Denison, B. The need for documenting validation transactions: a qualitative component of the testing validation process. Qual Quant 45, 1303–1311 (2011). https://doi.org/10.1007/s11135-011-9457-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-011-9457-6