Evaluating Students’ Errors on Cognitive Tasks: Applications of Polytomous Item Response Theory and Log-Linear Modeling

  • Jonna M. Kulikowich
  • Patricia A. Alexander
Chapter
Part of the Perspectives on Individual Differences book series (PIDF)

Abstract

The adage, “We learn from our mistakes,” is a familiar one. Most of us recognize that some of our most meaningful learning experiences have come about as a result of saying or doing the wrong thing. The value of mistakes, however, is dependent upon our ability to recognize them as such and to gather information from them that points us in a more positive direction. The errors that students make in classrooms can also be instructive if we acknowledge that mistakes typically arise from thoughtful, albeit misguided or incomplete, processing and if there is a systematic way to identify these mistakes and to unlock the diagnostic information they hold (Alexander, 1989; Alexander, Pate, Kulikowich, Farrell, & Wright, 1989).

Keywords

Item Response Theory Error Pattern Item Response Theory Model Human Biology Educational Measurement 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alexander, P. A. (1989). Categorizing learner responses on domain-specific analogy tests: A case for error analysis. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.Google Scholar
  2. Alexander, P. A. (1992). Domain knowledge: Evolving themes and emerging concerns. Educational Psychologist, 27, 33–51.CrossRefGoogle Scholar
  3. Alexander, P. A., & Judy, J. E. (1988). The interaction of domain-specific and strategic knowledge in academic performance. Review of Educational Research, 58, 375–404.CrossRefGoogle Scholar
  4. Alexander, P. A., Pate, P. E., Kulikowich, J. M., Farrell, D. M., & Wright, N. L. (1989). Domain-specific and strategic knowledge: Effects of training on students of differing ages or competence levels. Learning and Individual Differences, 1, 283–325.CrossRefGoogle Scholar
  5. Alexander, P. A., Willson, V. L., White, C. S., & Fuqua, J. D. (1987). Analogical reasoning in young children. Journal of Educational Psychology, 26, 401–408.CrossRefGoogle Scholar
  6. Ashlock, R. B. (1986). Error patterns in computation: A semi-programmed approach. Columbus, OH: Merrill.Google Scholar
  7. Baker, E. L., & Herman, J. L. (1983). Task structure design: Beyond linkage. Journal of Educational Measurement, 20, 149–164.CrossRefGoogle Scholar
  8. Bishop, Y. M. M., Fienberg, S. E., & Holland, P. W. (1975). Discrete multivariate analysis: Theory and practice. Cambridge, MA: MIT Press.Google Scholar
  9. Birenbaum, M., & Tatsuoka, K. K. (1983). The effect of a scoring system based on the algorithm underlying the students’ response patterns on the dimensionality of achievement test data of the problem solving type. Journal of Educational Measurement, 20, 17–26.CrossRefGoogle Scholar
  10. Bock, R. D. (1972). Estimating item parameters and latent proficiency when the responses are scored in two or more nominal categories. Psychometrika, 37, 29–51.CrossRefGoogle Scholar
  11. Brown, J. S., & Burton, R. (1978). Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science, 2, 155–192.CrossRefGoogle Scholar
  12. Brown, J. S., & VanLehn, K. (1980). Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379–426.CrossRefGoogle Scholar
  13. Carey, S. (1985). Are children fundamentally different kinds of thinkers and learners than adults? In S. F. Chipman, J. W. Segal, and R. Glaser (Eds.), Thinking and learning skills (Vol. 2: pp. 485–517 ). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  14. Chi, M. T. H. (1985). Interactive roles of knowledge and strategies in the development of organized sorting and recall. In S. F. Chipman, J. W. Segal, and R. Glaser (Eds.), Thinking and learning skills (Vol. 2: pp. 457–484 ). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  15. Clement, J. (1982). Students’ preconceptions in introductory mechanics. American Journal of Physics, 50, 66–71.CrossRefGoogle Scholar
  16. Embretson, S. E. (1984). A general latent trait model for response processes. Psychometrika, 49, 175–186.CrossRefGoogle Scholar
  17. Embretson, S. E. (1985). Multicomponent latent trait models for test design. In S. E. Embretson (Ed.), Test design: Developments in psychology and psychometrics (pp. 195218 ). Orlando, FL: Academic Press.Google Scholar
  18. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215–251.CrossRefGoogle Scholar
  19. Garner, R. (1987). Metacognition and reading comprehension. Norwood, NJ: Ablex. Garner, R., Alexander, P. A., Gillingham, M. G., Kulikowich, J. M., & Brown, R. (1991).Google Scholar
  20. Interest and learning from text. American Educational Research Journal, 28,643–659.Google Scholar
  21. Geboyts, R. J., & Claxton-Oldfield, S. P. (1989). Errors in the quantification of uncertainty: A product of heuristics or minimal probability knowledge base? Applied Cognitive Psychology, 3, 157–170.CrossRefGoogle Scholar
  22. Green, B. F., Crone, C. R., & Folk, V. G. (1989). A method of studying differential distractor functioning. Journal of Educational Measurement, 26, 147–160.CrossRefGoogle Scholar
  23. Gronlund, N. E., & Linn, R. L. (1990). Measurement and evaluation in teaching. New York: Macmillan.Google Scholar
  24. Guttman, L., & Schlesinger, I. M. (1967). Systematic construction of distractors for ability and achievement test items. Educational and Psychological Measurement, 27, 569–580.CrossRefGoogle Scholar
  25. Hambleton, R. K., & Cook, L. L. (1977). Latent trait models and their use in the analysis of educational test data. Journal of Educational Measurement, 14, 75–96.CrossRefGoogle Scholar
  26. Hambleton, R. K., Roberts, D., & Traub, R. E. (1970). A comparison of the reliability and validity of two methods for assessing partial knowledge on a multiple-choice test. Journal of Educational Measurement, 7, 75–82.CrossRefGoogle Scholar
  27. Judy, J. E., Alexander, P. A., Kulikowich, J. M., & Willson, V. L. (1988). Effects of two instructional approaches and peer tutoring on gifted and nongifted sixth graders’ analogy performance. Reading Research Quarterly, 23, 236–256.CrossRefGoogle Scholar
  28. Kulikowich, J. M. (1990). Application of latent trait and multidimensional scaling models to cognitive domain-specific tests. Unpublished doctoral dissertation, Texas AandM University, College Station, TX.Google Scholar
  29. Kulikowich, J. M., & Alexander, P. A. (1990). Application of a General Euclidean Model to analyze hierarchically-constructed achievement tests. Paper presented at the annual meeting of the American Educational Research Association, Boston.Google Scholar
  30. Linn, R. L. (1990). Has item response theory increased the validity of achievement test scores? Applied Measurement in Education, 3, 115–141.CrossRefGoogle Scholar
  31. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149174.Google Scholar
  32. Matz, M. (1982). A process model for high school algebra errors. In D. Sleeman and J. S. Brown (Eds.), Intelligent tutoring systems. London: Academic.Google Scholar
  33. Mislevy, R. J., Yamamoto, K., & Anacker, S. (1991). Toward a test theory for assessing student understanding. (Tech. Rep. No. RR–91–32–0NR). Princeton, NJ: Educational Testing Service.Google Scholar
  34. Pate, P. E., Alexander, P. A., & Kulikowich, J. M. (1989). Assessing the effects of training social studies content and analogical reasoning processes on sixth-graders’ domain-specific and strategic knowledge. In D. B. Strahan (Ed.), Middle school research: Selected studies 1989 (pp. 19–29 ). Columbus, OH: Research Committee of the National Middle School Association.Google Scholar
  35. Payne, S. J., & Squibb, H. R. (1990). Algebra mal-rules and cognitive accounts of error. Cognitive Science, 14, 445–481.CrossRefGoogle Scholar
  36. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danmarks Paedagogiske Institut.Google Scholar
  37. Resnick, L. B. (1989). Treating mathematics as an ill-structured discipline. In R. I. Charles and E. A. Silver (Eds.), The teaching and assessing of mathematical problem solving (pp. 32–60 ). Reston, VA: National Council of Teachers of Mathematics.Google Scholar
  38. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement No. 17.Google Scholar
  39. Sax, G. (1989). Principles of educational and psychological measurement and evaluation. Belmont, CA: Wadsworth.Google Scholar
  40. Sheehan, K., & Mislevy, R. J. (1990). Integrating cognitive and psychometric models to measure document literary. Journal of Educational Measurement, 27, 255–272.CrossRefGoogle Scholar
  41. Smith, R. (1987). Assessing partial knowledge in vocabulary. Journal of Educational Measurement, 13, 130–141.Google Scholar
  42. Snow, R. E., & Mandinach, E. B. (1991). Integrating assessment and instruction: A research and development agenda. (Tech. Rep. No. RR-91–8). Princeton, NJ: Educational Testing Service.Google Scholar
  43. Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on Item Response Theory. Journal of Educational Measurement, 20, 345–354.CrossRefGoogle Scholar
  44. Tatsuoka, K. K., & Tatsuoka, M. M. (1983). Spotting erroneous rules of operation by the individual consistency index. Journal of Educational Measurement, 3, 221–230.CrossRefGoogle Scholar
  45. Tatsuoka, K. K., & Tatsuoka, M. M. (1987). Bug distributions and statistical pattern classification. Psychometrika, 52, 193–206.CrossRefGoogle Scholar
  46. Thissen, D. (1976). Information in wrong responses to the Ravens Progressive Matrices. Journal of Educational Measurement, 13, 201–214.CrossRefGoogle Scholar
  47. Thissen, D., & Steinberg, L. (1984). A response model for multiple choice items. Psychometrika, 49, 501–519.CrossRefGoogle Scholar
  48. Thissen, D., Steinberg, L., & Fitzpatrick, A. R. (1989). Multiple-choice models: The dis-tractors are also part of the item. Journal of Educational Measurement, 26, 161–176.CrossRefGoogle Scholar
  49. White, R. T. (1985). Interview protocols and dimensions of cognitive structure. In L. H. T. West and A. L. Pines (Eds.), Cognitive structure and conceptual change (pp. 51–59 ). New York: Academic Press.Google Scholar
  50. Whitely, S. E. (1980). Multicomponent latent trait models for ability tests. Psychometrika, 45, 479–494.CrossRefGoogle Scholar
  51. Wilson, M. R. (1989). Saltus: A psychometric model of discontinuity in cognitive development. Psychological Bulletin, 105, 276–289.CrossRefGoogle Scholar
  52. Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago: MESA.Google Scholar

Copyright information

© Springer Science+Business Media New York 1994

Authors and Affiliations

  • Jonna M. Kulikowich
    • 1
  • Patricia A. Alexander
    • 2
  1. 1.Department of Educational PsychologyUniversity of ConnecticutStorrsUSA
  2. 2.Department of Educational PsychologyTexas A&M UniversityCollege StationUSA

Personalised recommendations