Shift Error Detection in Standardized Exams
- 405 Downloads
Computer-graded multiple choice examinations are a familiar and dreaded part of most student’s lives. Many test takers are particularly fearful of form-filling shift errors, where absent-mindedly marking the answer to (say) question 32 in position 31 causes a long run of answers to be successively displaced. Test-taking strategies where students answer questions out of sequence (such as answering easy questions first) seem particularly likely to cause unrecognized shift errors. Such errors can result in many correct answers being marked wrong, and significantly lower the exam taker’s score.
In this paper, we consider the question of whether these shift errors can be accurately recognized and corrected for. Our results suggest that students are right to fear such errors, and that a non-trivial fraction of multiple-choice exams appear to contain significant shift errors. In particular, we found evidence that 1%–2% of multiple-choice exam papers at SUNY Stony Brook contain likely shift errors, each causing the loss of about 10% of the student's grade. Given the importance of standardized examinations, clerical mistakes should not be allowed to have such an impact on the student's score. If our observed shift error rate holds across the millions of examinations given annually, this is a serious but yet unstudied problem. Indeed, based on the preliminary results in the paper, we have begun working with The College Board to study the prevalence of uncorrected shift-errors in SAT examinations.
KeywordsTest Taker Shift Error Standardize Exam Clerical Error Educational Test Service
Unable to display preview. Download preview PDF.
- M. G. Aamodt and T. Mcshane. A meta-analytic investigation of the effect of various test item characteristics on test scores and test completion times. Public Personnel Management, 21:151–160, 1992.Google Scholar
- J. A. Bath. Answer-changing behavior on objective examinations. Journal of Educational Research, 61:105–107, 1967.Google Scholar
- M. D. Beck. The effect of item response changes on scores on an elementary reading achievement test. Journal of Educational Research, 71:153–156, 1976.Google Scholar
- C. A. Clark. Should students change answers on objective tests? Chicago Schools Journal, 43:382–385, 1962.Google Scholar
- D. A. Copeland. Should chemistry students change answers on multiple-choice tests? Journal chemical Education, 49:258, 1972.Google Scholar
- V. Dancik. Expected length of Longest Common Subsequences. PhD thesis, University of Warwik, Warwick, UK, 1994.Google Scholar
- R. Hertzberg. Vice president of marketing research, scantron. Personal communication, September 1999.Google Scholar
- NCS Inc. Annual report. World Wide Web, http://www.ncs.edu, September 1999.
- S. S. Jacobs. Answer changing on objective tests: Some implications for test validity. Educational and Psychology Measurement, 32:1039–1044, 1972.Google Scholar
- D. O. Lynch and Smith. Item response changes: Effects on test scores. Measurement and Evaluation in Guidance, 7:220–224, 1975.Google Scholar
- C. O. Mathews. Erroneous first impressions on objective tests. Journal of Educational Psychology, 20:260–286, 1929.Google Scholar
- M. K. Matter. The relationship between achievement test response changes, ethnicity, and family income. PhD thesis, University of Texas, Austin, 1985.Google Scholar
- P. J. Pascale. Changing initial answers on multiple choice achievement tests. Measurement and Evaluation in Guidance, 6:236–238, 1974.Google Scholar
- A. Smith and J. C. Moore. The effects of changing answers on scores on non-test-sophisticated examinees. Measurement and Evaluation in Guidance, 8:252–254, 1976.Google Scholar
- E. W. Stark and G. Pemmasani. Implementation of a compositional performance analysis algorithm for probabilistic i/o automata. PAPM99, September 1999.Google Scholar
- G. R. Stoffer, K. E. Davis, and J. B. Brown. The consequences of changing initial answers on objective tests: A stable effect and a stable misconception. Journal of Educational research, 70:272–277, 1977.Google Scholar