The optimal number of options for multiple-choice questions on high-stakes tests: application of a revised index for detecting nonfunctional distractors
Research suggests that the three-option format is optimal for multiple choice questions (MCQs). This conclusion is supported by numerous studies showing that most distractors (i.e., incorrect answers) are selected by so few examinees that they are essentially nonfunctional. However, nearly all studies have defined a distractor as nonfunctional if it is selected by fewer than 5% of examinees. A limitation of this definition is that the proportion of examinees available to choose a distractor depends on overall item difficulty. This is especially problematic for mastery tests, which consist of items that most examinees are expected to answer correctly. Based on the traditional definition of nonfunctional, a five-option MCQ answered correctly by greater than 90% of examinees will be constrained to have only one functional distractor. The primary purpose of the present study was to evaluate an index of nonfunctional that is sensitive to item difficulty. A secondary purpose was to extend previous research by studying distractor functionality within the context of professionally-developed credentialing tests. Data were analyzed for 840 MCQs consisting of five options per item. Results based on the traditional definition of nonfunctional were consistent with previous research indicating that most MCQs had one or two functional distractors. In contrast, the newly proposed index indicated that nearly half (47.3%) of all items had three or four functional distractors. Implications for item and test development are discussed.
KeywordsAssessment Multiple-choice questions Test development Item-writing guidelines High-stakes testing
The authors express their gratitude to NBME for supporting this research. However, the opinions expressed here are those of the authors and do not necessarily reflect the position of NBME or the United States Medical Licensing Examination.
Compliance with ethical standards
Conflict of interest
The authors have no conflicts of interest to report. After IRB review by the American Institutes of Research, it was determined that this research is exempt from IRB review and oversight.
- Fisher, K. M., Wandersee, J. H., & Moody, D. E. (2000). Mapping biology knowledge. Boston, MA: Kluwer Academic.Google Scholar
- Roid, G. H., & Haladyna, T. M. (1982). A technology for test-item writing. New York: Academic Press.Google Scholar
- Wakefield, J. A. (1958). Does the fifth choice strengthen a test item? Public Personnel Review, 19, 44–48.Google Scholar
- Wallach, P. M., Crespo, L. M., Holtzman, K. Z., Galbraith, R. M., & Swanson, D. B. (2006). Use of a committee review process to improve the quality of course examination. Academic Medicine, 77(2), 156–161.Google Scholar