Abstract
We present a decision analysis approach to the problems faced by people subject to multiple-choice examinations, as often encountered in their education, in looking for a job, or in getting a driving permit.
From the candidate’s viewpoint, each question in this form of examination is a decision problem, where the decision space depends on the examination rules and the expected utility is some function of the expected score. We analyse this problem for the two basic situations which occur in practice, namely when the candidate wants to maximize his or her expected score, and when he or she wants to maximize the probability of obtaining the minimum grade required to pass, and we derive the corresponding optimal strategies.
We argue that for multiple-choice examinations to be fair, candidates should be required to provide a probability distribution over the possible answers to each question, rather than merely marking the answers judged to be more likely; we then discuss the appropriate scoring rules and the corresponding optimal strategies. As an interesting byproduct, we deduce some illuminating consequences on the scoring procedures of multiple-choice examinations, as they are currently performed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bernardo, J. M. (1979). Expected information as expected utility. Ann. Statist. 7, 686–690.
Bernardo, J. M. and Smith, A. F. M. (1994). Bayesian Theory. Chichester: Wiley.
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Month. Weather Rev. 78, 1–3.
Chernoff, H. (1962). The scoring of multiple-choice questionnaires. Ann. Math. Statist. 33, 375–393.
de Finetti, B. (1962). Does it make sense to speak of `Good Probability Appraisers’?, The Scientist Speculates: An Anthology of Partly-Baked Ideas (I. J. Good, ed.). New York: Wiley, 257–364. Reprinted in 1972, Probability, Induction and Statistics New York: Wiley, 19–23.
de Finetti, B. (1965). Methods for discriminating levels of partial knowledge concerning a test item. British J. Math. Statist. Psychol. 18, 87–123. Reprinted in 1972, Probability, Induction and Statistics New York: Wiley, 25–63.
Good, I. J. (1950). Probability and the Weighing of Evidence. London: Griffin; New York: Hafner Press.
Hsu, J. S. J., Leonard, T. and Tsui, K.-W. (1991). Statistical inference for multiple-choice tests. Psychometrika 56, 327–348.
Hutchinson, T. P. (1991). Ability, Partial Information, Guessing: Statistical Modelling Applied to Multiple-Choice Tests. Adelaide, Australia: Rumsby.
Hutchinson, T. P. (1993). Second attempts at multiple-choice test items. J. Statist. Computation and Simulation 47, 108–112.
Klein, S. P. (1992). Statistical evidence of cheating in multpiple-choice tests. Chance 5, 23–27.
Lierly, S. B. (1951). A note for correcting for chance success in objective tests. Psychometrika 22, 63–73.
Lindley, D. V. (1982). Scoring rules and the inevitability of probability. Internat. Statist. Rev. 50, 1–26 (with discussion).
Martín, A. and Luna, J. D. (1989). Test and intervals in multiple-choice tests: A modification of the simplest classical model. British J. Math. Statist. Philosophy 42, 251–264.
Martín, A. and Luna, J. D. (1990). Multiple-choice tests: Power, lengh and optimal number of choices per item. British J. Math. Statist. Philosophy 43, 57–72.
Nogaki, A. (1984). Some remarks on multiple-choice questions in competitive examinations. Behaviormetrika 16, 13–19.
Pollard, G. H. (1985). Scoring in multiple-choice examinations. Math. Scientist 10, 93–97.
Post, G. V. (1994). A quantal choice model for the detection of copying on multiple-choice examinations. Decision Sciences 25, 123–142.
Savage, L. J. (1971). Elicitation of personal probabilities and expectations. J. Amer. Statist. Assoc. 66, 781–801. Reprinted in 1974 in Studies in Bayesian Econometrics and Statistics: in Honor of Leonard J. Savage (S. E. Fienberg and A. Zenner, eds.). Amsterdam: North-Holland, 111–156.
Solomon, H. (1955). Item analysis and classification techniques. Proc. Third Berkeley Symp. 5 (J. Neyman and E. L. Scott, eds.). Berkeley: Univ. California Press, 169— 184.
Thissen, D. and Steinberg, L. (1984). A response model for multiple choice items. Psychometrika 49, 501–519.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1998 Springer Science+Business Media New York
About this chapter
Cite this chapter
Bernardo, J.M. (1998). A Decision Analysis Approach to Multiple-Choice Examinations. In: Girón, F.J. (eds) Applied Decision Analysis. Springer, Dordrecht. https://doi.org/10.1007/978-94-017-0759-6_16
Download citation
DOI: https://doi.org/10.1007/978-94-017-0759-6_16
Publisher Name: Springer, Dordrecht
Print ISBN: 978-90-481-5777-8
Online ISBN: 978-94-017-0759-6
eBook Packages: Springer Book Archive