Abstract
In the context of educational testing, estimation of the variance components and of the reliability coefficients is at best of secondary importance to the pivotal task — assigning scores to the essays (performances, problem solutions, or the like) in a way that reflects their quality as faithfully as possible. In ideal circumstances, this would correspond to reconstructing the true score α i for each essay. A more realistic target is to get as close to α i as possible. This chapter discusses improvements on the trivial estimator of the true score, the mean score over the K sessions, \({y_{i,.}} = ({y_{i,{j_{i1}}}} + \ldots + {y_{i,{j_{iK}}}})/K,\) by means of several adjustment schemes. The variance components, σ 2 a and σ 2 b and σ 2 e play an important role in these schemes. To motivate them, consider the following extreme cases: when everybody has the same true score, σ 2 a = 0, each examinee should be given the same score, irrespective of the grades given by the raters. Similarly, when the raters vary a great deal in their severities (large σ 2 b ), or the rating is very inconsistent (large σ 2 a ), an extreme score (say, 0 or 9 on the scale 0–9) is not a strong evidence of very poor or very high quality of the essay. On average, it may be prudent to ‘pull’ the extreme scores closer to the mean, so as to hedge our bets against the largest possible errors.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 1995 Springer-Verlag New York, Inc.
About this chapter
Cite this chapter
Longford, N.T. (1995). Adjusting subjectively rated scores. In: Models for Uncertainty in Educational Testing. Springer Series in Statistics. Springer, New York, NY. https://doi.org/10.1007/978-1-4613-8463-2_3
Download citation
DOI: https://doi.org/10.1007/978-1-4613-8463-2_3
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4613-8465-6
Online ISBN: 978-1-4613-8463-2
eBook Packages: Springer Book Archive