Conclusions, Discussions, and Future Directions

Abstract

Scoring constructed response items is more time consuming and complex than scoring multiple-choice items. Many educational and noneducational institutions adapt open-ended questions in examinations for admission, certification, graduation, accountability, and licensing purposes. These examinations often are administered on a large-scale basis. Large volumes of examinees are tested and yet only a short time (usually a few weeks) is available for scoring. Many raters are recruited to score the examinations and it is infeasible to assign all the raters to score every one of the examinees. Thus each examinee will be scored by a selection of raters, leading to sparsely-filled data sets and also unbalanced designs, and also causing potentially biased and imprecise estimators. The current monograph developed and validated a method, called the subdividing method, to resolve this problem. The subdividing method, drawing on the concept that one could obtain more stable estimators by synthesizing multiple data sources than using just one source, is set out to improve the accuracy and precision of estimates quantifying measurement errors in the framework of G theory. The implementation of this method was discussed in section 3.1.