An Option-Based Partial Credit Item Response Model

Abstract

Multiple-choice (MC) tests have been criticized for allowing guessing and the failure to credit partial knowledge, and alternative scoring methods and response formats (Ben-Simon et al., Appl Psychol Meas 21:65–88, 1997) have been proposed to address this problem. Modern test theory addresses these issues by using binary item response models (e.g., 3PL) with guessing parameters, or with polytomous IRT models. We propose an option-based partial credit IRT model and a new scoring rule based on a weighted Hamming distance between the option key and the option response vector. The test taker (TT)’s estimated ability is based on information from both correct options and distracters. These modifications reduce the TT’s ability to guess and credit the TT’s partial knowledge. The new model can be tailored to different formats, and some popular IRT models, such as the 2PL and Bock’s nominal model, are special cases of the proposed model. Markov Chain Monte Carlo (MCMC) analysis was used to estimate the model parameters and it provides satisfactory estimates of the model parameters. Simulation studies show that the weighted Hamming distance scores have the highest correlation with TTs’ true abilities, and their distribution is also less skewed than those of the other scores considered.

Appendices

Appendix 1

We can use the following form to derive the Fisher information for an item

$$ I\left(\theta \right)=E\left[\left.{\left(\frac{\partial }{\partial \theta } \log p\left(X;\theta \right)\right)}^2\right|\theta \right]. $$

The logarithm of the likelihood for the model given in Eq. (22) is

$$ log\left[p\left(\boldsymbol{r}\Big|\theta, \boldsymbol{r}\in R\right)\right]={\displaystyle \sum_{k=1}^K}{r}_k log({x}_k)- log\left[{\displaystyle \sum_{\boldsymbol{r}\in R}}\left({\displaystyle \prod_{k=1}^K}{x}_k^{r_k}\right)\right]. $$

Note that

$$ \frac{\partial }{\partial \theta } \log \left({x}_k\right)={a}_k,\ \frac{\partial {x}_k^{r_k}}{\partial \theta }={r}_k{a}_k{x}_k^{r_k}. $$

So

$$ \frac{\partial }{\partial \theta}\left\{ log\left[p\left(\boldsymbol{r}\Big|\theta, \boldsymbol{r}\in R\right)\right]\right\}={\displaystyle \sum_{k=1}^K}{r}_k{a}_k-{\left[{\displaystyle \sum_{\boldsymbol{r}\in R}}\left({\displaystyle \prod_{k=1}^K}{x}_k^{r_k}\right)\right]}^{-1}{\displaystyle \sum_{\boldsymbol{r}\in R}}\left[\frac{\partial }{\partial \theta}\left({\displaystyle \prod_{k=1}^K}{x}_k^{r_k}\right)\right]. $$

The derivative in the last term may be simplified as follows:

$$ \frac{\partial }{\partial \theta}\left({\displaystyle \prod_{k{=}1}^K}{x}_k^{r_k}\right)\!{=}\!{\displaystyle \sum_{h{=}1}^K}\left[\left(\frac{\partial {x}_h^{r_h}}{\partial \theta}\right){\displaystyle \prod_{k\ne h}^K}{x}_k^{r_k}\right]{=}{\displaystyle \sum_{h=1}^K}\left[\left({r}_h{a}_h{x}_h^{r_h}\right){\displaystyle \prod_{k\ne h}^K}{x}_k^{r_k}\right]{=}{\displaystyle \prod_{k{=}1}^K}{x}_k^{r_k}\left[{\displaystyle \sum_{h{=}1}^K}\left({r}_h{a}_h\right)\right]\!{.} $$

Thus, we may write the derivative of the log likelihood for an item as

$$ \begin{array}{lll}{}&\frac{\partial }{\partial \theta}\left\{ log\left[p\left(\boldsymbol{r}\Big|\theta, \boldsymbol{r}\in R\right)\right]\right\}\\ &\quad ={\displaystyle \sum_{k=1}^K}{r}_k{a}_k-{\left[{\displaystyle \sum_{\boldsymbol{r}\in R}}\left({\displaystyle \prod_{k=1}^K}{x}_k^{r_k}\right)\right]}^{-1}{\displaystyle \sum_{\boldsymbol{r}\in R}}\left\{{\displaystyle \prod_{k=1}^K}{x}_k^{r_k}\left[{\displaystyle \sum_{h=1}^K}\left({r}_h{a}_h\right)\right]\right\}.\end{array} $$

Appendix 2

Start with the expression for the derivative of the log likelihood,

$$ \begin{array}{lll}{}&\frac{\partial }{\partial \theta}\left\{ log\left[p\left(\boldsymbol{r}\Big|\theta, \boldsymbol{r}\in R\right)\right]\right\}\\ &\quad ={\displaystyle \sum_{k=1}^K}{r}_k{a}_k-{\left[{\displaystyle \sum_{\boldsymbol{r}\in R}}\left({\displaystyle \prod_{k=1}^K}{x}_k^{r_k}\right)\right]}^{-1}{\displaystyle \sum_{\boldsymbol{r}\in R}}\left\{{\displaystyle \prod_{k=1}^K}{x}_k^{r_k}\left[{\displaystyle \sum_{h=1}^K}\left({r}_h{a}_h\right)\right]\right\}\end{array} $$

and notice that the second term is actually the expected value of the quantity ${\displaystyle \sum_{h=1}^K}\left({r}_h{a}_h\right)$. Specially, if we define

$$ s\left(\boldsymbol{r}\right)={\displaystyle \sum_{h=1}^K}\left({r}_h{a}_h\right), $$

we may write

$$ \begin{array}{lll} {} & E\left[s\left(\boldsymbol{r}\right)\Big|\theta,\ \boldsymbol{r}\in R\right]\\ & ={\displaystyle \sum_{r\in R}}\left[p\left(\boldsymbol{r}\Big|\theta, \boldsymbol{r}\in R\right)s\left(\boldsymbol{r}\right)\right]={\displaystyle \sum_{r\in R}}\left\{\left[\frac{{\displaystyle {\prod}_{k=1}^K}{x}_k^{r_k}}{{\displaystyle {\sum}_{\boldsymbol{r}\in R}}\left({\displaystyle {\prod}_{k=1}^K}{x}_k^{r_k}\right)}\right]\left[{\displaystyle \sum_{h=1}^K}\left({r}_h{a}_h\right)\right]\right\}.\end{array} $$

This allows us to rewrite the derivative of the log likelihood as

$$ \begin{array}{lll}\frac{\partial }{\partial \theta}\left\{ log\left[p\left(\boldsymbol{r}\Big|\theta, \boldsymbol{r}\in R\right)\right]\right\}&{=}\,{\displaystyle \sum_{k=1}^K}{r}_k{a}_k{-}E\left[s\left(\boldsymbol{r}\right)\Big|\theta,\ \boldsymbol{r}\in R\right]\\ &{=}\,s\left(\boldsymbol{r}\right){-}E\left[s\left(\boldsymbol{r}\right)\Big|\theta,\ \boldsymbol{r}\in R\right].\end{array} $$

The item information function then becomes

$$ \begin{array}{lll} I\left(\theta \right)&={\displaystyle \sum_{r\in R}}\left[p\left(\boldsymbol{r}\Big|\theta, \boldsymbol{r}\in R\right){\left(\frac{\partial }{\partial \theta}\left\{ \log \left[p\left(\boldsymbol{r}\Big|\theta, \boldsymbol{r}\in R\right)\right]\right\}\right)}^2\right]\\ &={\displaystyle \sum_{r\in R}}\left[p\left(\boldsymbol{r}\Big|\theta, \boldsymbol{r}\in R\right){\left(s\left(\boldsymbol{r}\right)-E\left[s\left(\boldsymbol{r}\right)\Big|\theta,\ \boldsymbol{r}\in R\right]\right)}^2\right].\end{array} $$

Since the right-hand side of this expression is the conditional variance of s(r), we may write

$$ I\left(\theta \right)= var\left[s\left(\boldsymbol{r}\right)\Big|\theta, \boldsymbol{r}\in R\right]. $$