Restricted Recalibration of Item Response Theory Models

Abstract

In item response theory (IRT), it is often necessary to perform restricted recalibration (RR) of the model: A set of (focal) parameters is estimated holding a set of (nuisance) parameters fixed. Typical applications of RR include expanding an existing item bank, linking multiple test forms, and associating constructs measured by separately calibrated tests. In the current work, we provide full statistical theory for RR of IRT models under the framework of pseudo-maximum likelihood estimation. We describe the standard error calculation for the focal parameters, the assessment of overall goodness-of-fit (GOF) of the model, and the identification of misfitting items. We report a simulation study to evaluate the performance of these methods in the scenario of adding a new item to an existing test. Parameter recovery for the focal parameters as well as Type I error and power of the proposed tests are examined. An empirical example is also included, in which we validate the pediatric fatigue short-form scale in the Patient-Reported Outcome Measurement Information System (PROMIS), compute global and local GOF statistics, and update parameters for the misfitting items.

Appendix A The Asymptotic Distribution of Residuals

1.1 A.1 The Pseudo-Maximum Likelihood Estimator

Under the setup of RR and Conditions A1–A6 of Gong and Samaniego (1981), the following expansion of the n-sample likelihood equation (Eq. 4) is defined for $(\hat{\varvec{\xi }}, \hat{\varvec{\eta }}){}^\top$ in some neighborhood of $(\underline{\varvec{\xi }}, \underline{\varvec{\eta }}){}^\top$:

$$\begin{aligned} \mathbf{0}= & {} \frac{1}{n}\sum _{i=1}^{n}\hat{\mathbf{g}}(\mathbf{y}_i) = \frac{1}{n}\sum _{i=1}^{n}\underline{\mathbf{g}}(\mathbf{y}_i) + \left[ \frac{1}{n}\sum _{i=1}^{n}\underline{\mathbf{H}}_{\varvec{\eta \xi }}(\mathbf{y}_i)\right] (\hat{\varvec{\xi }} - \underline{\varvec{\xi }}) \nonumber \\&+\left[ \frac{1}{n}\sum _{i=1}^{n}\underline{\mathbf{H}}_{ \varvec{\eta }}(\mathbf{y}_i)\right] (\hat{\varvec{\eta }} - \underline{\varvec{\eta }}) + \mathbf{R}_n, \end{aligned}$$

(24)

in which $\mathbf{g}(\mathbf{y}_i; {\varvec{\xi }}, {\varvec{\eta }}) = \partial \log \pi (\mathbf{y}_{i};{\varvec{\xi }}, {\varvec{\eta }})/\partial {\varvec{\eta }}$, $\mathbf{H}_{\varvec{\eta }}(\mathbf{y}_i; {\varvec{\xi }}, {\varvec{\eta }}) = \partial ^2\log \pi (\mathbf{y}_{i};{\varvec{\xi }}, {\varvec{\eta }})/\partial {\varvec{\eta }}\partial {\varvec{\eta }}{}^\top$, $\mathbf{H}_{\varvec{\eta \xi }}(\mathbf{y}_i; {\varvec{\xi }}, {\varvec{\eta }}) = \partial ^2\log \pi (\mathbf{y}_{i};{\varvec{\xi }}, {\varvec{\eta }})/\partial {\varvec{\eta }}\partial {\varvec{\xi }}{}^\top$, and $\mathbf{R}_n$ denotes the remainder term. As usual, the hat symbol and underline indicate evaluations at the pseudo-ML estimates and the true values of parameters, respectively. If $\frac{1}{n}\sum _{i=1}^{n}\underline{\mathbf{H}}_{ \varvec{\eta }}(\mathbf{y}_i)$ is invertible, then Eq. 24 can be rewritten as

$$\begin{aligned} \sqrt{n}(\hat{\varvec{\eta }} - {\underline{\varvec{\eta }}})= & {} \left[ -\frac{1}{n}\sum _{i=1}^{n} {\underline{\mathbf {H}}}_{\varvec{\eta }}({\mathbf {y}}_i)\right] ^{-1} \left\{ \frac{1}{\sqrt{n}}\sum _{i=1}^{n}{\underline{\mathbf {g}}} (\mathbf {y}_i) \right. \nonumber \\&\quad \left. + \sqrt{\frac{n}{n'}}\left[ \frac{1}{n}\sum _{i=1}^{n}\underline{\mathbf {H}}_{ \varvec{\eta \xi }}(\mathbf {y}_i)\right] \sqrt{n'}(\hat{\varvec{\xi }} - {\underline{\varvec{\xi }}}) + \sqrt{n}{\mathbf {R}}_n\right\} . \end{aligned}$$

(25)

The assumed regularity conditions guarantee that

$$\begin{aligned}&-\frac{1}{n}\sum _{i=1}^{n}{\underline{\mathbf {H}}}_{\varvec{\eta }} (\mathbf {y}_i){}{\mathop {\rightarrow }\limits ^{p}} \underline{\varvec{\mathcal {I}}}_{\varvec{\eta }},\ -\frac{1}{n}\sum _{i=1}^{n}\underline{\mathbf {H}}_{\varvec{\eta \xi }} (\mathbf {y}_i){}{\mathop {\rightarrow }\limits ^{p}} \underline{\varvec{\mathcal {I}}}_{\varvec{\eta \xi }},\ \frac{n}{n'}\rightarrow {\underline{c}},\ {\sqrt{n}} \mathbf {R}_n{}{\mathop {\rightarrow }\limits ^{p}}\mathbf {0},\nonumber \\&\frac{1}{\sqrt{n}}\sum _{i=1}^{n}\underline{\mathbf {g}}(\mathbf {y}_i){} {\mathop {\rightarrow }\limits ^{d}}\mathcal {N}(\mathbf {0}, \underline{\varvec{\mathcal {I}}}_{\varvec{\eta }}),\ \text{ and } \sqrt{n'}(\hat{\varvec{\xi }} - \underline{\varvec{\xi }}){} {\mathop {\rightarrow }\limits ^{d}}\mathcal {N}(\mathbf {0}, \underline{\varvec{\Omega }}_{\varvec{\xi }}). \end{aligned}$$

(26)

Equation 6 is established by combining Eqs. 25 and 26 and applying Slutsky’s lemma.

1.2 A.2 The Asymptotic Covariance Matrix of Residuals

It is straightforward to show that $\mathbf{p} - \underline{\varvec{\pi }}$, $\hat{\varvec{\xi }} - \underline{\varvec{\xi }}$, and $\hat{\varvec{\eta }} - \underline{\varvec{\eta }}$ are jointly asymptotically normal when the model is correctly specified:

$$\begin{aligned} \sqrt{n}\begin{pmatrix} \mathbf{p} - \underline{\varvec{\pi }}\\ \hat{\varvec{\xi }} - \underline{\varvec{\xi }}\\ \hat{\varvec{\eta }} - \underline{\varvec{\eta }} \end{pmatrix}{}{\mathop {\rightarrow }\limits ^{d}}\mathcal{N}\left( \mathbf{0}, \begin{pmatrix} \underline{\varvec{\Gamma }} &{} \underline{\varvec{\Theta }}{}^\top _{21} &{} \underline{\varvec{\Theta }}{}^\top _{31}\\ \underline{\varvec{\Theta }}_{21} &{} \underline{c}\underline{\varvec{\Omega }}_{\varvec{\xi }} &{} \underline{\varvec{\Omega }}{}^\top _{\varvec{\eta \xi }}\\ \underline{\varvec{\Theta }}_{31} &{} \underline{\varvec{\Omega }}_{\varvec{\eta \xi }} &{} \underline{\varvec{\Omega }}_{\varvec{\eta }} &{} \\ \end{pmatrix} \right) , \end{aligned}$$

(27)

in which $\underline{\varvec{\Theta }}_{21} = \mathbf {0}$ because $\hat{\varvec{\xi }}$ and $\mathbf {Y}'$ are independent, and

$$\begin{aligned} {\underline{\varvec{\Theta }}_{31}} = {\underline{\varvec{\mathcal {I}}}}_{\varvec{\eta }}^{-1}{\underline{\varvec{\Delta }}}{}^\top _{\varvec{\eta }}{\underline{\mathbf {D}}}^{-1}\text{ Cov }(\sqrt{n}{} \mathbf {p}) = {\underline{\varvec{\mathcal {I}}}}_{\varvec{\eta }}^{-1}{\underline{\varvec{\Delta }}}{}^\top _{\varvec{\eta }}. \end{aligned}$$

(28)

By the Delta method,

$$\begin{aligned} \mathbf {e} = \begin{pmatrix} \mathbf {I}_{C} &{}{} -{\underline{\varvec{\Delta }}}_{\varvec{\xi }} &{}{} -{\underline{\varvec{\Delta }}}_{\varvec{\eta }}\\ \end{pmatrix} \begin{pmatrix} \mathbf {p} - \underline{\varvec{\pi }}\\ \hat{\varvec{\xi }} - \underline{\varvec{\xi }}\\ \hat{\varvec{\eta }} - \underline{\varvec{\eta }} \end{pmatrix} + o_p(1), \end{aligned}$$

(29)

in which $\mathbf{I}_{C}$ denotes a $C\times C$ identity matrix. Combining Eqs. 27 and 29 yield

$$\begin{aligned} \varvec{\Sigma }({\varvec{\xi }},{\varvec{\eta }},c) = \begin{pmatrix} \mathbf {I}_{C} &{}{} -\varvec{\Delta }_{\varvec{\xi }} &{}{} -\varvec{\Delta }_{\varvec{\eta }}\\ \end{pmatrix} \begin{pmatrix} \varvec{\Gamma } &{}{} \varvec{\Theta }_{21}^\top &{}{} \varvec{\Theta }_{31}^\top \\[1mm] \varvec{\Theta }_{21} &{}{} c{\varvec{\Omega }}_{\varvec{\xi }} &{}{} {\varvec{\Omega }}{}^\top _{\varvec{\eta \xi }}\\[1mm] \varvec{\Theta }_{31} &{}{} {\varvec{\Omega }}_{\varvec{\eta \xi }} &{}{} {\varvec{\Omega }}_{\varvec{\eta }} &{}{} \\[1mm] \end{pmatrix} \begin{pmatrix} \mathbf {I}_{C} \\[1mm] -\varvec{\Delta }{}^\top _{\varvec{\xi }} \\[1mm] -\varvec{\Delta }{}^\top _{\varvec{\eta }}\\ \end{pmatrix}, \end{aligned}$$

(30)

which simplifies to Eq. 14.