Bootstrap-Calibrated Interval Estimates for Latent Variable Scores in Item Response Theory

Abstract

In most item response theory applications, model parameters need to be first calibrated from sample data. Latent variable (LV) scores calculated using estimated parameters are thus subject to sampling error inherited from the calibration stage. In this article, we propose a resampling-based method, namely bootstrap calibration (BC), to reduce the impact of the carryover sampling error on the interval estimates of LV scores. BC modifies the quantile of the plug-in posterior, i.e., the posterior distribution of the LV evaluated at the estimated model parameters, to better match the corresponding quantile of the true posterior, i.e., the posterior distribution evaluated at the true model parameters, over repeated sampling of calibration data. Furthermore, to achieve better coverage of the fixed true LV score, we explore the use of BC in conjunction with Jeffreys’ prior. We investigate the finite-sample performance of BC via Monte Carlo simulations and apply it to two empirical data examples.

Appendix A. Theoretical Details

1.1 A.1. The ML Estimator

Let $\mathbf{f}({\varvec{\xi }})$ be the vector of all response pattern probabilities (with elements $f(\mathbf{y}_i;{\varvec{\xi }})$ defined in Eq. 4), and $\mathbf p$ denote the corresponding observed proportions. Consider a calibration sample of n i.i.d. item response patterns $\mathbf{Y}_1,\dots ,\mathbf{Y}_n$, each of which is generated from the unidimensional IRT model characterized by $\mathbf{f}({\varvec{\xi }_0})$. In some neighborhood of $\varvec{\xi }_0$, denoted $U_0$, suppose that standard regularity conditions (Birch, 1964; Bishop et al., 1975; Cox, 1984) hold, and that $\mathbf{f}({\varvec{\xi }})$ has continuous fourth partial derivatives, then there exists a four-time continuously differentiable function ${\varvec{\xi }}(\cdot )$ such that ${\varvec{\xi }}_0 = {\varvec{\xi }}(\mathbf{f}(\varvec{\xi }_0))$ and the ML estimator $\hat{\varvec{\xi }}={\varvec{\xi }}(\mathbf{p})$ for all admissible probability vector $\mathbf{p}\in N_0$ where $N_0$ is some neighborhood of $\mathbf{f}({\varvec{\xi }_0})$. As n tends to infinity, $\mathbf p$ concentrates on $N_0$ exponentially fast: It follows from Hoeffding’s (1963) inequality that

$$\begin{aligned} P_{\varvec{\xi }_0}^\mathbf{Y}\{\mathbf{p}\notin N_0\}\le B\exp (-cn) \end{aligned}$$

(17)

for some positive constants B and c.

The following equation, which resembles Beran’s (1990) Assumption A(4), can be verified by a Taylor series expansion argument similar to Lehmann and Casella (1998, p. 430):

$$\begin{aligned} E_{\varvec{\xi }_0}^\mathbf{Y}\left[ \left( \varvec{\xi }(\mathbf{p}) - \varvec{\xi }_0\right) ^\mathbf{r}{\mathbb I}\{\mathbf{p}\in N_0\}\right] = \sum ^2_{s = \lfloor (|\mathbf{r}| + 1)/2\rfloor }n^{-s} b_{\mathbf{r},s}({\varvec{\xi }}_0)+ o(n^{-2}),\quad |\mathbf{r}|=1,\dots ,4. \end{aligned}$$

(18)

In Eq. 18, the Q-tuple of nonnegative integers $\mathbf{r} = (r_1,\dots ,r_q){}^\top$ serves as a multi-index such that $\varvec{\xi }^\mathbf{r} = \xi _1^{r_1}\cdots \xi _q^{r_q}$ for any q-dimensional vector $\varvec{\xi } = (\xi _1,\dots ,\xi _q){}^\top$, and $|\mathbf{r}| = r_1+\cdots +r_q$. ${\mathbb I}\{\cdot \}$ denotes the indicator function. The functions $b_{\mathbf{r},s}({\varvec{\xi }}_0)$ for all $\mathbf r$ and s are related to the partial derivatives of $\varvec{\xi }(\cdot )$ and the moments of $\mathbf p$ up to the fourth order. Equation 18 indicates that the first and second moments of the truncated ML estimator are of order $O(n^{-1})$, and that the third and fourth moments are of order $O(n^{-2})$.

1.2 A.2. The Plug-in Method

Let the cdf $G(\cdot ;{\varvec{\xi }})$ be continuous for all $\varvec{\xi }$ in the closure of the parameter space. In particular, we assume that $G(\theta ; \varvec{\xi })$ is strictly monotonic in $\theta$ and four-time continuously differentiable with respect to both $\theta$ and $\varvec{\xi }$ for all $\theta \in \mathbb {R}$ and $\varvec{\xi }\in U_0$. These assumptions imply that $C(\alpha ; {\varvec{\xi }}, {\varvec{\xi }}_0)$ = $G(G^{-1}(\alpha ;\varvec{\xi }); \varvec{\xi }_0)$ is always defined and has continuous fourth partial derivatives with respect to $\varvec{\xi }$ for any $\varvec{\xi }\in U_0$. Then, $C(\alpha ; {\varvec{\xi }}, \varvec{\xi }_0)$ has the following Taylor series expansion at $\varvec{\xi }=\varvec{\xi }_0$:

$$\begin{aligned} C(\alpha ; \varvec{\xi }, \varvec{\xi }_0) = \alpha + \sum _{|\mathbf{r}|=1}^3\frac{\partial ^{|\mathbf{r}|}C}{\partial \varvec{\xi }^\mathbf{r}}(\alpha , \varvec{\xi }_0, {\varvec{\xi }}_0)\frac{(\varvec{\xi } - \varvec{\xi }_0)^\mathbf{r}}{\mathbf{r}!} + \sum _{|\mathbf{r}|=4}\frac{\partial ^4C}{\partial \varvec{\xi }^\mathbf{r}}(\alpha , \bar{\varvec{\xi }}, \varvec{\xi }_0)\frac{(\varvec{\xi } - \varvec{\xi }_0)^\mathbf{r}}{\mathbf{r}!} \end{aligned}$$

(19)

in which $\mathbf{r}! = r_1!\cdots r_Q!$, and $\bar{\varvec{\xi }}$ lies between $\varvec{\xi }$ and $\varvec{\xi }_0$. Note that $C(\alpha ; \varvec{\xi }_0)$ = $E_{\varvec{\xi }_0}^\mathbf{Y}[C(\alpha ; \hat{\varvec{\xi }}, \varvec{\xi }_0)]$ is essentially $E_{\varvec{\xi }_0}^\mathbf{Y}\left[ C(\alpha ; \hat{\varvec{\xi }}, \varvec{\xi }_0){\mathbb I}\{\mathbf{p}\in N_0\}\right]$ plus an exponentially small term because of Eq. 17 and the boundedness of $C(\alpha ; \varvec{\xi }, \varvec{\xi }_0)$. It follows from Eqs. 18 and 19 that

$$\begin{aligned} C(\alpha ; {\varvec{\xi }_0}) = \alpha + n^{-1}d(\alpha ,\varvec{\xi }_0) + O(n^{-2}), \end{aligned}$$

(20)

in which

$$\begin{aligned} d(\alpha ,\varvec{\xi }_0) = \sum _{|\mathbf{r}|=1}^2\frac{\partial ^{|\mathbf{r}|}C}{\partial \varvec{\xi }^\mathbf{r}}(\alpha ,\varvec{\xi }_0,\varvec{\xi }_0)\frac{b_{\mathbf{r},1}({\varvec{\xi }}_0)}{\mathbf{r}!}. \end{aligned}$$

(21)

1.3 A.3. Predictive Calibration

Analogously, expanding $C(C^{-1}(\alpha ; {\varvec{\xi }_0}); \varvec{\xi }, \varvec{\xi }_0)$ at $\varvec{\xi }=\varvec{\xi }_0$ and taking expectation yield

$$\begin{aligned} C^{-1}(\alpha ; {\varvec{\xi }_0}) = \alpha - n^{-1}d(\alpha ,\varvec{\xi }_0) + O(n^{-2}). \end{aligned}$$

(22)

The assumptions we made imply that Eq. 22 still holds with the same $d(\cdot ,\cdot )$ if we replace $\varvec{\xi }_0$ by $\varvec{\xi }$ in some neighborhood $V_0\subset U_0$,⁸ and $\sup _{{\varvec{\xi }}\in V_0}|C^{-1}(\alpha ; {\varvec{\xi }}) - \alpha + n^{-1}d(\alpha ,\varvec{\xi })|$ = $O(n^{-2})$. Plugging in the expansion of $d(\alpha ,\varvec{\xi })$ at $\varvec{\xi }={\varvec{\xi }}_0$, we obtain

(23)

in which $\breve{\varvec{\xi }}$ falls between $\varvec{\xi }$ and $\varvec{\xi }_0$.

We now expand $C( C^{-1}(\alpha ; \varvec{\xi }); {\varvec{\xi }}, \varvec{\xi }_0)$ with respect to the first argument at $C^{-1}(\alpha ; {\varvec{\xi }}_0)$ and then further expand $\frac{\partial C}{\partial \alpha }( C^{-1}(\alpha ; \varvec{\xi }_0), {\varvec{\xi }}, \varvec{\xi }_0)$ with respect to the second argument at $\varvec{\xi }_0$:

$$\begin{aligned}&C( C^{-1}(\alpha ; \varvec{\xi }); \varvec{\xi }, {\varvec{\xi }}_0)\nonumber \\&= C( C^{-1}(\alpha ; {\varvec{\xi }}_0); {\varvec{\xi }}, {\varvec{\xi }}_0) + \frac{\partial C}{\partial \alpha }( C^{-1}(\alpha ; \varvec{\xi }_0), {\varvec{\xi }}, {\varvec{\xi }}_0)[ C^{-1}(\alpha ; {\varvec{\xi }}) - C^{-1}(\alpha ; \varvec{\xi }_0)]\nonumber \\&+ \frac{1}{2}\frac{\partial ^2 C}{\partial \alpha ^2}(\bar{\alpha }, {\varvec{\xi }}, {\varvec{\xi }}_0)[ C^{-1}(\alpha ; {\varvec{\xi }}) - C^{-1}(\alpha ; \varvec{\xi }_0)]^2\nonumber \\&= C( C^{-1}(\alpha ; {\varvec{\xi }}_0); {\varvec{\xi }}, {\varvec{\xi }}_0) + \biggr \{\frac{\partial C}{\partial \alpha }( C^{-1}(\alpha ; {\varvec{\xi }_0}), {\varvec{\xi }}_0, {\varvec{\xi }}_0) + ({\varvec{\xi }} - {\varvec{\xi }}_0){}^\top \left[ \frac{\partial ^2 C}{\partial \alpha \partial \varvec{\xi }{}}( C^{-1}(\alpha ; {\varvec{\xi }_0}), \tilde{\varvec{\xi }}, {\varvec{\xi }_0})\right] \biggr \}\nonumber \\&\cdot [ C^{-1}(\alpha ; {\varvec{\xi }}) - C^{-1}(\alpha ; \varvec{\xi }_0)] + \frac{1}{2}\frac{\partial ^2 C}{\partial \alpha ^2}(\bar{\alpha }, {\varvec{\xi }}, {\varvec{\xi }}_0)[ C^{-1}(\alpha ; {\varvec{\xi }}) - C^{-1}(\alpha ; \varvec{\xi }_0)]^2, \end{aligned}$$

(24)

in which $\bar{\alpha }$ falls between $C^{-1}(\alpha ; {\varvec{\xi }}_0)$ and $C^{-1}(\alpha ; {\varvec{\xi }})$, and $\tilde{\varvec{\xi }}$ fall between $\varvec{\xi }_0$ and $\varvec{\xi }$. By combining Eqs. 23 with 24 and taking expectation, we conclude that the calibrated predictive coverage matches $\alpha$ up to an $O(n^{-2})$ error term.