Two-Step Estimation of Models Between Latent Classes and External Variables

Abstract

We consider models which combine latent class measurement models for categorical latent variables with structural regression models for the relationships between the latent classes and observed explanatory and response variables. We propose a two-step method of estimating such models. In its first step, the measurement model is estimated alone, and in the second step the parameters of this measurement model are held fixed when the structural model is estimated. Simulation studies and applied examples suggest that the two-step method is an attractive alternative to existing one-step and three-step methods. We derive estimated standard errors for the two-step estimates of the structural model which account for the uncertainty from both steps of the estimation, and show how the method can be implemented in existing software for latent variable modelling.

Appendix: Score Functions and Information Matrices for Latent Class Models

Consider first in general terms a model which involves a latent class variable X with a total of C latent classes (here X may also represent all combinations of the classes of several latent class variables). Suppose that the model depends on parameters $\varvec{\theta }$ (in this Appendix, we take all vectors to be column vectors, the opposite of the practice in Sect. 2 where they were row vectors for simplicity of notation). The log likelihood contribution for a single unit i is then $l_{i}= \log L_{i} = \log \sum _{c=1}^{C} L_{ic}$, where $L_{ic}=\exp (l_{ic})$ is the term in $L_{i}$ which refers to latent class $c=1,\ldots ,C$. The contribution of unit i to the score function is then $\mathbf {u}_{i}=\partial l_{i}/\partial \varvec{\theta }=\mathbf {h}_{i}/L_{i}$ where $\mathbf {h}_{i} = \partial L_{i}/\partial \varvec{\theta }=\sum _{c} L_{ic} \mathbf {u}_{ic}$ and $\mathbf {u}_{ic}=\partial l_{ic}/\partial \varvec{\theta }$, and the contribution to the observed information matrix is

$$\begin{aligned} \mathbf {J}_{i}=-\frac{\partial ^{2} l_{i}}{\partial \varvec{\theta } \partial \varvec{\theta }'} = -\frac{1}{L_{i}^{2}} \, \left\{ L_{i} \, \left[ \sum _{c} L_{ic} \left( \mathbf {u}_{ic}\mathbf {u}_{ic}'-\mathbf {J}_{ic} \right) \right] - \mathbf {h}_{i}\mathbf {h}_{i}' \right\} , \end{aligned}$$

where $\mathbf {J}_{ic}=-\,\partial ^{2} l_{ic}/(\partial \varvec{\theta } \partial \varvec{\theta }')$.

Suppose that observations for different units $i=1,\ldots ,n$ are independent. Point estimation of $\varvec{\theta }$ is easiest with the EM algorithm (Dempster, Laird, & Rubin, 1977). For this, let $l_{i}^{*}$ be the same expression as $l_{ic}$ but now regarded as a function of c. At the E step of the $(t+1)$th iteration of EM, we calculate $Q^{(t+1)}=\sum _{i} \text {E}[l^{*}_{i}|\mathbf {D},\varvec{\theta }^{(t)}]= \sum _{i}\left( \sum _{c} \pi _{ic}^{(t)} l_{ic}\right)$ where $\pi _{ic}^{(t)}=p(X_{i}=c|\mathbf {D},\varvec{\theta }^{(t)})$, $\mathbf {D}$ denotes all the observed data, and $\varvec{\theta }^{(t)}$ is the estimate of $\varvec{\theta }$ from the tth iteration. At the M step, $Q^{(t+1)}$ is maximized with respect to $\varvec{\theta }$ to produce an updated estimate $\varvec{\theta }^{(t+1)}$. This is relatively straightforward because $\partial Q^{(t+1)}/\partial \varvec{\theta }= \sum _{i}\sum _{c} \left( \pi _{ic}^{(t)} \mathbf {u}_{ic}\right)$ and $-\,\partial ^{2} Q^{(t+1)}/\partial \varvec{\theta }\partial \varvec{\theta }'= \sum _{i}\sum _{c} \left( \pi _{ic}^{(t)} \mathbf {J}_{ic}\right)$, i.e. these are the score function and observed information matrix for a model where X is known, fitted to pseudo-data of $n\times C$ observations with fractional weights $\pi _{ic}^{(t)}$.

The information matrix $\varvec{\mathcal {I}}$ for the model can be estimated by $n^{-1}\sum _{i} \mathbf {J}_{i}$ or $n^{-1}\sum _{i} \mathbf {u}_{i}\mathbf {u}_{i}'$. Together with $\mathbf {u}_{i}$, these could also be used to implement other estimation algorithms than EM. When evaluated at the final estimate of $\varvec{\theta }$, they give estimates of the $\varvec{\mathcal{I}}_{22}$ and $\varvec{\mathcal {I}}_{12}$ which are needed for the two-step variance matrix (5). An estimate of the $\mathbf {\Sigma }_{11}$ that is also needed there is obtained similarly from the estimated information matrix for step 1 latent class model.

What then remains to be done for any specific model is to evaluate $l_{ic}$, $\mathbf {u}_{ic}$ and (if used) $\mathbf {J}_{ic}$ for it. As an example, consider the model with covariates $\mathbf {Z}_{p}$, one latent class variable X, and a response variable $Z_{o}$ which is considered in Sect. 2.1. The $L_{i}$ for it is given by (2) with $p(\mathbf {Z}_{pi})$ omitted. Then $l_{ic}=\log p(X_{i}=c|\mathbf {Z}_{pi}) + \log p(Z_{oi}|\mathbf {Z}_{pi},X_{i}=c) + \sum _{k} \log p(Y_{ik}|X_{i}=c)$ $\equiv l^{(x)}_{ic} + l^{(z)}_{ic} + \sum _{k} l^{(yk)}_{ic}$. If the parameters for the different components of the model are distinct, which should be the case for most sensible models, we only need to consider the separate derivatives of the terms in this sum. Suppose that $X_{i}$ given $\mathbf {Z}_{pi}$ is given by the multinomial logistic model (3), writing it now as $p(X_{i}=c\vert \mathbf Z _{pi})\equiv \pi ^{(x)}_{ic}= \exp (\varvec{\alpha }_{c}'\mathbf {Z}^{*}_{i})/ \sum \limits _{s=1}^{C}\exp (\varvec{\alpha }_{s}'\mathbf {Z}^{*}_{i})$ for $c=1,\ldots ,C$, with $\varvec{\alpha }_{1}=\mathbf {0}$, $\varvec{\alpha }_{c}=(\beta _{0c},\varvec{\beta }_{c}')'$ for $c\ne 1$, and $\mathbf {Z}_{i}^{*}=(1,\mathbf {Z}_{pi}')'$. Then $l^{(x)}_{ic}=\log \pi ^{(x)}_{ic}$, $\partial l^{(x)}_{ic}/\partial \varvec{\alpha }_{r}=(I(r=c)-\pi ^{(x)}_{ir})\mathbf {Z}_{i}^{*}$, and $\partial ^{2} l^{(x)}_{ic}/\partial \varvec{\alpha }_{r}\partial \varvec{\alpha }'_{s}=-\,(I(s=r)\pi ^{(x)}_{ir}-\pi ^{(x)}_{ir}\pi ^{(x)}_{is})\mathbf {Z}_{i}^{*}\mathbf {Z}_{i}^{*\prime }$ for $r,s=2,\ldots ,C$. The measurement models for the items $Y_{ik}$ can also be formulated as multinomial logistic models, by writing them as $p(Y_{ik}=r|X_{i}=c)\equiv \pi ^{(yk)}_{icr}= \exp (\varvec{\gamma }_{kr}'\mathbf {X}_{ic}^{*})/ \sum _{s=1}^{R_{k}}\exp (\varvec{\gamma }_{ks}'\mathbf {X}_{ic}^{*})$ for $r=1,\ldots ,R_{k}$, with $\mathbf {X}_{ic}^{*}=(I(c=1),\ldots ,I(c=C))'$ and $\varvec{\gamma }_{k1}=\mathbf {0}$. Then $l^{(yk)}_{ic}= \sum _{r=1}^{R_{k}} I(Y_{ik}=r)\log \pi ^{(yk)}_{icr}$, and if the parameters for different items k are also distinct, the terms in $\sum _{k} l^{(yk)}_{ic}$ can be differentiated separately. Their derivatives are $\partial l^{(yk)}_{ic}/\partial \varvec{\gamma }_{r}= (I(Y_{ik}=r)-\pi ^{(yk)}_{icr})\mathbf {X}_{ic}^{*}$ and $\partial ^{2} l^{(yk)}_{ic}/\partial \varvec{\gamma }_{r}\partial \varvec{\gamma }'_{s}=-\,(I(s=r)\pi ^{(yk)}_{icr}-\pi ^{(yk)}_{icr}\pi ^{(yk)}_{ics})\mathbf {X}_{ic}^{*}\mathbf {X}_{ic}^{*\prime }$ for $r,s=2,\ldots ,R_{k}$. For $l^{(z)}_{ic}$, suppose, for example, that $Z_{oi}$ is normally distributed with mean $\mu _{i}=\varvec{\delta }'\mathbf {Z}_{i}^{**}$ and variance $\tau ^{-1}$, where $\mathbf {Z}_{i}^{**}=(\mathbf {X}_{ic}',\mathbf {Z}_{pi}')'$. Defining $e_{i}=Z_{oi}-\mu _{i}$, then $l^{(z)}_{ic}=(\log \tau -\tau e_{i}^{2})/2$, $\partial l^{(z)}_{ic}/\partial \varvec{\delta }=\tau e_{i}\mathbf {Z}_{i}^{**}$, $\partial l^{(z)}_{ic}/\partial \tau =(1/\tau -e_{i}^{2})/2$, $\partial ^{2} l^{(z)}_{ic}/\partial \varvec{\delta } \partial \varvec{\delta }'=-\tau (\mathbf {Z}_{i}^{**})(\mathbf {Z}_{i}^{**})'$, $\partial ^{2} l^{(z)}_{ic}/\partial ^{2} \tau =-1/(2\tau ^{2})$, and $\partial ^{2} l^{(z)}_{ic}/\partial \varvec{\delta }\partial \tau =e_{i}\mathbf {Z}_{i}^{**}$. The formulas for the situations considered in our simulations and examples are obtained from these results by setting $\mathbf {Z}_{i}^{*}=1$ for the case with no $Z_{pi}$, and omitting $l^{(z)}_{ic}$ for the case with no $Z_{oi}$. Finally, doing both of these things gives the formulas for the basic latent class model which is estimated in step 1 of the two-step method.