Variable selection in model-based clustering and discriminant analysis with a regularization approach

Abstract

Several methods for variable selection have been proposed in model-based clustering and classification. These make use of backward or forward procedures to define the roles of the variables. Unfortunately, such stepwise procedures are slow and the resulting algorithms inefficient when analyzing large data sets with many variables. In this paper, we propose an alternative regularization approach for variable selection in model-based clustering and classification. In our approach the variables are first ranked using a lasso-like procedure in order to avoid slow stepwise algorithms. Thus, the variable selection methodology of Maugis et al. (Comput Stat Data Anal 53:3872–3882, 2000b) can be efficiently applied to high-dimensional data sets.

Procedures to maximize penalized empirical contrasts

1.1 The model-based clustering case

The EM algorithm for maximizing criterion (2) is as follows (Zhou et al. 2009). The penalized complete loglikelihood of the centered data set $\bar{\mathbf {y}} = \big (\bar{\mathbf {y}}_1, \ldots , \bar{\mathbf {y}}_n \big )'$ is given by

$$\begin{aligned} L_{\text {c},(\lambda , \rho )} (\bar{\mathbf {y}}, \mathbf {z}, \alpha )= & {} \sum _{i =1}^n \sum _{k = 1}^K z_{ik} \big [ \ln (\pi _k) + \ln \phi (\bar{\mathbf {y}}_i \mid \mu _k, \varSigma _k)\big ] - \lambda \sum ^K_{k = 1} \left\| \mu _k\right\| _1 \nonumber \\&- \rho \sum _{k = 1}^K \left\| \varTheta _k\right\| _1 , \end{aligned}$$

(5)

where $\varTheta _k=\varSigma _k^{-1}$ denotes the precision matrix of the kth mixture component. The EM algorithm of Zhou et al. (2009) maximizes at each iteration the conditional expectation of (5) given $\bar{\mathbf {y}}$ and a current parameter vector $\alpha ^{(s)}$: ${\mathbb {E}}\Big [L_{\text {c},(\lambda , \rho )}\big (\bar{\mathbf {y}}, \mathbf {z}, \alpha \big ) \mid \bar{\mathbf {y}}, \alpha ^{(s)}\Big ].$ The following two steps are repeated from an initial $\alpha ^{(0)}$ until convergence. At the sth iteration of the EM algorithm:

• E-step: The conditional probabilities $t^{(s)}_{ik}$ that the ith observation belongs to the kth cluster are computed for $i=1,\ldots ,n$ and $k=1,\ldots ,K$,

  $$\begin{aligned} t^{(s)}_{ik} = \mathbb {P}\big (z_{ik} = 1 \mid \bar{\mathbf {y}}, \alpha ^{(s)}\big ) = \frac{\pi _k^{(s)} \phi \Big ( \bar{\mathbf {y}}_i \mid \mu ^{(s)}_k,\varSigma ^{(s)}_k \Big )}{ \sum _{k' = 1}^K \pi _{k'}^{(s)} \phi \Big (\bar{\mathbf {y}}_i \mid \mu ^{(s)}_{k'}, \varSigma ^{(s)}_{k'} \Big )}. \end{aligned}$$

• M-step : This step consists of maximizing the expected complete log-likelihood derived from the E-step. It leads to the following mixture parameter updates:

  • The updated proportions are $\pi ^{(s+1)}_k = \frac{1}{n} \sum _{i = 1}^n t^{(s)}_{ik}$ for $k=1,\ldots ,K$.

  • Compute the updated means $\mu ^{(s+1)}_1, \ldots , \mu ^{(s+1)}_K$ using formulas (14) et (15) of Zhou et al. (2009): the jth coordinate of $\mu ^{(s+1)}_k$ is the solution of the following equations:

    $$\begin{aligned} \text {if} \quad \left| \sum _{i = 1}^n t^{(s)}_{ik} \left[ \underset{v \ne j}{\sum _{v = 1}^p} \left( \bar{\mathbf {y}}_{ij} - \mu ^{(s)}_{k v}\right) \varTheta ^{(s)}_{k,v j} + \bar{\mathbf {y}}_{ij} \varTheta ^{(s)}_{k,jj}\right] \right| \le \lambda , \quad \text {then} \quad \mu ^{(s+1)}_{kj} = 0, \end{aligned}$$

    otherwise:

    $$\begin{aligned}&\left[ \sum _{i = 1}^n t^{(s)}_{i k}\right] \mu ^{(s+1)}_{kj} \varTheta ^{(s)}_{k,jj} + \ \lambda \ \text {sign}\left( \mu ^{(s+1)}_{kj}\right) \\&\quad = \sum _{i = 1}^n t_{ik}^{(s)} \sum _{v = 1}^p \bar{y}_{iv} \varTheta ^{(s)}_{k,v j} \\&\qquad - \left[ \sum _{i = 1}^n t^{(s)}_{i k}\right] \left[ \left( \sum _{v = 1}^p \mu ^{(s)}_{kv} \varTheta ^{(s)}_{k,vj}\right) - \mu ^{(s)}_{kj} \varTheta ^{(s)}_{k, jj}\right] . \end{aligned}$$

  • For all $k=1,\ldots ,K$, the covariance matrix $\varSigma _k^{(s+1)}$ is obtained via the precision matrix $\varTheta _k^{(s+1)}$. The glasso algorithm (available in the R package glasso of Friedman et al. 2014) is used to solve the following minimization problem on the set of symmetric positive definite matrices (denoted $\varTheta \succ 0$):

    $$\begin{aligned} \underset{\varTheta \succ 0}{\mathop {\mathrm {arg\,min}}}\left\{ -\ln \text {det}\left( \varTheta \right) + \text {trace}\left( S_k^{(s+1)} \varTheta \right) + \rho _k^{(s+1)} \Vert \varTheta \Vert _1\right\} , \end{aligned}$$

    where $\rho _k^{(s+1)} = 2 \rho \left( \sum _{i = 1}^n t^{(s)}_{ik}\right) ^{-1}$ and

    $$\begin{aligned} S^{(s+1)}_k = \frac{\sum _{i = 1}^n t^{(s)}_{ik}(\bar{\mathbf {y}}_i - \mu ^{(s+1)}_k) (\bar{\mathbf {y}}_i - \mu ^{(s+1)}_k)^\top }{\sum _{i = 1}^n t^{(s)}_{ik}}. \end{aligned}$$

1.2 The classification case

The maximization of the regularized criterion (4) at $\mu _1, \ldots , \mu _K$ and $\varTheta _1, \ldots , \varTheta _K$ is achieved using an algorithm similar to the one presented in Sect. A.1 when the labels $z_i$ are known.

The jth coordinate of the mean vector $\mu _k$ is the solution of the following equations:

$$\begin{aligned} \text {if} \quad \left| \sum _{i = 1}^n \mathbb {1}_{\{z_i = k\}} \left[ \underset{v \ne j}{\sum _{v = 1}^p} \left( \bar{\mathbf {y}}_{ij} - \mu _{k v}\right) \varTheta _{k,v j} + \bar{\mathbf {y}}_{ij} \varTheta _{k,jj}\right] \right| \le \lambda , \quad \text {then} \quad \mu _{kj} = 0, \end{aligned}$$

otherwise:

$$\begin{aligned}&\left[ \sum _{i = 1}^n \mathbb {1}_{\{z_i = k\}} \right] \mu _{kj} \varTheta _{k,jj} + \lambda \ \text {sign}\left( \mu _{kj}\right) \\&\quad = \sum _{i = 1}^n \mathbb {1}_{\{z_i = k\}} \sum _{v = 1}^p \bar{y}_{iv} \varTheta _{k,v j} \\&\qquad - \left[ \sum _{i = 1}^n \mathbb {1}_{\{z_i = k\}} \right] \left[ \left( \sum _{v = 1}^p \mu _{kv} \varTheta _{k,v j}\right) - \mu _{kj} \varTheta _{k, jj}\right] . \end{aligned}$$

To estimate the sparse precision matrices $\varTheta _1, \ldots , \varTheta _K$ from the data set $\mathbf {y}$ and the labels $\mathbf {z}$, we use the glasso algorithm to solve the following minimization problem on the set of symmetric positive definite matrices

$$\begin{aligned} \widehat{\varTheta }_k = \underset{\varTheta \succ 0}{\mathop {\mathrm {arg\,min}}}\Big \{ -\ln \text {det}\big (\varTheta \big ) + \text {trace}\big (S_k \varTheta \big ) + \rho _k \Vert \varTheta \Vert _1\Big \}, \end{aligned}$$

(6)

for each $k = 1, \ldots , K$. The $\ell _1$ regularization parameter in (6) is given by $\rho _k = 2 \rho \left( \sum _{i = 1}^n \mathbb {1}_{\{z_{i} = k\}}\right) ^{-1}$ and the empirical covariance matrix $S_k$ by

$$\begin{aligned} S_k = \frac{\sum _{i = 1}^n \mathbb {1}_{\{z_i=k\}} (\bar{\mathbf {y}}_i -\mu _k) (\bar{\mathbf {y}}_i - \mu _k)^\top }{ \sum _{i =1}^n\mathbb {1}_{\{z_i=k\}} }. \end{aligned}$$

Then, coordinate descent maximization in $(\mu _1, \ldots , \mu _K)$ and $(\varTheta _1, \ldots , \varTheta _K)$ is run until convergence.