A robust approach to model-based classification based on trimming and constraints

Abstract

In a standard classification framework a set of trustworthy learning data are employed to build a decision rule, with the final aim of classifying unlabelled units belonging to the test set. Therefore, unreliable labelled observations, namely outliers and data with incorrect labels, can strongly undermine the classifier performance, especially if the training size is small. The present work introduces a robust modification to the Model-Based Classification framework, employing impartial trimming and constraints on the ratio between the maximum and the minimum eigenvalue of the group scatter matrices. The proposed method effectively handles noise presence in both response and exploratory variables, providing reliable classification even when dealing with contaminated datasets. A robust information criterion is proposed for model selection. Experiments on real and simulated data, artificially adulterated, are provided to underline the benefits of the proposed method.

Appendices

Appendix A

Proof of Proposition 1

Considering the random variable ${\mathcal {Z}}_{mg}$ corresponding to $z_{mg}$, the E-step on the $(k+1)$th iteration requires the calculation of the conditional expectation of ${\mathcal {Z}}_{mg}$ given ${\mathbf {y}}_m$:

$$\begin{aligned} \begin{aligned} E _{\hat{\varvec{\theta }}^{(k)}}({\mathcal {Z}}_{mg}|{\mathbf {y}}_m)&={\mathbb {P}}\left( {\mathcal {Z}}_{mg}=1|{\mathbf {y}}_m;{\hat{\theta }}^{(k)}\right) \\&=\frac{{\mathbb {P}}\left( {\mathbf {y}}_m|{\mathcal {Z}}_{mg}=1;{\hat{\theta }}^{(k)}\right) {\mathbb {P}}\left( {\mathcal {Z}}_{mg}=1;{\hat{\theta }}^{(k)}\right) }{\sum _{j=1}^G {\mathbb {P}}\left( {\mathbf {y}}_m|{\mathcal {Z}}_{mj}=1;{\hat{\theta }}^{(k)}\right) {\mathbb {P}}\left( {\mathcal {Z}}_{mj}=1;{\hat{\theta }}^{(k)}\right) }\\&=\frac{{\hat{\tau }}^{(k)}_g \phi \left( {\mathbf {y}}_m; \hat{\varvec{\mu }}^{(k)}_g, \hat{\varvec{\varSigma }}^{(k)}_g \right) }{\sum _{j=1}^G{\hat{\tau }}_j^{(k)} \phi \left( {\mathbf {y}}_m; \hat{\varvec{\mu }}^{(k)}_j, \hat{\varvec{\varSigma }}^{(k)}_j\right) }\\&={\hat{z}}_{mg}^{(k+1)} \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, g=1,\ldots , G; \,\,\,\, m=1,\ldots , M. \end{aligned} \end{aligned}$$

(23)

Therefore, the Q function, to be maximized with respect to $\varvec{\theta }$ in the M-step, is given by

$$\begin{aligned} \begin{aligned} Q(\varvec{\theta };\hat{\varvec{\theta }}^{(k)})&= \sum _{n=1}^N \zeta ({\mathbf {x}}_n) \sum _{g=1}^G l_{ng} \log {\left[ \tau _g \phi ({\mathbf {x}}_n; \varvec{\mu }_g, \varvec{\varSigma }_g)\right] } \\&\quad +\, \sum _{m=1}^M \varphi ({\mathbf {y}}_m) \sum _{g=1}^G \hat{z}_{mg}^{(k+1)} \log {\left[ \tau _g \phi ({\mathbf {y}}_m; \varvec{\mu }_g, \varvec{\varSigma }_g)\right] .} \end{aligned} \end{aligned}$$

(24)

The maximization of (24) according to the mixture proportion $\tau _g$, $\sum _{j=1}^G\tau _j=1$ is solved considering the Lagrangian ${\mathcal {L}}(\varvec{\theta }, \kappa )$:

$$\begin{aligned} {\mathcal {L}}(\varvec{\theta }, \kappa )=Q\left( \varvec{\theta };\hat{\varvec{\theta }}^{(k)}\right) -\kappa \left( \sum _{j=1}^G\tau _j-1\right) \end{aligned}$$

(25)

with $\kappa$ the Lagrangian coefficient. The partial derivative of (25) with respect to $\tau _g$ has the form:

$$\begin{aligned} \frac{\partial }{\partial \tau _g}{\mathcal {L}}(\varvec{\theta }, \kappa )=\frac{\sum _{n=1}^N \zeta ({\mathbf {x}}_n)l_{ng}}{\tau _g}+ \frac{\sum _{m=1}^M \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}}{\tau _g}-\kappa \end{aligned}$$

(26)

and setting (26) equal to 0 for all $g=1,\ldots , G$ we obtain:

$$\begin{aligned} \sum _{n=1}^N \zeta ({\mathbf {x}}_n)l_{ng}+ \sum _{m=1}^M \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}-\kappa \tau _g=0. \end{aligned}$$

(27)

Summing (27) over g, $g=1,\ldots , G$, provides the value of $\kappa =\lceil N(1-\alpha _{l})\rceil +M(1-\alpha _{u})\rceil$ and substituting it in the previous expression yields the ML estimate for $\tau _g$:

$$\begin{aligned} {\hat{\tau }}_g^{(k+1)}=\frac{\sum _{n=1}^N \zeta ({\mathbf {x}}_n)l_{ng}+ \sum _{m=1}^M \varphi ({\mathbf {y}}_m){\hat{z}}_{mg}^{(k+1)}}{\lceil N(1-\alpha _{l})\rceil +\lceil M(1-\alpha _{u})\rceil }\,\,\,\,\, g=1,\ldots , G. \end{aligned}$$

(28)

The partial derivative of (24) with respect to the mean vector $\varvec{\mu }_g$ reads:

$$\begin{aligned} \begin{aligned} \frac{\partial }{\partial \varvec{\mu }_g}Q\left( \varvec{\theta };\varvec{\theta }^{(k)}\right)&= -\varvec{\varSigma }_g^{-1}\left[ \sum _{n=1}^N \zeta ({\mathbf {x}}_n)l_{ng}\left( {\mathbf {x}}_n-\varvec{\mu }_g\right) +\sum _{m=1}^M \varphi ({\mathbf {y}}_m){\hat{z}}_{mg}^{(k+1)}\left( {\mathbf {y}}_m-\varvec{\mu }_g\right) \right] \\&=-\varvec{\varSigma }_g^{-1}\left[ \sum _{n=1}^N \zeta ({\mathbf {x}}_n)l_{ng}{\mathbf {x}}_n + \sum _{m=1}^M \varphi ({\mathbf {y}}_m){\hat{z}}_{mg}^{(k+1)}{\mathbf {y}}_m +\right. \\&\left. \quad -\varvec{\mu }_g\left( \sum _{n=1}^N \zeta ({\mathbf {x}}_n)l_{ng} + \sum _{m=1}^M \varphi ({\mathbf {y}}_m){\hat{z}}_{mg}^{(k+1)} \right) \right] . \end{aligned} \end{aligned}$$

(29)

Equating (29) to 0 and rearranging terms provides the ML estimate of $\varvec{\mu }_g$:

$$\begin{aligned} \hat{\varvec{\mu }}_g^{(k+1)}=\frac{\sum _{n=1}^N \zeta ({\mathbf {x}}_n)l_{ng}{\mathbf {x}}_n+\sum _{m=1}^M\varphi ({\mathbf {y}}_m){\hat{z}}_{mg}^{(k+1)}{\mathbf {y}}_m}{\sum _{n=1}^N\zeta ({\mathbf {x}}_n)l_{ng}+\sum _{m=1}^M\varphi ({\mathbf {y}}_m){\hat{z}}_{mg}^{(k+1)}}\,\,\,\,\, g=1,\ldots , G. \end{aligned}$$

(30)

Discarding quantities that do not depend on $\varvec{\varSigma }_g$, we can rewrite (24) as follows:

$$\begin{aligned}&\sum _{n=1}^{N} \sum _{g=1}^{G} \zeta ({\mathbf {x}}_n)l_{ng}\left( {\mathbf {x}}_{n}\right) \left[ -\log \left| \varvec{\varSigma }_{g}\right| ^{1 / 2}-\frac{1}{2}\left( {\mathbf {x}}_{n}-\varvec{\mu }_{g}\right) ^{\prime } \varvec{\varSigma }_{g}^{-1}\left( {\mathbf {x}}_{n}-\varvec{\mu }_{g}\right) \right] \nonumber \\&\qquad +\sum _{m=1}^{M} \sum _{g=1}^{G} \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}\left( {\mathbf {y}}_{m}\right) \left[ -\log \left| \varvec{\varSigma }_{g}\right| ^{1 / 2}-\frac{1}{2}\left( {\mathbf {y}}_{m}-\varvec{\mu }_{g}\right) ^{\prime } \varvec{\varSigma }_{g}^{-1}\left( {\mathbf {y}}_{m}-\varvec{\mu }_{g}\right) \right] \nonumber \\&\quad =-\frac{1}{2}\left[ \sum _{n=1}^{N} \sum _{g=1}^{G} \zeta ({\mathbf {x}}_n)l_{ng}\left( {\mathbf {x}}_{n}\right) \log \left| \varvec{\varSigma }_{g}\right| \right. \nonumber \\&\qquad +\sum _{n=1}^{N} \sum _{g=1}^{G} \zeta ({\mathbf {x}}_n)l_{ng} \left[ \underbrace{\left( {\mathbf {x}}_{n}-\varvec{\mu }_{g}\right) ^{\prime } \varvec{\varSigma }_{g}^{-1}\left( {\mathbf {x}}_{n}-\varvec{\mu }_{g}\right) }_{\text{ a } \text{ scalar } }\right] \nonumber \\&\qquad +\sum _{m=1}^{M} \sum _{g=1}^{G} \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}\left( {\mathbf {y}}_{m}\right) \log \left| \varvec{\varSigma }_{g}\right| \nonumber \\&\qquad \left. +\sum _{m=1}^{M} \sum _{g=1}^{G} \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}\left[ \underbrace{\left( {\mathbf {y}}_{m}-\varvec{\mu }_{g}\right) ^{\prime } \varvec{\varSigma }_{g}^{-1}\left( {\mathbf {y}}_{m}-\varvec{\mu }_{g}\right) }_{ \text{ a } \text{ scalar } }\right] \right] \nonumber \\&\quad =-\frac{1}{2}\left[ \sum _{g=1}^{G} \log \left| \varvec{\varSigma }_{g}\right| \left( \sum _{n=1}^{N} \zeta ({\mathbf {x}}_n)l_{ng}\left( {\mathbf {x}}_{n}\right) + \sum _{m=1}^{M} \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}\left( {\mathbf {y}}_{m}\right) \right) \right. \nonumber \\&\qquad +\sum _{n=1}^{N} \sum _{g=1}^{G} \zeta ({\mathbf {x}}_n)l_{ng} tr \left[ \varvec{\varSigma }_{g}^{-1} \left( {\mathbf {x}}_{n}-\varvec{\mu }_{g}\right) \left( {\mathbf {x}}_{n}-\varvec{\mu }_{g}\right) ^{\prime }\right] \nonumber \\&\qquad \left. + \sum _{m=1}^{M} \sum _{g=1}^{G} \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}tr \left[ \varvec{\varSigma }_{g}^{-1} \left( {\mathbf {y}}_{m}-\varvec{\mu }_{g}\right) \left( {\mathbf {y}}_{m}-\varvec{\mu }_{g}\right) ^{\prime }\right] \right] \nonumber \\&\quad =-\frac{1}{2}\left[ \sum _{g=1}^{G} \log \left| \varvec{\varSigma }_{g}\right| \left( \sum _{n=1}^{N} \zeta ({\mathbf {x}}_n)l_{ng}\left( {\mathbf {x}}_{n}\right) + \sum _{m=1}^{M} \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}\left( {\mathbf {y}}_{m}\right) \right) \right. \nonumber \\&\qquad +\left. \sum _{g=1}^{G} tr \left[ \varvec{\varSigma }^{-1}_{g}{\varvec{W}}_g^{X} \right] + \sum _{g=1}^{G} tr \left[ \varvec{\varSigma }_{g}^{-1}{\varvec{W}}_g^{Y} \right] \right] \nonumber \\&\qquad -\frac{1}{2}\left[ \sum _{g=1}^{G} \log \left| \varvec{\varSigma }_{g}\right| \left( \sum _{n=1}^{N} \zeta ({\mathbf {x}}_n)l_{ng}\left( {\mathbf {x}}_{n}\right) + \sum _{m=1}^{M} \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}\left( {\mathbf {y}}_{m}\right) \right) \right. \nonumber \\&\qquad \left. + \sum _{g=1}^{G} tr \left[ \varvec{\varSigma }^{-1}_{g}\left( {\varvec{W}}_g^{X} + {\varvec{W}}_g^{Y}\right) \right] \right] \end{aligned}$$

(31)

where ${\varvec{W}}_g^{X}=\sum _{n=1}^{N} \zeta ({\mathbf {x}}_n)l_{ng}\left[ \left( {\mathbf {x}}_{n}-\varvec{\mu }_{g}\right) \left( {\mathbf {x}}_{n}-\varvec{\mu }_{g}\right) ^{\prime }\right]$ and ${\varvec{W}}_g^{Y}=\sum _{m=1}^{M} \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}\left[ \left( {\mathbf {y}}_{m}-\varvec{\mu }_{g}\right) \left( {\mathbf {y}}_{m}-\varvec{\mu }_{g}\right) ^{\prime }\right]$. Finally, considering the eigenvalue decomposition $\varvec{\varSigma }_g=\lambda _g{\varvec{D}}_g{\varvec{A}}_g{\varvec{D}}^{'}_g$, (31) simplifies to:

$$\begin{aligned} \begin{aligned}&-\frac{1}{2}\left[ \sum _{g=1}^{G} p \log \lambda _g \left( \sum _{n=1}^{N} \zeta ({\mathbf {x}}_n)l_{ng}\left( {\mathbf {x}}_{n}\right) + \sum _{m=1}^{M} \varphi ({\mathbf {y}}_m)\hat{z}_{mg}^{(k+1)}\left( {\mathbf {y}}_{m}\right) \right) \right. \\&\quad + \left. \sum _{g=1}^{G} \frac{1}{\lambda _g} tr \left[ {\varvec{D}}_{g} {\varvec{A}}^{-1} {\varvec{D}}_{g}^{\prime } \left( {\varvec{W}}_g^{X} + {\varvec{W}}_g^{Y}\right) \right] \right] \end{aligned} \end{aligned}$$

(32)

The partial derivative of (32) with respect to $\left( \lambda _g,{\varvec{A}}_g, {\varvec{D}}_g\right)$ depends on the considered patterned structure: for a thorough derivation the reader is referred to Bensmail and Celeux (1996). If (8) is not satisfied, the constraints are enforced as detailed in “Appendix C”. Lastly, notice that in performing the concentration step the optimal observations of both training and test sets are retained, i.e. the ones with the highest contribution to the objective function.

The afore-described procedure falls within the structure of a general EM algorithm, for which the likelihood function does not decrease after an EM iteration, as shown in Dempster et al. (1977) and reported in page 78 of McLachlan and Krishnan (2008). $\square$

Appendix B

This appendix details the structure of the Simulation Study in Sect. 4.2.1. We consider a data generating process given by a mixture of $G=4$ components of multivariate t-distributions (McLachlan and Peel 1998; Peel and McLachlan 2000), according to the following parameters:

$$\begin{aligned}&\varvec{\tau }=(0.2, 0.4, 0.1, 0.3)', \quad \nu =6, \\&\varvec{\mu }_1=(0, 0, 0, 0, 0, 0, 0,0,0,0,0)', \\&\varvec{\mu }_2=(4, -\,4, 4, -\,4,4, -\,4,4, -\,4,4, -\,4)', \\&\varvec{\mu }_3=(0,0,7,7,7,3,6,8,-\,4,-\,4)', \\&\varvec{\mu }_4=(8, 0, 8, 0, 8, 0, 8,0,8,0,8)', \\&\varvec{\varSigma }_1 = diag(1,1,1,1,1,1,1,1,1,1), \\&\varvec{\varSigma }_2 = diag(2,2,2,2,2,2,2,2,2,2), \\&\varvec{\varSigma }_3 = \varvec{\varSigma }_4\\= & {} \begin{bmatrix} 5.05&1.26&-\,0.35&-\,0.00&-\,1.04&-\,1.35&0.29&0.07&0.69&1.17 \\ 1.26&2.57&0.17&0.00&0.27&0.11&0.61&0.11&0.59&0.89 \\ -\,0.35&0.17&6.74&-\,0.00&-\,0.26&-\,0.31&-\,0.01&0.00&0.08&0.14 \\ -\,0.00&0.00&-\,0.00&5.47&-\,0.00&-\,0.00&0.00&0.00&0.00&0.00 \\ -\,1.04&0.27&-\,0.26&-\,0.00&6.80&-\,0.76&-\,0.12&-\,0.01&0.09&0.21 \\ -\,1.35&0.11&-\,0.31&-\,0.00&-\,0.76&7.75&-\,0.26&-\,0.04&-\,0.03&0.03 \\ 0.29&0.61&-0.01&0.00&-0.12&-\,0.26&4.76&0.06&0.38&0.60 \\ 0.07&0.11&0.00&0.00&-\,0.01&-\,0.04&0.06&4.18&0.07&0.11 \\ 0.69&0.59&0.08&0.00&0.09&-0.03&0.38&0.07&3.23&0.60 \\ 1.17&0.89&0.14&0.00&0.21&0.03&0.60&0.11&0.60&3.24 \\ \end{bmatrix}. \end{aligned}$$

Fig. 7

Generalized pairs plot of the simulated data under the Simulation Setup described in 4.2.1. Both label noise and outliers are present in the data units

A generalized pairs plot of contaminated labelled units under the afore-described Simulation Setup is reported in Fig. 7.

Appendix C

This final Section presents feasible and computationally efficient algorithms for enforcing the eigenvalue-ratio constraint according to the different patterned models in Table 1. At the $k-$th iteration of the M-step, the goal is to update the estimates for the covariance matrices $\hat{\varvec{\varSigma }}_g^{(k+1)}={\hat{\lambda }}_g^{(k+1)}\hat{{\varvec{D}}}_g^{(k+1)}\hat{{\varvec{A}}}_g^{(k+1)}\hat{{\varvec{D}}}^{'(k+1)}_g$, $g=1,\ldots ,G$ such that,

$$\begin{aligned} \frac{\max _{g=1\ldots G}\max _{l=1\ldots p}{\hat{\lambda }}_g^{(k+1)}{\hat{a}}_{lg}^{(k+1)}}{\min _{g=1\ldots G}\min _{l=1\ldots p}{\hat{\lambda }}_g^{(k+1)}{\hat{a}}_{lg}^{(k+1)}} \le c \end{aligned}$$

(33)

where ${\hat{a}}_{lg}^{(k+1)}$ indicates the diagonal entries of matrix $\hat{{\varvec{A}}}_g^{(k+1)}$.

Denote with ${\hat{\varSigma }}_g^{U}={\hat{\lambda }}_g^{U} \hat{{\varvec{D}}}_g^{U}\hat{{\varvec{A}}}_g^{U}\hat{{\varvec{D}}}_g^{'U}$ the estimates for the variance covariance matrices obtained following Bensmail and Celeux (1996) without enforcing the eigenvalues-ratio restriction in (33). Lastly, denote with $\hat{\varvec{\varDelta }}^U_g={\hat{\lambda }}_g^{U}\hat{{\varvec{A}}}_g^{U}$ the matrix of eigenvalues for $\hat{\varvec{\varSigma }}_g^{U}$, with diagonal entries ${\hat{d}}_{lg}^U={\hat{\lambda }}_g^{U}{\hat{a}}_{lg}^{U}$, $l=1,\ldots ,p$.

1.1 Constrained maximization for VII, VVI and VVV models

• 1. Compute $\varvec{\varDelta }_g$ applying the optimal truncation operator defined in Fritz et al. (2013) to $\left\{ \hat{\varvec{\varDelta }}^U_1,\ldots ,\hat{\varvec{\varDelta }}^U_G\right\}$, under condition (33).

• 2. Set ${\hat{\lambda }}_g^{(k+1)}=|\varvec{\varDelta }_g|^{1/p}$, $\hat{{\varvec{A}}}_g^{(k+1)}=\frac{1}{{\hat{\lambda }}_g^{(k+1)}}\varvec{\varDelta }_g$, $\hat{{\varvec{D}}}_g^{(k+1)}=\hat{{\varvec{D}}}_g^{U}.$

1.2 Constrained maximization for VVE model

• 1. Compute $\varvec{\varDelta }_g$ applying the optimal truncation operator defined in Fritz et al. (2013) to $\left\{ \hat{\varvec{\varDelta }}^U_1,\ldots ,\hat{\varvec{\varDelta }}^U_G\right\}$, under condition (33).

• 2. Given $\varvec{\varDelta }_g$, compute the common principal components ${\varvec{D}}$ via, for example, a majorization-minimization (MM) algorithm (Browne and McNicholas 2014).

• 3. Set ${\hat{\lambda }}_g^{(k+1)}=|\varvec{\varDelta }_g|^{1/p}$, $\hat{{\varvec{A}}}_g^{(k+1)}=\frac{1}{{\hat{\lambda }}_g^{(k+1)}}\varvec{\varDelta }_g$, $\hat{{\varvec{D}}}_g^{(k+1)}={\varvec{D}}.$

1.3 Constrained maximization for EVI, EVV models

• 1. Compute $\varvec{\varDelta }_g$ applying the optimal truncation operator defined in Fritz et al. (2013) to $\left\{ \hat{\varvec{\varDelta }}^U_1,\ldots ,\hat{\varvec{\varDelta }}^U_G\right\}$, under condition (33).

• 2. Compute $\varvec{\varDelta }^{\star }_g$ constraining $\varvec{\varDelta }_g$ such that $\varvec{\varDelta }^{\star }_g=\lambda ^{\star }{\varvec{A}}_g^{\star }$. That is, constraining $|\varvec{\varDelta }^{\star }_g|$ to be equal across groups (Maronna and Jacovkis 1974; Gallegos 2002). Details are given in section 3.2 of Fritz et al. (2012).

• 3. Iterate $1-2$ until (33) is satisfied.

• 4. Set ${\hat{\lambda }}_g^{(k+1)}=\lambda ^{\star }$, $\hat{{\varvec{A}}}_g^{(k+1)}={\varvec{A}}_g^{\star }$, $\hat{{\varvec{D}}}_g^{(k+1)}=\hat{{\varvec{D}}}_g^{U}.$

1.4 Constrained maximization for EVE model

• 1. Compute $\varvec{\varDelta }_g$ applying the optimal truncation operator defined in Fritz et al. (2013) to $\left\{ \hat{\varvec{\varDelta }}^U_1,\ldots ,\hat{\varvec{\varDelta }}^U_G\right\}$, under condition (33).

• 2. Compute $\varvec{\varDelta }^{\star }_g$ constraining $\varvec{\varDelta }_g$ such that $\varvec{\varDelta }^{\star }_g=\lambda ^{\star }{\varvec{A}}_g^{\star }$. Details are given in section 3.2 of Fritz et al. (2012).

• 3. Iterate 1–2 until (33) is satisfied.

• 4. Given ${\varvec{A}}^{\star }_g$, compute the common principal components ${\varvec{D}}$ via, for example, a majorization-minimization (MM) algorithm (Browne and McNicholas 2014).

• 5. Set ${\hat{\lambda }}_g^{(k+1)}=\lambda ^{\star }$, $\hat{{\varvec{A}}}_g^{(k+1)}={\varvec{A}}_g^{\star }$, $\hat{{\varvec{D}}}_g^{(k+1)}={\varvec{D}}$.

1.5 Constrained maximization for VEI, VEV models

• 1. Set $\varvec{\varDelta }_g=\hat{\varvec{\varDelta }}^U_g$.

• 2. Set $\lambda _g^{\star }={\hat{\lambda }}_g^{U}$, $g=1,\ldots ,G$.

• 3. Compute $\varvec{\varDelta }^{\star }_g$ applying the optimal truncation operator defined in Fritz et al. (2013) to $\left\{ \varvec{\varDelta }_1,\ldots ,\varvec{\varDelta }_G\right\}$, under condition (33).

• 4. Compute ${\varvec{A}}^{\star }=\left. \sum _{g=1}^G\frac{1}{\lambda _g^{\star }}\varvec{\varDelta }^{\star }_g \Bigg / \left| \sum _{g=1}^G\frac{1}{\lambda _g^{\star }}\varvec{\varDelta }^{\star }_g \right| ^{1/p} \right.$.

• 5. Compute $\lambda _g^{\star }=\frac{1}{p}tr\left( \varvec{\varDelta }^{\star }_g {{\varvec{A}}^{\star }}^{-1}\right) .$

• 6. Set $\varvec{\varDelta }_g=\lambda _g^{\star }{\varvec{A}}^{\star }$.

• 7. Iterate 3–6 until (33) is satisfied.

• 8. Set ${\hat{\lambda }}_g^{(k+1)}=\lambda ^{\star }_g$, $\hat{{\varvec{A}}}_g^{(k+1)}={\varvec{A}}^{\star }$, $\hat{{\varvec{D}}}_g^{(k+1)}=\hat{{\varvec{D}}}_g^{U}$.

1.6 Constrained maximization for VEE model

• 1. Set ${\varvec{K}}_g=\hat{\varvec{\varSigma }}^U_g$.

• 2. Set $\lambda _g^{\star }={\hat{\lambda }}_g^{U}$, $g=1,\ldots ,G$.

• 3. Compute ${\varvec{K}}_g^{\star }$ applying the optimal truncation operator defined in Fritz et al. (2013) to $\left\{ {\varvec{K}}_1,\ldots ,{\varvec{K}}_G \right\}$, under condition (33).

• 4. Compute ${\varvec{C}}^{\star }=\left. \sum _{g=1}^G\frac{1}{\lambda _g^{\star }}{\varvec{K}}^{\star }_g \Bigg / \left| \sum _{g=1}^G\frac{1}{\lambda _g^{\star }}{\varvec{K}}^{\star }_g \right| ^{1/p} \right.$.

• 5. Compute $\lambda _g^{\star }=\frac{1}{p}tr\left( {\varvec{K}}^{\star }_g {{\varvec{C}}^{\star }}^{-1}\right)$.

• 6. Set ${\varvec{K}}_g=\lambda _g^{\star }{\varvec{C}}^{\star }$.

• 7. Iterate $3-6$ until (33) is satisfied.

• 8. Considering the spectral decomposition for ${\varvec{C}}^{\star }={\varvec{D}}^{\star }{\varvec{A}}^{\star }{{\varvec{D}}^{\star }}^{'}$, set ${\hat{\lambda }}_g^{(k+1)}=\lambda ^{\star }_g$, $\hat{{\varvec{A}}}_g^{(k+1)}={\varvec{A}}^{\star }$, $\hat{{\varvec{D}}}_g^{(k+1)}={\varvec{D}}^{\star }$.