Compressed Covariance Estimation with Automated Dimension Learning

Abstract

We propose a method for estimating a covariance matrix that can be represented as a sum of a low-rank matrix and a diagonal matrix. The proposed method compresses high-dimensional data, computes the sample covariance in the compressed space, and lifts it back to the ambient space via a decompression operation. A salient feature of our approach relative to existing literature on combining sparsity and low-rank structures in covariance matrix estimation is that we do not require the low-rank component to be sparse. A principled framework for estimating the compressed dimension using Stein’s Unbiased Risk Estimation theory is demonstrated. Experimental simulation results demonstrate the efficacy and scalability of our proposed approach.

Appendices

Appendix A: Proof of Lemma 3.1

We obtain unbiased estimators of Eqs. 3.2, 3.3, and 3.4. Suppose $\{X_{i}\}_{i = 1}^{n}$ is a random sample from N(μ,Σ) where, without loss of generality, we let μ = 0. We have,

$$ {\mathbb{E}}(({\widetilde{\sigma}}_{ij}^{s})^{2}) = \frac{n}{n-1}{\sigma_{ij}^{2}} + \frac{\sigma_{ii}\sigma_{jj}}{n-1} $$

(A.1)

and,

$$ {\mathbb{E}}({\widetilde{\sigma}}_{ii}^{s}{\widetilde{\sigma}}_{jj}^{s}) = \frac{n + 1}{n-1}{\sigma_{ii}\sigma_{jj}} + \frac{2(n + 2)}{n(n-1)}{\sigma_{ij}^{2}}. $$

(A.2)

Equations A.1 and A.2 are obtained from Yi and Zou (2013). Solving (A.1) and (A.2) simultaneously, we obtain unbiased estimators for ${\sigma _{ij}^{2}}$ and σ_iiσ_jj below

$$ {\mathbb{E}}\left[\frac{n(n^{2} - 1)}{n^{3} + n^{2} - 2n -4}({\widetilde{\sigma}}_{ij}^{s})^{2} - \frac{n(n - 1)}{n^{3} + n^{2} - 2n -4}{\widetilde{\sigma}}_{ii}^{s}{\widetilde{\sigma}}_{jj}^{s}\right] = {\sigma_{ij}^{2}}. $$

(A.3)

$$ {\mathbb{E}}\left[\frac{2(n-1)(n + 2)}{2n + 4 - n^{3} - n^{2}}({\widetilde{\sigma}}_{ij}^{s})^{2} - \frac{n^{2}(n - 1)}{2n + 4 - n^{3} - n^{2}}{\widetilde{\sigma}}_{ii}^{s}{\widetilde{\sigma}}_{jj}^{s}\right] = {\sigma_{ii}}{\sigma_{jj}}. $$

(A.4)

An unbiased estimator of ${\text {Var}}({\widetilde {\sigma }}_{ij}^{s})$ is given by

$$ \widehat{\text{Var}}({\widetilde{\sigma}}_{ij}^{s}) = \frac{{\widehat{\sigma}}_{ij}^{2} + {\widehat{\sigma_{ii}\sigma_{jj}}}}{n - 1}. $$

(A.5)

Substituting (A.3) and (A.4) in (A.5) gives (3.2).

Equation 3.3 is obtained from Eq. 3.2 trivially. To obtain (3.4), note that

$$ {\text{Cov}}({\widetilde{\sigma}}_{jj}^{s}, {\widetilde{\sigma}}_{ii}^{s}) = \frac{2(n + 2)}{n(n - 1)}{\sigma_{ij}^{2}} + \frac{2}{n - 1}{\sigma_{ii}}{\sigma_{jj}}. $$

(A.6)

Substituting the unbiased estimators of ${\sigma }_{ij}^{2}$ and σ_iiσ_jj in Eq. A.6, obtained from Eqs. A.3 and A.4, gives us Eq. 3.4. This completes the proof.

Appendix B: Proof of Theorem 3.2

We analyze each term in Eq. 3.1 one at a time. Consider $\|\widehat {{\Sigma }}_{\text {CD}}(k) - {\widehat {{\Sigma }}}\|_{F}^{2}$, a natural unbiased estimator of ${\mathbb {E}}\|\widehat {{\Sigma }}_{\text {CD}}(k) - {\widehat {{\Sigma }}}\|_{F}^{2}$. Then

$$\begin{array}{@{}rcl@{}} \|\widehat{{\Sigma}}_{\text{CD}}(k) - {\widehat{{\Sigma}}}\|_{F}^{2} & =& \displaystyle\sum\limits_{i,j}^{p}{\left[(\eta - 1){\widehat{\sigma}}_{ij} + {\gamma}I(i = j)\right]}^{2} \\ & = &{\displaystyle\sum\limits_{i \ne j}^{p}}{(\eta - 1)^{2}{\widehat{\sigma}}_{ij}^{2}} + \displaystyle\sum\limits_{i=j}^{p}\left[(\eta - 1){\widehat{\sigma}}_{ii} + \gamma\right]^{2} \\ & =& (\eta - 1)^{2}\|{\widehat{{\Sigma}}} \circ {\widehat{{\Sigma}}} \|_{F} + p{\gamma^{2}} + 2{\gamma}(\eta - 1){\text{Tr}}({\widehat{{\Sigma}}}), \end{array} $$

(B.1)

where (B.1) is obtained by noting that ${\displaystyle \sum \limits _{i\ne j}^{p}}{\widetilde {\sigma }}_{ij}^{2} = {\displaystyle \sum \limits _{i = 1}^{p}}{\displaystyle \sum \limits _{\underset {j \ne i}{j = 1}}^{p}}{\widetilde {\sigma }}_{ij}^{2} = \|{\widetilde {{\Sigma }}}\circ {\widetilde {{\Sigma }}}\|_{F} \;- \;\text {Tr}({\widetilde {{\Sigma }}}\circ {\widetilde {{\Sigma }}})$. Consider optimism, the third term on the right hand side of Eq. 3.1. Then

$$\begin{array}{@{}rcl@{}} {\text{optimism}} & =& {\displaystyle\sum\limits_{i \ne j}^{p}} {\eta}{\text{Var}}({\widehat{\sigma}}_{ij}) + {\displaystyle\sum\limits_{i = 1}^{p}}\left\{\eta{\text{Var}}({\widehat{\sigma}}_{ii}) + \gamma\;{\text{cov}}\left( {\displaystyle\sum\limits_{\underset{l \ne i}{l = 1}}^{p}}{\widehat{\sigma}}_{ll} + {\widehat{\sigma}}_{ii}, {\widehat{\sigma}}_{ii}\right)\right\} \\ & =& {\displaystyle\sum\limits_{i \ne j}^{p}} {\eta}{\text{Var}}({\widehat{\sigma}}_{ij}) + {\displaystyle\sum\limits_{i = 1}^{p}}\left\{(\eta + \gamma){\text{Var}}({\widehat{\sigma}}_{ii}) + \gamma{\displaystyle\sum\limits_{\underset{l \ne i}{l = 1}}^{p}}{\text{cov}}\left( {\widehat{\sigma}}_{ll}, {\widehat{\sigma}}_{ii}\right)\right\}. \end{array} $$

(B.2)

Using Lemma 3.1, we have

$$\begin{array}{@{}rcl@{}} {\widehat{\text{optimism}}} & =& {\eta}a_{n}{\displaystyle\sum\limits_{i\ne j}^{p}}{\widetilde{\sigma}}_{ij}^{2} + {\eta}b_{n}{\displaystyle\sum\limits_{i = 1}^{p}}{\widetilde{\sigma}}_{ii}{\displaystyle\sum\limits_{j \ne i}^{p}}{\widetilde{\sigma}}_{jj} + c_{n}(\gamma + \eta){\displaystyle\sum\limits_{i = 1}^{p}}{\widetilde{\sigma}}_{ii}^{2} + d_{n}{\gamma}{\displaystyle\sum\limits_{i \ne l}^{p}}{\widetilde{\sigma}}_{il}^{2} + e_{n}{\displaystyle\sum\limits_{i \ne l}^{p}}{\widetilde{\sigma}}_{ii}{\widetilde{\sigma}}_{ll} \\ & =& 2\left\{(a_{n}\eta + d_{n}\gamma)\left( \|{\widetilde{{\Sigma}}}\circ{\widetilde{{\Sigma}}}\|_{F} \;- \;\text{Tr}({\widetilde{{\Sigma}}}\circ{\widetilde{{\Sigma}}})\right) + (b_{n}\eta + c_{n}\gamma) \right. \\ && \left.\left( {{\text{Tr}}({\widetilde{{\Sigma}}})}^{2} - {\text{Tr}}({\widetilde{{\Sigma}}} \circ {\widetilde{{\Sigma}}})\right) + c_{n}(\eta + \gamma){\text{Tr}}\left( {\widetilde{{\Sigma}}} \circ {\widetilde{{\Sigma}}}\right)\right\}, \end{array} $$

(B.3)

where (B.3) is obtained by writing ${\displaystyle \sum \limits _{i = 1}^{p}}{\widetilde {\sigma }}_{ii}{\displaystyle \sum \limits _{\underset {j \ne i}{j = 1}}^{p}}{\widetilde {\sigma }}_{jj} = {{\text {Tr}}({\widetilde {{\Sigma }}})}^{2} - {\text {Tr}}({\widetilde {{\Sigma }}} \circ {\widetilde {{\Sigma }}})$ and ${\displaystyle \sum \limits _{i = 1}^{p}}{\widetilde {\sigma }}_{ii}^{2} = {\text {Tr}}({\widetilde {{\Sigma }}} \circ {\widetilde {{\Sigma }}})$.

The proof is completed by combining (B.1) and (B.3) to obtain an unbiased estimator of the Frobenius risk of ${\widehat {{\Sigma }}}_{CD}(k)$.