Efficient, certifiably optimal clustering with applications to latent variable graphical models

Abstract

Motivated by the task of clustering either d variables or d points into K groups, we investigate efficient algorithms to solve the Peng–Wei (P–W) K-means semi-definite programming (SDP) relaxation. The P–W SDP has been shown in the literature to have good statistical properties in a variety of settings, but remains intractable to solve in practice. To this end we propose FORCE, a new algorithm to solve this SDP relaxation. Compared to off-the-shelf interior point solvers, our method reduces the computational complexity of solving the SDP from ${\widetilde{{\mathcal {O}}}}(d^7\log \epsilon ^{-1})$ to ${\widetilde{{\mathcal {O}}}}(d^{6}K^{-2}\epsilon ^{-1})$ arithmetic operations for an $\epsilon$-optimal solution. Our method combines a primal first-order method with a dual optimality certificate search, which when successful, allows for early termination of the primal method. We show for certain variable clustering problems that, with high probability, FORCE is guaranteed to find the optimal solution to the SDP relaxation and provide a certificate of exact optimality. As verified by our numerical experiments, this allows FORCE to solve the P–W SDP with dimensions in the hundreds in only tens of seconds. For a variation of the P–W SDP where K is not known a priori a slight modification of FORCE reduces the computational complexity of solving this problem as well: from ${\widetilde{{\mathcal {O}}}}(d^7\log \epsilon ^{-1})$ using a standard SDP solver to ${\widetilde{{\mathcal {O}}}}(d^{4}\epsilon ^{-1})$.

Appendices

Appendix A: Proofs omitted in Sect. 3.2

First we have a lemma regarding the concentration of the noise terms ${\mathbf {E}}$ about their mean. Sometimes rather than state these concentration results in terms of d, we state them in terms of $t \ge d$ to allow for more precise control of constants in our main theorems. We let ${\mathcal {E}}$ denote the event that $||{{\widehat{\varvec{\varGamma }}}} - \varvec{\varGamma }^*||_{\infty } \le p_1||\varvec{\varGamma }^*||_{\max }\sqrt{\frac{\log d}{n}}$.

Lemma 9

Under the notation and assumptions from previous sections, if $t\ge d$ then

$$\begin{aligned} \left| \sum _{j=1}^n {\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j\varvec{E}_{G^*_{i}}^{jT}{\mathbf {1}}- {\mathbf {1}}^T\varvec{\varGamma }^*_{G^*_{i},G^*_{i}}{\mathbf {1}}\right| \le c_0 ||\varvec{\varGamma }^*||_{\infty }\sqrt{|G^*_{i}|^2 n \log t}, \end{aligned}$$

with probability at least $1-\frac{2}{t}$, where $c_0 = c'(1+\sqrt{p_0})$ is a constant that depends only on $p_0$ and the absolute constant $c'$ from Proposition 11. Similarly with probability at least $1-\frac{2}{t}$, for $a \in G^*_{i}$,

$$\begin{aligned} \left| \sum _{j=1}^n {\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j E^j_{a} - \gamma ^*_a \right| \le c_0 ||\varvec{\varGamma }^*||_{\infty }\sqrt{|G^*_{i}| n \log t} , \end{aligned}$$

Proof

To obtain the result, we observe that

$$\begin{aligned} \sum _{j=1}^n {\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j\varvec{E}_{G^*_{i}}^{jT}{\mathbf {1}}- {\mathbf {1}}^T\varvec{\varGamma }^*_{G^*_{i},G^*_{i}}{\mathbf {1}}\end{aligned}$$

is a quadratic form of a $n|G^*_{i}|$-dimensional Gaussian random vector with independent entries. In particular, if we define ${\mathbf {M}}$ to be block diagonal with the ith $n\times n$ diagonal block as $(\varvec{\varGamma }^*_{G^*_{i},G^*_{i}})^{1/2}{\mathbf {1}}{\mathbf {1}}^T(\varvec{\varGamma }^*_{G^*_{i},G^*_{i}})^{1/2}$, then we can apply Corollary 1 with matrix ${\mathbf {M}}$. Because $||{\mathbf {M}}||_2 \le ||\varvec{\varGamma }^*||_{\infty }|G^*_{i}|$ and $||{\mathbf {M}}||_F \le ||\varvec{\varGamma }^*||_{\infty }|G^*_{i}|\sqrt{n}$, applying the corollary gives

$$\begin{aligned} \left| \sum _{j=1}^n {\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j\varvec{E}_{G^*_{i}}^{jT}{\mathbf {1}}- {\mathbf {1}}^T\varvec{\varGamma }^*_{G^*_{i},G^*_{i}}{\mathbf {1}}\right| \le c' ||\varvec{\varGamma }^*||_{\infty }\left( \sqrt{|G^*_{i}|^2 n \log t} + |G^*_{i}|\log t \right) , \end{aligned}$$

with probability at least $1-\frac{2}{t}$. Using the assumption $\log d \le p_0 n$ gives the desired result. The proof of the second statement follows similarly, taking instead the diagonal blocks of ${\mathbf {M}}$ as $(\varvec{\varGamma }^*_{G^*_{i},G^*_{i}})^{1/2}{\mathbf {1}}{\mathbf {e}}_a^T(\varvec{\varGamma }^*_{G^*_{i},G^*_{i}})^{1/2}$, giving $||{\mathbf {M}}||_2 \le ||\varvec{\varGamma }^*||_{\infty }\sqrt{|G^*_{i}|}$ and $||{\mathbf {M}}||_F \le ||\varvec{\varGamma }^*||_{\infty }\sqrt{n|G^*_{i}|}$. $\square$

1.1 Proof of Lemma 6

Step 1 For notation, $c_i$ will be used to denote absolute constants. The first step is to decompose ${\mathbf {Q}}_{i}^{\perp }(\varvec{X})$. Recall that under the G-Latent model, ${\mathbf {D}}= -{{\widehat{\varvec{\varSigma }}}} + {{\widehat{\varvec{\varGamma }}}}$. Substituting that into the expression for ${\mathbf {Q}}_{i}^{\perp }(\varvec{X})$ gives

$$\begin{aligned} {\mathbf {Q}}_{i}^{\perp }(\varvec{X})&= \mathop {\underbrace{-\frac{1}{|G^*_{i}|^2} \left( {\mathbf {1}}^T{\widehat{\varvec{\varSigma }}}_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) {\mathbf {1}}{\mathbf {1}}^T + \frac{1}{|G^*_{i}|}\left( {\mathbf {1}}{\mathbf {1}}^T {\widehat{\varvec{\varSigma }}}_{G^*_{i},G^*_{i}} + {\widehat{\varvec{\varSigma }}}_{G^*_{i},G^*_{i}}{\mathbf {1}}{\mathbf {1}}^T\right) - {\widehat{\varvec{\varSigma }}}_{G^*_{i},G^*_{i}}}}\limits _{\mathrm{(i)}}\\&\quad + \mathop {\underbrace{\frac{1}{|G^*_{i}|^2} \left( {\mathbf {1}}^T{\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) {\mathbf {1}}{\mathbf {1}}^T - \frac{1}{|G^*_{i}|}\left( {\mathbf {1}}{\mathbf {1}}^T {\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}} + {\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}{\mathbf {1}}{\mathbf {1}}^T\right) + {\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}}}\limits _{\mathrm{(ii)}}. \end{aligned}$$

For (i), we recall that by the definition of the G-Latent model that

$$\begin{aligned} {\widehat{\varvec{\varSigma }}}_{G^*_{i},G^*_{i}} = \frac{1}{n}\sum _{j=1}^n\varvec{X}_{G^*_{i}}^j\varvec{X}_{G^*_{i}}^{jT} = \sum _{j=1}^n(Z_i^j + \varvec{E}_{G^*_{i}}^j)(Z_i^j + \varvec{E}_{G^*_{i}}^j)^T. \end{aligned}$$

Plugging this into (i) and simplifying gives us that

$$\begin{aligned} \text {(i)} = \frac{1}{n}\sum _{j=1}^n\left( -\frac{{\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j\varvec{E}_{G^*_{i}}^{jT}{\mathbf {1}}}{|G^*_{i}|^2} {\mathbf {1}}{\mathbf {1}}^T + \frac{{\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j}{|G^*_{i}|}\left( {\mathbf {1}}\varvec{E}_{G^*_{i}}^{jT} + \varvec{E}_{G^*_{i}}^j{\mathbf {1}}^T\right) - \varvec{E}_{G^*_{i}}^j\varvec{E}_{G^*_{i}}^{jT} \right) . \end{aligned}$$

Now we see that, again, the expression for ${\mathbf {Q}}_{i}^{\perp }(\varvec{X})$ has eight terms. We first show that each concentrates to its mean at the desired rate, and then use the triangle inequality to obtain the final result. Fortunately, we can subtract the mean for each of the 8 terms to the expression for ${\mathbf {Q}}_{i}^{\perp }(\varvec{X})$ as the means for (i) are offset by the means for (ii). To give the new decomposition of ${\mathbf {Q}}_{i}^{\perp }(\varvec{X})$ explicitly,

$$\begin{aligned} {\mathbf {Q}}_{i}^{\perp }(\varvec{X})&= -\mathop {\underbrace{\sum _{j=1}^n\frac{{\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j\varvec{E}_{G^*_{i}}^{jT}{\mathbf {1}}}{n|G^*_{i}|^2} {\mathbf {1}}{\mathbf {1}}^T}}\limits _{\mathrm{(i).a}} + \mathop {\underbrace{\sum _{j=1}^n\frac{{\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j}{n|G^*_{i}|}{\mathbf {1}}\varvec{E}_{G^*_{i}}^{jT}}}\limits _{\mathrm{(i).b}} + \mathop {\underbrace{\sum _{j=1}^n\frac{{\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j}{n|G^*_{i}|}\varvec{E}_{G^*_{i}}^j{\mathbf {1}}^T}}\limits _{\mathrm{(i).c}} - \mathop {\underbrace{\frac{1}{n}\sum _{j=1}^n\varvec{E}_{G^*_{i}}^j\varvec{E}_{G^*_{i}}^{jT}}}\limits _{\mathrm{(i).d}}\nonumber \\&\quad + \mathop {\underbrace{\frac{1}{|G^*_{i}|^2} \left( {\mathbf {1}}^T{\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) {\mathbf {1}}{\mathbf {1}}^T}}\limits _{\mathrm{(ii).a}} - \mathop {\underbrace{\frac{1}{|G^*_{i}|}{\mathbf {1}}{\mathbf {1}}^T {\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}}}\limits _{\mathrm{(ii).b}} + \mathop {\underbrace{\frac{1}{|G^*_{i}|}{\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}{\mathbf {1}}{\mathbf {1}}^T}}\limits _{\mathrm{(ii).c}} + \mathop {\underbrace{{\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}}}\limits _{\mathrm{(ii).d}}. \end{aligned}$$

(30)

Step 2 For the term (i).a, we can directly apply Lemma 9. Doing so, it follows immediately that with probability at least $1-\frac{2}{t}$

$$\begin{aligned} \left| \left| {\sum _{j=1}^n\frac{{\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j\varvec{E}_{G^*_{i}}^{jT}{\mathbf {1}}}{n|G^*_{i}|^2} {\mathbf {1}}{\mathbf {1}}^T - \frac{1}{|G^*_{i}|^2}\left( {\mathbf {1}}^T\varvec{\varGamma }^*_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) {\mathbf {1}}{\mathbf {1}}^T}_2\right| \right| \le c_0 ||\varvec{\varGamma }^*||_{\infty }\sqrt{\frac{\log t}{n}}. \end{aligned}$$

For the term (i).c (and so by symmetry (i).b), we observe that has the form ${\mathbf {u}}{\mathbf {v}}^T$ and that $||{\mathbf {u}}{\mathbf {v}}^T||_2 = ||{\mathbf {u}}||_2||{\mathbf {v}}||_2$. Therefore, we can apply Lemma 9 and obtain that with probability at least $1-2|G^*_{i}|/t^2$,

$$\begin{aligned} \bigg |\bigg |\sum _{j=1}^n\frac{{\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j}{n|G^*_{i}|}\varvec{E}_{G^*_{i}}^j{\mathbf {1}}^T - \frac{1}{|G^*_{i}|}{\mathbf {1}}{\mathbf {1}}^T \varvec{\varGamma }^*_{G^*_{i},G^*_{i}}\bigg |\bigg |_2 \le c_0 ||\varvec{\varGamma }^*||_{\infty } \sqrt{\frac{2\log t}{n}}. \end{aligned}$$

Step 3 Now we control the term (i).d, the sample covariance matrix of the errors. We can directly apply Corollary 3 to obtain that with probability at least $1-2/t$

$$\begin{aligned}&\left| \left| {\frac{1}{n}\sum _{j=1}^n\varvec{E}_{G^*_{i}}^j\varvec{E}_{G^*_{i}}^{jT} - \varvec{\varGamma }^*_{G^*_{i},G^*_{i}}}\right| \right| _2 \\&\quad \le ||\varvec{\varGamma }^*||_{\infty }\left( \frac{|G^*_{i}|}{n} + 2\frac{\sqrt{2|G^*_{i}|\log t}}{n} + 2\sqrt{\frac{|G^*_{i}|}{n}} + (2+\sqrt{p_0})\sqrt{\frac{2\log t}{n}}\right) \\&\quad \le ||\varvec{\varGamma }^*||_{\infty }\left( \frac{d}{n} + (2+2\sqrt{2p_0})\sqrt{\frac{d}{n}} + (2+\sqrt{p_0})\sqrt{\frac{2\log t}{n}}\right) . \end{aligned}$$

Step 4 For the terms in (ii), consider first (ii).a. We see that

$$\begin{aligned} \bigg |\bigg |\left( {\mathbf {1}}^T{\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) {\mathbf {1}}{\mathbf {1}}^T - \left( {\mathbf {1}}^T\varvec{\varGamma }^*_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) {\mathbf {1}}{\mathbf {1}}^T\bigg |\bigg |_{\max } \le |G^*_{i}|||{\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}} - \varvec{\varGamma }^*_{G^*_{i},G^*_{i}} ||_{\infty } \end{aligned}$$

Conditional on event ${\mathcal {E}}$,

$$\begin{aligned} \bigg |\bigg |\frac{1}{|G^*_{i}|^2} \left( {\mathbf {1}}^T{{\widehat{\varvec{\varGamma }}}}_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) {\mathbf {1}}{\mathbf {1}}^T - \frac{1}{|G^*_{i}|^2} \left( {\mathbf {1}}^T\varvec{\varGamma }^*_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) {\mathbf {1}}{\mathbf {1}}^T\bigg |\bigg |_{\max } \le \frac{p_1||\varvec{\varGamma }^*||_{\infty }}{|G^*_{i}|}\sqrt{\frac{\log d}{n}}. \end{aligned}$$

Because the matrices above are a multiple of ${\mathbf {1}}{\mathbf {1}}^T$, it follows that

$$\begin{aligned} \bigg |\bigg |\frac{1}{|G^*_{i}|^2} \left( {\mathbf {1}}^T{{\widehat{\varvec{\varGamma }}}}_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) {\mathbf {1}}{\mathbf {1}}^T - \frac{1}{|G^*_{i}|^2} \left( {\mathbf {1}}^T\varvec{\varGamma }^*_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) {\mathbf {1}}{\mathbf {1}}^T\bigg |\bigg |_{2} \le p_1 ||\varvec{\varGamma }^*||_{\infty }\sqrt{\frac{\log d}{n}}. \end{aligned}$$

Next for (ii).b (and (ii).c by symmetry), we can see that

$$\begin{aligned} \bigg |\bigg |\frac{1}{|G^*_{i}|}{\mathbf {1}}{\mathbf {1}}^T {{\widehat{\varvec{\varGamma }}}}_{G^*_{i},G^*_{i}} - \frac{1}{|G^*_{i}|}{\mathbf {1}}{\mathbf {1}}^T \varvec{\varGamma }^*_{G^*_{i},G^*_{i}}\bigg |\bigg |_{2} = \frac{1}{|G^*_{i}|}\bigg |\bigg |{\mathbf {1}}{\mathbf {1}}^T \left( {{\widehat{\varvec{\varGamma }}}}_{G^*_{i},G^*_{i}} - \varvec{\varGamma }^*_{G^*_{i},G^*_{i}} \right) \bigg |\bigg |_{2}. \end{aligned}$$

(31)

Because ${\widehat{\varvec{\varGamma }}}$ and $\varvec{\varGamma }^*$ are diagonal, we can use event ${\mathcal {E}}$ and the fact that for matrices of the form ${\mathbf {u}}{\mathbf {v}}^T$, $||{\mathbf {u}}{\mathbf {v}}^T||_2 = ||{\mathbf {u}}||_2||{\mathbf {v}}||_2$, to obtain

$$\begin{aligned} \bigg |\bigg |\frac{1}{|G^*_{i}|}{\mathbf {1}}{\mathbf {1}}^T {{\widehat{\varvec{\varGamma }}}}_{G^*_{i},G^*_{i}} - \frac{1}{|G^*_{i}|}{\mathbf {1}}{\mathbf {1}}^T \varvec{\varGamma }^*_{G^*_{i},G^*_{i}}\bigg |\bigg |_{2} \le p_1 |\varvec{\varGamma }^*|_{\infty }\sqrt{\frac{\log d}{n}} \end{aligned}$$

The same result is immediate for (ii).a by (4). Therefore by combining the above, applying the triangle inequality to (30), using that ${\mathcal {E}}$ occurs with probability at least $1-p_2/d^2$, and choosing $t=d^2$, we find that with probability at least $1-\frac{c_2}{d^2}$

$$\begin{aligned} ||{\mathbf {Q}}_{i}^{\perp }(\varvec{X})||_2 \le c_1||\varvec{\varGamma }^*||_{\infty }\left( \frac{d}{n} +\sqrt{\frac{d}{n}} + \sqrt{\frac{\log d}{n}} \right) , \end{aligned}$$

concluding the proof. $\square$

1.1.1 Proof of Lemma 7

Under the G-Latent model,

$$\begin{aligned} y'_{a,b}({\mathbf {X}},y_T) = -\mathop {\underbrace{{\widehat{\varSigma }}_{ a, b }}}\limits _{\mathrm{(i)}} + \mathop {\underbrace{y_a({\mathbf {X}},y_T)}}\limits _{\mathrm{(ii)}} + \mathop {\underbrace{y_b({\mathbf {X}},y_T)}}\limits _{\mathrm{(iii)}} \end{aligned}$$

Above, we saw that

$$\begin{aligned} y_a(\varvec{X},y_T) = \frac{1}{2|G^*_{i}|^2}{\mathbf {1}}^T{\mathbf {D}}_{G^*_{i},G^*_{i}}{\mathbf {1}}- \frac{1}{|G^*_{i}|}{\mathbf {D}}_{a,G^*_{i}}{\mathbf {1}}- \frac{1}{2|G^*_{i}|}y_T, \end{aligned}$$

and likewise for $y_b$. Below we denote by $\sigma _1 = \max _i C_{i,i}^*$ and $\sigma _2 = \max \{\max _i C_{i,i}^*,||\varvec{\varGamma }^*||_{\infty }\}$. Following the same decomposition as in Lemma 6, we get that

$$\begin{aligned} y_a(\varvec{X},y_T)&= -\frac{1}{2|G^*_{i}|^2}{\mathbf {1}}^T{\widehat{\varvec{\varSigma }}}_{ G^*_{i},G^*_{i} }{\mathbf {1}}+ \frac{1}{2|G^*_{i}|^2}{\mathbf {1}}^T{\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}{\mathbf {1}}+ \frac{1}{|G^*_{i}|}{\widehat{\varvec{\varSigma }}}_{ a,G^*_{i} } {\mathbf {1}}- {\widehat{\varGamma }}_{a,a} - \frac{1}{2|G^*_{i}|}y_T \\&= \mathop {\underbrace{\frac{1}{n}\sum _{l=1}^n \frac{1}{2}(Z_i^l)^2}}\limits _{\mathrm{(ii).a}} - \mathop {\underbrace{\frac{1}{2n|G^*_{i}|^2}\sum _{l=1}^n ({\mathbf {1}}^T\varvec{E}_{G^*_{i}}^l)^2}}\limits _{\mathrm{(ii).b}} + \mathop {\underbrace{\frac{1}{n|G^*_{i}|}\sum _{l=1}^n E_a^l {\mathbf {1}}^T\varvec{E}_{G^*_{i}}^l}}\limits _{\mathrm{(ii).c}} + \mathop {\underbrace{\frac{1}{n}\sum _{l=1}^n E^l_a Z_i^l}}\limits _{\mathrm{(ii).d}}\\&\quad + \mathop {\underbrace{\frac{1}{2|G^*_{i}|^2}{\mathbf {1}}^T{\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}{\mathbf {1}}}}\limits _{\mathrm{(ii).e}} - \mathop {\underbrace{\frac{1}{|G^*_{i}|}{\widehat{\varGamma }}_{a,a}}}\limits _{\mathrm{(ii).f}} - \frac{1}{2|G^*_{i}|}y_T. \end{aligned}$$

As in the proof of Lemma 6, the means of (ii).b and (ii).c offset the means of (ii).e and (ii).f. To control terms (ii).b and (ii).c, by Lemma 9 with probability at least $1-1/t$,

$$\begin{aligned} \frac{1}{2n|G^*_{i}|^2}\sum _{j=1}^n\left( {\mathbf {1}}^T\varvec{E}_{G^*_{i}}^j\varvec{E}_{G^*_{i}}^{jT}{\mathbf {1}}- {\mathbf {1}}^T\varvec{\varGamma }^*_{G^*_{i},G^*_{i}}{\mathbf {1}}\right) \le \frac{c_0 ||\varvec{\varGamma }^*||_{\infty }}{2} \sqrt{\frac{\log t}{n |G^*_{i}|^2 }}. \end{aligned}$$

Likewise, by Lemma 9,

$$\begin{aligned} \frac{1}{n|G^*_{i}|}\sum _{i=1}^n \left( E_a \varvec{E}_{G^*_{i}}^{jT}{\mathbf {1}}- \gamma _{a}^*\right) \ge -c_0 ||\varvec{\varGamma }^*||_{\infty }\sqrt{\frac{\log t}{n |G^*_{i}| }}, \end{aligned}$$

with probability at least $1-1/t$. Conditional on event ${\mathcal {E}}$, (4) shows that

$$\begin{aligned} \frac{1}{2|G^*_{i}|^2}\left( {\mathbf {1}}^T{\widehat{\varvec{\varGamma }}}_{G^*_{i},G^*_{i}}{\mathbf {1}}- {\mathbf {1}}^T\varvec{\varGamma }^*_{G^*_{i},G^*_{i}}{\mathbf {1}}\right)&\ge -p_1 ||\varvec{\varGamma }^*||_{\infty }\sqrt{\frac{\log d}{n |G^*_{i}| }},\\ \frac{1}{|G^*_{i}|}\left( {\widehat{\varGamma }}_{a,a} - \varGamma ^*_{a,a} \right)&\le p_1||\varvec{\varGamma }^*||_{\infty }\sqrt{\frac{\log d}{n |G^*_{i}| }}. \end{aligned}$$

Lastly, if we denote by $\sigma _1= \max _i C_{i,i}^*$, term (ii).d can be bounded by using Corollary 1, which gives that

$$\begin{aligned} \frac{1}{n}\sum _{l=1}^n E^l_a Z_i^l \ge - c_0||\varvec{\varGamma }^*||_{\infty }^{1/2}\sigma _1^{1/2} \sqrt{\frac{\log t}{n}}, \end{aligned}$$

(32)

with probability at least $1-1/t$. The same results can be obtained for $y_b$. For the terms in (i), we expand as before:

$$\begin{aligned} {\widehat{\varSigma }}_{ a,b, \, }= \mathop {\underbrace{\frac{1}{n}\sum _{l=1}^n Z_i^lZ_j^l}}\limits _{\mathrm{(i).a}} + \mathop {\underbrace{\frac{1}{n}\sum _{l=1}^nE_a^lZ_j^l}}\limits _{\mathrm{(i).b}} + \mathop {\underbrace{\frac{1}{n}\sum _{l=1}^nE_b^lZ_i^l}}\limits _{\mathrm{(i).c}} + \mathop {\underbrace{\frac{1}{n}\sum _{l=1}^n E_a^l E_b^l}}\limits _{\mathrm{(i).d}}. \end{aligned}$$

Terms (i).b and (i).c can be bounded in the same way as (32). Term (i).d can be bounded by Corollary 1, giving that

$$\begin{aligned} \frac{1}{n}\sum _{l=1}^n E_a^l E_b^l \ge -c_0||\varvec{\varGamma }^*||_{\infty }\sqrt{\frac{\log t}{n}}, \end{aligned}$$

with probability at least $1-1/t$. All that remains is to bound the terms (i).a, (ii).a and (iii).a. Fortunately, these correspond to the population quantity $\varDelta {\mathbf {C}}_{ }^*$. Observing that this is just a quadratic form of 2n-dimensional Gaussian vector, we can applying Lemma 9. Doing so gives that

$$\begin{aligned} \frac{1}{2n}\left( \sum _{l=1}^n (Z_i^l)^2 + \sum _{l=1}^n (Z_j^l)^2 - 2\sum _{l=1}^n Z_i^lZ_j^l \right) \ge \frac{1}{2}\left( C_{i,i}^* + C_{j,j}^* - C_{i,j}^*\right) - 2c_0 \sigma _1\sqrt{\frac{\log t}{n}} \end{aligned}$$

with probability at least $1-1/t$. Combining all the bounds for (i)-(iii), using that ${\mathcal {E}}$ occurs with probability at least $1-p_2/d^3$, and selecting $t=d^3$, we can see that, with probability at least $1-c_1/d^3$

$$\begin{aligned} y'_{a,b}&\ge \frac{1}{2}(C_{i,i}^* + C_{j,j}^* - 2C_{i,j}^*) - \frac{1}{2|G^*_{i}|}y_T - \frac{1}{2|G^*_{j}|}y_T - c_1||\varvec{\varGamma }^*||_{\infty }\sqrt{\frac{\log d}{n |G^*_{i}| }} - c_2\sigma \sqrt{\frac{\log d}{n}}\\&\ge \frac{1}{2}\varDelta ({\mathbf {C}}_{ }^*) - \frac{1}{2|G^*_{i}|}y_T - \frac{1}{2|G^*_{j}|}y_T - c_1||\varvec{\varGamma }^*||_{\infty }\sqrt{\frac{\log d}{n |G^*_{i}| }} - c_2\sigma \sqrt{\frac{\log d}{n}}. \end{aligned}$$

$\square$

Appendix B: Some technical lemmas

Lemma 10

Let ${\mathbf {M}}$ be a $d\times d$ real, symmetric matrix of the form

$$\begin{aligned} {\mathbf {M}}= a{\mathbf {I}}+ b{\mathbf {1}}{\mathbf {1}}^T. \end{aligned}$$

where $a,b \in {\mathbb {R}}$ then ${\mathbf {M}}$ has eigenvalues $a+b$ with multiplicity 1 and a with multiplicity $d-1$. If $a,b > 0$, then ${\mathbf {M}}$ also has the property that

$$\begin{aligned} {\mathbf {M}}^{1/2} =\,&\sqrt{a}{\mathbf {I}}+ \frac{\sqrt{a+db} - \sqrt{a}}{d}{\mathbf {1}}{\mathbf {1}}^T,\\ {\mathbf {M}}^{-1} =\,&\frac{1}{a}{\mathbf {I}}- \frac{b}{a^2+abd}{\mathbf {1}}{\mathbf {1}}^T,\\ {\mathbf {M}}^{-1/2} =\,&\frac{1}{\sqrt{a}}{\mathbf {I}}- \frac{\sqrt{a+db} - \sqrt{a}}{d\sqrt{a^2 + dab}}{\mathbf {1}}{\mathbf {1}}^T. \end{aligned}$$

Proof

Using the Sherman-Morrison formula, a matrix of the form ${\mathbf {M}}= a{\mathbf {I}}+ b{\mathbf {1}}{\mathbf {1}}^T$, where $a,b > 0$ has the inverse

$$\begin{aligned} {\mathbf {M}}^{-1} = \frac{1}{a}{\mathbf {I}}- \frac{b}{a^2+abd}{\mathbf {1}}{\mathbf {1}}^T. \end{aligned}$$

Because ${\mathbf {M}}\succ 0$, all eigenvalues are strictly positive and denote by $\lambda _i$ and $q_i$ the eigenvalues and corresponding eigenvectors. Without loss of generality, let $q_i$ be orthonormal. Then we can write ${\mathbf {M}}= \sum _i \lambda _i {\mathbf {q}}_i{\mathbf {q}}_i^T$. By the form of ${\mathbf {M}}$, clearly $\frac{1}{\sqrt{d}}{\mathbf {1}}$ is always an eigenvector of ${\mathbf {M}}$ with eigenvalue $a+db$, so we can take $q_1 = \frac{1}{\sqrt{d}}{\mathbf {1}}$ and $\lambda _1 = 1$. The remaining $q_i$ span $({\mathbf {1}}{\mathbf {1}}^T)^{\perp }$ and have corresponding eigenvalues $\lambda _i = a$. Therefore,

$$\begin{aligned} {\mathbf {M}}^{1/2} = \frac{\sqrt{a+db}}{\sqrt{d}}{\mathbf {1}}{\mathbf {1}}^T + \sum _{i=2}^d \sqrt{a}{\mathbf {q}}_i{\mathbf {q}}_i^T. \end{aligned}$$

Because this eigen-decomposition is unique, the above gives

$$\begin{aligned} {\mathbf {M}}^{1/2} = \sqrt{a}{\mathbf {I}}+ \frac{\sqrt{a+db} - \sqrt{a}}{d}{\mathbf {1}}{\mathbf {1}}^T. \end{aligned}$$

Using the expression for ${\mathbf {M}}^{-1}$ given above, it follows that

$$\begin{aligned} {\mathbf {M}}^{-1/2} = \frac{1}{\sqrt{a}}{\mathbf {I}}- \frac{\sqrt{a+db} - \sqrt{a}}{d\sqrt{a^2 + dab}}{\mathbf {1}}{\mathbf {1}}^T. \end{aligned}$$

$\square$

The following result for quadratic forms of standard multivariate Gaussian random variables can be found in many forms in the literature (for example, Rudelson and Vershynin [34]).

Lemma 11

(Hanson-Wright inequality for Gaussian random variables) Let $\varvec{X}\sim N(0,{\mathbf {I}})$ be a d-dimensional random vector and let ${\mathbf {A}}$ be a $d \times d$ matrix in ${\mathbb {R}}^{d \times d}$. Then

$$\begin{aligned} {\mathbb {P}}\left( |\varvec{X}^T {\mathbf {A}}\varvec{X}- {\mathbb {E}}\left[ \varvec{X}^T {\mathbf {A}}\varvec{X}\right] | \ge t \right) \le 2 \exp \left( -c \min \left\{ \frac{t^2}{||{\mathbf {A}}||_F^2},\frac{t}{||{\mathbf {A}}||_2} \right\} \right) , \end{aligned}$$

for some absolute constant c.

In particular, the following corollary is useful.

Corollary 1

Let $\varvec{X}\sim N(0,{\mathbf {I}})$ be a d-dimensional random vector and let ${\mathbf {A}}$ be a $d \times d$ matrix in ${\mathbb {R}}^{d \times d}$. Then

$$\begin{aligned} {\mathbb {P}}\left( |\varvec{X}^T {\mathbf {A}}\varvec{X}- {\mathbb {E}}\left[ \varvec{X}^T {\mathbf {A}}\varvec{X}\right] | \ge ||{\mathbf {A}}||_F\sqrt{t} + ||{\mathbf {A}}||_2 t \right) \le 2 \exp \left( -c t \right) , \end{aligned}$$

for some absolute constant c. Equivalently,

$$\begin{aligned} |\varvec{X}^T {\mathbf {A}}\varvec{X}- {\mathbb {E}}\left[ \varvec{X}^T {\mathbf {A}}\varvec{X}\right] | \le c'\left( ||{\mathbf {A}}||_F\sqrt{\log t} + ||{\mathbf {A}}||_2 \log t \right) \end{aligned}$$

with probability at least $1-2/t$ for some absolute constant $c'$.

Below we are concerned with the rate of concentration in the spectral norm of a sample covariance matrix to its mean: $||{{\widehat{\varvec{\varSigma }}}} - \varvec{\varSigma }^*||_2$. If we write ${{\widehat{\varvec{\varSigma }}}} = \frac{1}{n}\varvec{X}^T\varvec{X}$, where $\varvec{X}$ refers to the $n\times d$ matrix in which the rows are the observations $\varvec{X}_i$, we see how such a result is directly applicable to the problem at hand. We repeat the statement of Gordon’s Theorem given in Vershynin [35] below as Proposition 1. We use the notation from Vershynin [35] of $s_{\min }$ and $s_{\max }$ to denote the smallest and largest singular values, respectively.

Proposition 1

Let $\varvec{X}$ be an $n \times d$ matrix whose entries are independent standard normal random variables. Then

$$\begin{aligned} \sqrt{n} - \sqrt{d} \le {\mathbb {E}}[s_{\min }(\varvec{X})] \le {\mathbb {E}}[s_{\max }(\varvec{X})] \le \sqrt{n} + \sqrt{d} \end{aligned}$$

Using the result on sub-Gaussian concentration of a Lipschitz function of independent random variables, we immediately obtain the following corollary (also given in Vershynin [35]).

Corollary 2

Let $\varvec{X}$ be an $n \times d$ matrix whose entries are independent standard normal random variables, then for every $t \ge 0$

$$\begin{aligned} \sqrt{n} - \sqrt{d} -t \le s_{\min }(\varvec{X}) \le s_{\max }(\varvec{X}) \le \sqrt{n} + \sqrt{d} + t \end{aligned}$$

with probability at least $1-2\exp (-t^2/2)$.

Proof

Observing that the functions $s_{\min }$ and $s_{\max }$ are 1-Lipschitz and using the sub-Gaussian tail bound, the result is immediate from the above. $\square$

Corollary 3

Let $\varvec{X}_i$, for $i=1,\dots ,n$, be a d-dimensional random vector sampled from $N(0,\varvec{\varSigma })$. Denoting ${\widehat{\varvec{\varSigma }}}_{ }:= n^{-1}\sum _{i=1}^n\varvec{X}_i\varvec{X}_i^\top$, we have that

$$\begin{aligned} \lambda _{\min }\left( {\widehat{\varvec{\varSigma }}}_{ }-\varvec{\varSigma }\right)&\ge \lambda _{\min }(\varvec{\varSigma })\left( \frac{d}{n} + \frac{2t\sqrt{d}}{n} + \frac{t^2}{n} - \frac{2(\sqrt{d} + t)}{\sqrt{n}} \right) ,\\ \lambda _{\max }\left( {\widehat{\varvec{\varSigma }}}_{ }-\varvec{\varSigma }\right)&\le \lambda _{\max }(\varvec{\varSigma })\left( \frac{d}{n} + \frac{2t\sqrt{d}}{n} + \frac{t^2}{n} + \frac{2(\sqrt{d} + t)}{\sqrt{n}} \right) , \end{aligned}$$

with probability at least $1-2\exp (-t^2/2)$.

Proof

This follows directly from Corollary 2. $\square$

Appendix C: Numerical results for unknown K

In this section we consider the exact same setup as in Sect. 6, but now we solve (22) using FORCE. For K unknown we do not have a readily available baseline comparison in high dimensions—prior work considers an ADMM based approach for (2), but not (22)—so we present only the results from FORCE and FORCE-P.

The main differences between the results displayed in Tables 3 and 4 and those in Sect. 6 are that (1) SDPNAL reached its iteration limit in some cases, leading to a higher average error than for K known, and (2) for the setting $(d,k,\rho ,\gamma ) = (500,100,0.3,3.0)$ the SNR was too low and FORCE did not converge. Otherwise, the results shown below, are very similar to those when K is known, and we refer the reader to the discussion in Sect. 6. The fact that we do not observe any significant difference in empirical performance between K fixed and K unknown—indeed K unknown appears to take longer in practice as intuition suggests—indicates that Theorem 1 either may not be tight or instances on which it achieves the worst-case bound are encountered with low probability.

Table 3 Benchmark results for low dimensional designs comparing FORCE and FORCE-P with MOSEK and SDPNAL+ with unknown K

Table 4 Benchmark results for high dimensional designs comparing FORCE and FORCE-P for K unknown