CUR LRA at Sublinear Cost Based on Volume Maximization

Abstract

A matrix algorithm runs at sublinear cost if it uses much fewer memory cells and arithmetic operations than the input matrix has entries. Such algorithms are indispensable for Big Data Mining and Analysis, where input matrices are so immense that one can only access a small fraction of all their entries. Typically, however, such matrices admit their Low Rank Approximation (LRA), which one can access and process at sublinear cost. Can, however, we compute LRA at sublinear cost? Adversary argument shows that no algorithm running at sublinear cost can output accurate LRA of worst case input matrices or even of the matrices of small families of our Appendix A, but we prove that some sublinear cost algorithms output a reasonably close LRA of a matrix W if (i) this matrix is sufficiently close to a low rank matrix or (ii) it is a Symmetric Positive Semidefinite (SPSD) matrix that admits LRA. In both cases supporting algorithms are deterministic and output LRA in its special form of CUR LRA, particularly memory efficient. The design of our algorithms and the proof of their correctness rely on the results of extensive previous study of CUR LRA in Numerical Linear Algebra using volume maximization. In case (i) we apply Cross-Approximation (C-A) iterations, running at sublinear cost and computing accurate LRA worldwide for more than a decade. We provide the first formal support for this long-known empirical efficiency assuming non-degeneracy of the initial submatrix of at least one C-A iteration. We cannot ensure non-degeneracy at sublinear cost for a worst case input but prove that it holds with a high probability (whp) for any initialization in the case of a random or randomized input. Empirically we can replace randomization with sparse multiplicative preprocessing of an input matrix, performed at sublinear cost. In case (ii) we make no additional assumptions about the input class of SPSD matrices admitting LRA or about initialization of our sublinear cost algorithms for CUR LRA, which promise to be practically valuable. We hope that proper combination of our deterministic techniques with randomized LRA methods, popular among Computer Science researchers, will lead them to further progress in LRA.

Appendices

Appendix

A Small Families of Hard Inputs for Sublinear Cost LRA

Any sublinear cost LRA algorithm fails on the following small input families.

Example 1

Define a family of $m\times n$ matrices of rank 1 (we call them $\delta$-matrices):

$$\{\varDelta _{i,j},~{i=1, \dots ,m;~j=1,\dots ,n}\}.$$

Also include the $m\times n$ null matrix $O_{m,n}$ into this family. Now fix any sublinear cost algorithm; it does not access the (i, j)th entry of its input matrices for some pair of i and j. Therefore it outputs the same approximation of the matrices $\varDelta _{i,j}$ and $O_{m,n}$, with an undetected error at least 1/2. Apply the same argument to the set of $mn+1$ small-norm perturbations of the matrices of the above family and to the $mn+1$ sums of the latter matrices with any fixed $m\times n$ matrix of low rank. Finally, the same argument shows that a posteriori estimation of the output errors of an LRA algorithm applied to the same input families cannot run at sublinear cost.

This example actually covers randomized LRA algorithms as well. Indeed suppose that with a positive constant probability an LRA algorithm does not access K entries of an input matrix. Apply this algorithm to two matrices of low rank whose difference at all these K entries is equal to a large constant C. Then, clearly, with a positive constant probability the algorithm has errors at least C/2 at at least K/2 of these entries.

B Definitions for Matrix Computations and a Lemma

Next we recall some basic definitions for matrix computations (cf. [GL13]).

$\mathbb C^{m\times n}$ is the class of $m\times n$ matrices with complex entries.

$I_s$ denotes the $s\times s$ identity matrix. $O_{q,s}$ denotes the $q\times s$ matrix filled with zeros.

$\mathrm {diag}(B_1,\dots ,B_k)=\mathrm {diag}(B_j)_{j=1}^k$ denotes a $k\times k$ block diagonal matrix with diagonal blocks $B_1,\dots ,B_k$.

$(B_1~|~\dots ~|~B_k)$ and $(B_1,\dots ,B_k)$ denote a $1\times k$ block matrix with blocks $B_1,\dots ,B_k$.

$W^T$ and $W^*$ denote the transpose and the Hermitian transpose of an $m\times n$ matrix $W=(w_{ij})_{i,j=1}^{m,n}$, respectively. $W^*=W^T$ if the matrix W is real.

For two sets $\mathcal I\subseteq \{1,\dots ,m\}$ and $\mathcal J\subseteq \{1,\dots ,n\}$ define the submatrices

$$\begin{aligned} W_{\mathcal I,:}:=(w_{i,j})_{i\in \mathcal I; j=1,\dots , n}, W_{:,\mathcal J}:=(w_{i,j})_{i=1,\dots , m;j\in \mathcal J},~ W_{\mathcal I,\mathcal J}:=(w_{i,j})_{i\in \mathcal I;j\in \mathcal J}. \end{aligned}$$

(B.1)

An $m\times n$ matrix W is unitary (also orthogonal when real) if $W^*W=I_n$ or $WW^*=I_m$.

Compact SVD of a matrix W, hereafter just SVD, is defined by the equations

$$\begin{aligned} \begin{array}{c} W=S_W\varSigma _WT_W^*, \\ \mathrm{where}~S_W^*S_W= T_W^*T_W=I_{\rho },~\varSigma _W:=\mathrm {diag}(\sigma _j(W))_{j=1}^{\rho },~\rho ={\mathrm {rank}(W)}, \end{array} \end{aligned}$$

(B.2)

$\sigma _j(W)$ denotes the jth largest singular value of W for $j=1,\dots ,\rho$; $\sigma _j(W)=0~\mathrm{for}~j>\rho$.

$||W||=||W||_2$, $||W||_F$, and $||W||_C$ denote spectral, Frobenius, and Chebyshev norms of a matrix W, respectively, such that (see [GL13, Section 2.3.2 and Corollary 2.3.2])

$$||W||=\sigma _1(W),~||W||_F^2:=\sum _{i,j=1}^{m,n}|w_{ij}|^2=\sum _{j=1}^{\mathrm {rank}(W)}\sigma _j^2(W),~ ||W||_C:=\max _{i,j=1}^{m,n}|w_{ij}|,$$

$$\begin{aligned} ||W||_C\le ||W||\le ||W||_F\le \sqrt{mn}~||W||_C,~ ||W||_F^2\le \min \{m,n\}~||W||^2. \end{aligned}$$

(B.3)

$W^+:=T_W\varSigma _W^{-1}S_W^*$ is the Moore–Penrose pseudo inverse of an $m\times n$ matrix W.

$$\begin{aligned} ||W^+||\sigma _{r}(W)=1 \end{aligned}$$

(B.4)

for a full rank matrix W.

A matrix W has $\epsilon$-rank at most $r>0$ for a fixed tolerance $\epsilon >0$ if there is a matrix $W'$ of rank r such that $||W'-W||/||W||\le \epsilon$. We write $\mathrm {nrank}(W)=r$ and say that a matrix W has numerical rank r if it has $\epsilon$-rank r for a small $\epsilon$.

Lemma 3

Let $G\in \mathbb C^{k\times r}$, $\varSigma \in \mathbb C^{r\times r}$ and $H\in \mathbb C^{r\times l}$ and let the matrices G, H and $\varSigma$ have full rank $r\le \min \{k,l\}$. Then $||(G\varSigma H)^+|| \le ||G^+||~||\varSigma ^+||~||H^+||$.

A proof of this well-known result is included in [LPa].

C The Volume and r-Projective Volume of a Perturbed Matrix

Theorem 13

Suppose that $W'$ and E are $k\times l$ matrices, $\mathrm {rank}(W')=r\le \min \{k,l\}$, $W=W'+E$, and $||E||\le \epsilon$. Then

$$\begin{aligned} \Big (1-\frac{\epsilon }{\sigma _r(W)}\Big )^{r} \le \prod _{j=1}^r\Big (1-\frac{\epsilon }{\sigma _j(W)}\Big ) \le \frac{v_{2,r}(W)}{v_{2,r}(W')}\le \prod _{j=1}^r\Big (1+\frac{\epsilon }{\sigma _j(W)}\Big )\le \Big (1+\frac{\epsilon }{\sigma _r(W)}\Big )^r. \end{aligned}$$

(C.1)

If $\min \{k,l\}=r$, then $v_2(W)=v_{2,r}(W)$, $v_2(W')=v_{2,r}(W')$, and

$$\begin{aligned} \Big (1-\frac{\epsilon }{\sigma _r(W)}\Big )^{r} \le \frac{v_2(W)}{v_2(W')}= \frac{v_{2,r}(W)}{v_{2,r}(W')}\le \Big (1+\frac{\epsilon }{\sigma _r(W)}\Big )^r. \end{aligned}$$

(C.2)

Proof

Bounds (C.1) follow because a perturbation of a matrix within a norm bound $\epsilon$ changes its singular values by at most $\epsilon$ (see [GL13, Corollary 8.6.2]). Bounds (C.2) follow because $v_2(M)=v_{2,r}(M)=\prod _{j=1}^r\sigma _j(M)$ for any $k\times l$ matrix M with $\min \{k,l\}=r$, in particular for $M=W'$ and $M=W=W'+E$.

If the ratio $\frac{\epsilon }{\sigma _r(W)}$ is small, then $\Big (1-\frac{\epsilon }{\sigma _r(W)}\Big )^{r}=1-O\Big (\frac{r\epsilon }{\sigma _r(W)}\Big )$ and $\Big (1+\frac{\epsilon }{\sigma _r(W)}\Big )^r= 1+O\Big (\frac{r\epsilon }{\sigma _r(W)}\Big )$, which shows that the relative perturbation of the volume is amplified by at most a factor of r in comparison to the relative perturbation of the r largest singular values.

D The Volume and r-Projective Volume of a Matrix Product

Theorem 14

(Cf. [OZ18]). [Examples 2 and 3 below show some limitations on the extension of the theorem.]

Suppose that $W=GH$ for an $m\times q$ matrix G and a $q\times n$ matrix H. Then

• (i) $v_2(W)=v_2(G)v_2(H)$ if $q=\min \{m,n\}$; $v_2(W)=0\le v_2(G)v_2(H)$ if $q<\min \{m,n\}$.

• (ii) $v_{2,r}(W)\le v_{2,r}(G)v_{2,r}(H)$ for $1\le r\le q$,

• (iii) $v_2(W)\le v_2(G)v_2(H)$ if $m=n\le q$.

Example 2

If G and H are unitary matrices and if $GH=O$, then $v_2(G)=v_2(H)=v_{2,r}(G)=v_{2,r}(H)=1$ and $v_2(GH)=v_{2,r}(GH)=0$ for all $r\le q$.

Example 3

If $G=(1~|~0)$ and $H=\mathrm {diag}(1,0)$, then $v_2(G)=v_2(GH)=1$ and $v_2(H)=0$.