Cache Oblivious Sparse Matrix Multiplication

Abstract

We study the problem of sparse matrix multiplication in the Random Access Machine and in the Ideal Cache-Oblivious model. We present a simple algorithm that exploits randomization to compute the product of two sparse matrices with elements over an arbitrary field. Let $A \in \mathbb {F}^{n \times n}$ and $C \in \mathbb {F}^{n \times n}$ be matrices with h nonzero entries in total from a field $\mathbb {F}$. In the RAM model, we are able to compute all the k nonzero entries of the product matrix $AC \in \mathbb {F}^{n \times n}$ using $\tilde{\mathcal {O}}(h + kn)$ time and $\mathcal {O}(h)$ space, where the notation $\tilde{\mathcal {O}}(\cdot )$ suppresses logarithmic factors. In the External Memory model, we are able to compute cache obliviously all the k nonzero entries of the product matrix $AC \in \mathbb {F}^{n \times n}$ using $\tilde{\mathcal {O}}(h/B + kn/B)$ I/Os and $\mathcal {O}(h)$ space. In the Parallel External Memory model, we are able to compute all the k nonzero entries of the product matrix $AC \in \mathbb {F}^{n \times n}$ using $\tilde{\mathcal {O}}(h/PB + kn/PB)$ time and $\mathcal {O}(h)$ space, which makes the analysis in the External Memory model a special case of Parallel External Memory for $P=1$. The guarantees are given in terms of the size of the field and by bounding the size of $\mathbb {F}$ as ${|}\mathbb {F}{|} > kn \log (n^2/k)$ we guarantee an error probability of at most $1{\text{/ }}n$ for computing the matrix product.

Omitted Proofs

Proof

(Lemma 2 ). (i) If $\langle a,c \rangle \ne 0$, then there exist $i,j \in [n]$ such that $u_i,v_j \ne 0$ and $\langle a_i,c_j \rangle \ne 0$, hence, $(AC)_{i,j} \ne 0$. If there is a nonzero entry then $\langle a,c \rangle \ne 0$ with probability at least $1- 2/{|}\mathbb {F}{|} + 1/{|}\mathbb {F}{|}^2$. This is equivalent of saying that if there is a nonzero entry then $\langle a,c \rangle = 0$ with probability at most $2/{|}\mathbb {F}{|} - 1/{|}\mathbb {F}{|}^2$. Without loss of generality, let $i_1=i_2=i$ and $j_1=j_2=j$. Considering a bigger submatrix with exactly one nonzero entry leaves the probability unchanged, while considering more nonzero entries will only increase the probability of $\langle a,c \rangle \ne 0$. Therefore, we consider the case where we want to isolate, with high probability, the location of a single nonzero entry in a submatrix of unit size. It follows that, in order to query the submatrix we have to perform the following inner product $\langle a,c \rangle = \langle u_ia_i,v_jc_j \rangle$, where u, v are chosen uniformly at random from $\mathbb {F}$. Since $\langle a_i,c_j \rangle \ne 0$ by hypothesis, we have that $\mathbf {Pr}(\langle a,c \rangle = 0) \ge 2/{|}\mathbb {F}{|} - 1/{|}\mathbb {F}{|}^2$.

(ii) If the submatrix of AC with indices $[i_1,i_2] \times [j_1,j_2]$ is all zero then $\langle a,c \rangle = 0$ with probability at least $1 - 2k\log (n^2/k)/{|}\mathbb {F}{|} + k\log (n^2/k)/{|}\mathbb {F}{|}^2$. By Lemma 1 this is true. If $\langle a,c \rangle = 0$ then the submatrix of AC with indices $[i_1,i_2] \times [j_1,j_2]$ is all zero with probability at least $1 - 2k\log (n^2/k)/{|}\mathbb {F}{|} + k\log (n^2/k)/{|}\mathbb {F}{|}^2$. That is, if $\langle a,c \rangle = 0$ then the submatrix has a nonzero entry with probability at most $2k\log (n^2/k)/{|}\mathbb {F}{|} - k\log (n^2/k))/{|}\mathbb {F}{|}^2$. Without loss of generality, let $i_1=i_2=i$ and $j_1=j_2=j$. We have that $\langle a,c \rangle = \langle ua_i,vc_j \rangle = 0$, where u, v are chosen uniformly at random from $\mathbb {F}$. Therefore, $\mathbf {Pr}(\langle a_i,c_j \rangle \ne 0) \ge 2/{|}\mathbb {F}{|} - 1/{|}\mathbb {F}{|}^2$. The latter is a lower bound on the probability to not detect a nonzero entry in the output matrix. A union bound over the $k \log (n^2/k)$ queries needed to isolate the k nonzero entries, gives us the probability to incur in at least one false negative. By considering its complement, the claim follows.   $\square$

Proof

(Lemma 3 ). (i) If $\langle a,c \rangle \ne 0$, then there exist $i,j \in [m]$ such that $u_i,v_j \ne 0$ and $\langle a_i,c_j \rangle \ne 0$, hence, $(AC)_{i,j} \ne 0$. If there is a nonzero entry then $\langle a,c \rangle \ne 0$ with probability at least $1- 1/{|}\mathbb {F}^*{|}$. This is equivalent of saying that if there is a nonzero entry then $\langle a,c \rangle = 0$ with probability at most $1/{|}\mathbb {F}^*{|}$. If $i_1=i_2=i$, $j_1=j_2=j$ and $\langle a_i,c_j \rangle \ne 0$ then $\langle a,c \rangle \ne 0$ since scaling vectors with random elements from $\mathbb {F}^*$ preserves non orthogonality. If $i_1<i_2$ and $j_1<j_2$ and the submatrix contains exactly one nonzero entry, the same reasoning applies. If the submatrix has $\ell >1$ nonzero entries, then, without loss of generality, there exist $a_1,\dots ,a_\ell ,c_1, \dots ,c_\ell$ such that $\langle u_1a_1,v_1c_1 \rangle + \dots + \langle u_\ell a_\ell ,v_\ell c_\ell \rangle = 0$ and $\langle a_i,c_j \rangle \ne 0$, for all $i,j \in [\ell ]$. That is, $\ell$ inner products that generate as many nonzero entries and produce a false negative when the submatrix is queried. By the linearity of the inner product, we have that $\langle u_1a_1,v_1c_1 \rangle + \dots + \langle u_\ell a_\ell ,v_\ell c_\ell \rangle = u_1v_1 \langle a_1,c_1 \rangle + \dots + u_\ell v_\ell \langle a_\ell , c_\ell \rangle$. Hence, the sum cancels whenever $u_i = - ( u_1v_1 \langle a_1,c_1 \rangle + \dots + u_\ell v_\ell \langle a_\ell , c_\ell \rangle )/ v_i \langle a_i,c_i \rangle$ for a generic $i \in [\ell ]$. Note that, such a $u_i$ is in $\mathbb {F}$ since fields guarantee the existence of additive and multiplicative inverses. The probability to choose $u_i$ such that it cancels the other inner products is the same as choosing an element from $\mathbb {F}^*$ uniformly at random, i.e. $1/{|}\mathbb {F}^*{|}$.

(ii) If the submatrix of AC with indices $[i_1,i_2] \times [j_1,j_2]$ is all zero then $\langle a,c \rangle = 0$ with probability at least $1 - k\log ((n^2/k) -1)/{|}\mathbb {F}^*{|}$. By Lemma 1 this is true. If $\langle a,c \rangle = 0$ then the submatrix of AC with indices $[i_1,i_2] \times [j_1,j_2]$ is all zero with probability at least $1 - k\log ((n^2/k) -1)/{|}\mathbb {F}^*{|}$. That is, if $\langle a,c \rangle = 0$ then the submatrix of AC has a nonzero entry with probability at most $k\log ((n^2/k) -1)/{|}\mathbb {F}^*{|}$. If $\langle a,c \rangle = 0$ and $i_1=i_2=i$, $j_1=j_2=j$ then $\langle a_i,c_j \rangle = 0$. The same reasoning applies for $i_1<i_2$, $j_1<j_2$ and exactly one nonzero entry in the submatrix. Let $\langle a,c \rangle = 0$ and suppose there exist $a_1,\dots ,a_\ell ,c_1, \dots ,c_\ell$ such that $\langle u_1a_1,v_1c_1 \rangle + \dots + \langle u_\ell a_\ell ,v_\ell c_\ell \rangle = 0$ and $\langle a_i,c_j \rangle \ne 0$, for all $i,j \in [\ell ]$. Hence, as in (i), the sum cancels with probability $1/{|}\mathbb {F}^*{|}$. The latter is a lower bound on the probability to not detect a nonzero entry in the output matrix. A union bound over the $k\log ((n^2/k) -1)$ queries needed to isolate the k nonzero entries, gives us the probability to incur in at least one false negative.³ By considering its complement, the claim follows.   $\square$