Cache Oblivious Sparse Matrix Multiplication

Matrix multiplication is a fundamental operation in computer science and mathematics. Despite the effort, the computational complexity is still not settled, and it is not clear whether \(\mathcal {O}(n^2)\) operations are sufficient to multiply two dense \(n \times n\) matrices.

Given matrices \(A \in \mathbb {F}^{n \times n}\) and \(C \in \mathbb {F}^{n \times n}\), we define h as the number of nonzero entries in the input, i.e. \(h = \mathtt {nnz}(A) + \mathtt {nnz}(C)\), k as the number of nonzero entries in the output, i.e. \(k = \mathtt {nnz}(AC)\), where \(\mathtt {nnz}(A)\) denotes the number of nonzero entries in matrix A. Let \(A_{i,j}\) be the value of the entry in the matrix A with coordinates (i, j). We denote with \(A_{*,j}\) and \(C_{i,*}\) the j-th column of A and the i-th row of C respectively. Note that we can easily detect, by scanning the input matrices, null vectors. Hence, without loss of generality, we consider only the case where \(h \ge 2n\) and the rows of A (resp. columns of C) are not n-dimensional null vectors. Observe that, in contrast cancellations can lead to situations where \(k \le n\).¹ The algorithms presented in this paper can compute the product of matrices of arbitrary size and the bounds can be straightforwardly extended to rectangular matrices. However, for the ease of exposition, we restrict our analysis to square matrices. We denote with column major layout the lexicographic order where the entries of A are sorted by column first, and by row index within a column. Analogously, we denote with row major layout the order where the entries of A are sorted row-wise first, and column-wise within a row. Note that a row major layout can be obtained from column major layout by transposing A, and vice versa. Throughout this paper we use \(f(n) = \tilde{\mathcal {O}}(g(n))\) as shorthand for \(f(n) = \mathcal {O} (g(n) \log ^c g(n))\) for some constant c. The memory hierarchy we refer to is modeled by the I/O model by Aggarwal and Vitter [1], the Ideal Cache-Oblivious model by Frigo et al. [2] and the Parallel External Memory model by Arge et al. [3]. We denote with M the number of elements that can fit into internal memory, B the number of elements that can be transferred in a single block and P as the number of processors. The parameters M, and B are referred as the memory size and block size respectively. The Ideal Cache-Oblivious model resembles the I/O model except for memory and block size are unknown to the algorithm. Unless otherwise stated, it holds \(1 \le B \le M \ll h\). Note that, for our parallel algorithm, we consider a cache aware model since concurrency is nontrivial in external memory models whenever the block size B is unknown [4].

Williams and Yu [12] provided a simple output-sensitive algorithm for matrix multiplication. The intuition behind [12] is that nonzero entries in a submatrix of AC with indices in \([i_1,i_2] \times [j_1,j_2]\) can be detected by testing whether \(\langle a,c \rangle = 0\), where \( a = \sum _{k=i_1}^{i_2} u_k A_{k,*}\) and \(c = \sum _{k=j_1}^{j_2} v_k C_{*,k}\) are random (w.r.t \(u_k\) and \(v_k\)) linear combinations, i.e. sketches, of rows of A and columns of C respectively. A preprocessing phase, where prefix sums are involved, allows to compute sketches a and c of arbitrary size in linear time during the query process. When the input matrices are sparse, the prefix sums densify the matrices thus having to compute and store \(n^2\) elements. In addition, sparse matrices make nontrivial to compute linear combinations since row/column vectors are not explicitly stored.

We refine the analysis of [12] as follows. In order to detect the k positions (i, j) such that \((AC)_{i,j} \ne 0\), using binary search among the \(n^2\) feasible locations, we need at most \(k\log n^2\) comparisons. We note that the algorithms do not yield false positives when querying submatrices. That is, given an all-zero submatrix with related sketches a and c, it holds \(\langle a,c \rangle = 0\) always. This leads to the following lemma.

The algorithm from Theorem 1 computes k locations (i, j) to as many nonzero entries in \(AC \in \mathbb {F}^{n \times n}\). In order to compute \((AC)_{i,j}\) we can retrieve, using Formula (2), the entry value as follows \((AC)_{i,j} = \langle A_{i,*},C_{*,j} \rangle = \sum _k \big [(\mathcal {A}_{i,k} - \mathcal {A}_{i-1,k})(\mathcal {C}_{k,j} - \mathcal {C}_{k,j-1})\big ]/u_iv_j\) while querying unit length matrices.

We proceed to give guarantees on the probability of detecting non-zero entries in the output matrix and we study how altering the process of random generation alters the probability of detection. The guarantees are given in terms of the field size and not on the size of the matrix as, e.g., in [11]. Throughout the paper we gave no restriction on the field \(\mathbb {F}\). Nevertheless, when \(\mathbb {F}\) is infinite and countable, we require to sample from a finite subset of \( \mathbb {F}\). This constraint is justified since random variables cannot be uniformly distributed among infinite and countable sets. Fields, in contrast with other algebraic structures, guarantee the existence of the multiplicative inverse for elements of \(\mathbb {F}\), a property we use for proving the following lemmas.

\(\mathbf {Pr}(\langle a,c \rangle = \langle u_ia_i,v_jc_j \rangle = 0)\), with \(\langle a_i,c_j \rangle \ne 0\), is given by the probability of choosing either \(u_i\) or \(v_j\) zero uniformly at random from \(\mathbb {F}\). By altering the algorithm, such that random entries are now generated from \(\mathbb {F}^* = \mathbb {F}{\setminus }\{0\}\), we derive the following lemma.