Heuristics for exact nonnegative matrix factorization

Abstract

The exact nonnegative matrix factorization (exact NMF) problem is the following: given an m-by-n nonnegative matrix X and a factorization rank r, find, if possible, an m-by-r nonnegative matrix W and an r-by-n nonnegative matrix H such that $X = WH$. In this paper, we propose two heuristics for exact NMF, one inspired from simulated annealing and the other from the greedy randomized adaptive search procedure. We show empirically that these two heuristics are able to compute exact nonnegative factorizations for several classes of nonnegative matrices (namely, linear Euclidean distance matrices, slack matrices, unique-disjointness matrices, and randomly generated matrices) and as such demonstrate their superiority over standard multi-start strategies. We also consider a hybridization between these two heuristics that allows us to combine the advantages of both methods. Finally, we discuss the use of these heuristics to gain insight on the behavior of the nonnegative rank, i.e., the minimum factorization rank such that an exact NMF exists. In particular, we disprove a conjecture on the nonnegative rank of a Kronecker product, propose a new upper bound on the extension complexity of generic n-gons and conjecture the exact value of (i) the extension complexity of regular n-gons and (ii) the nonnegative rank of a submatrix of the slack matrix of the correlation polytope.

Appendices

Appendix: Sensitivity to the parameters $\alpha$ and ${\varDelta }t$

In this section, we show some numerical results to stress out that the heuristics are not too sensitive (in terms of number of exact NMF’s found) to the parameters $\alpha$ and ${\varDelta }t$ of the local search heuristic (Algorithm FR), as long as they are chosen sufficiently large; see Tables 8 and 9. This is the reason why we selected the rather conservative values of $\alpha = 0.99$ and ${\varDelta }t = 1$ in this paper.

Table 8 Comparison of different values of $\alpha$ with ${\varDelta }t=1$ combined with multi-start 2

Table 9 Comparison of different values of ${\varDelta }t$ with $\alpha = 0.99$ combined with multi-start 2

In practice however, it would be good to start the heuristics with smaller values for $\alpha$ and ${\varDelta }t$ and increase them progressively if the heuristic fails to identify exact NMF’s: for easily factorizable matrices (such as the randomly generated ones) it does not make sense to choose large parameters, while for difficult matrices choosing $\alpha$ and ${\varDelta }t$ too small does not allow the heuristics to find exact NMF’s because convergence of NMF algorithms can, in some cases, be too slow.

Parameters for simulated annealing

Table 10 shows the performance of SA for different initialization strategies described in Sect. 3.2 (for $T_0 = 0.1$, $T_{end} = 10^{-4}$, $J=2$, $N = 100$ and $K = 50$): it appears that SPARSE10 works on average the best hence we keep this initialization for SA. In particular, it is interesting to notice that SPARSE10 is able to compute exact NMF’s of 32-G while the other initializations have much more difficulties (only SPARSE00 finds one exact NMF).

Table 10 Comparison of the different initialization strategies combined with SA

Table 11 shows the performance for different values of $T_{end}$ (for $J=2$, $N = 100$ and $K = 50$): it appears that the value $T_{end} = 10^{-4}$ for the final temperature works well.

Table 11 Performance of simulated annealing for different values of $T_{end}$ ($J=2$, $N = 100$ and $K = 50$)

Table 12 shows the performance for different values of N and K, for $T_{end} = 10^{-4}$ and $J = 2$. It seems that $K = 50$ and $N = 100$ is a good compromise between number of exact NMF’s found and computational time.

Table 12 Performance of simulated annealing for different values of K and N ($T_{end} = 10^{-4}$ and $J = 2$)

Table 13 shows the performance for different values of J (for $T_{end} = 10^{-4}$, $K = 50$ and $N = 100$), and shows that $J = 2$ performs the best.

Table 13 Performance of simulated annealing for different values of J ($T_{end} = 10^{-4}$, $K = 50$ and $N = 100$)

Parameters for the rank-by-rank heuristic

Table 14 shows the performance of RBR for the different initialization strategies (for $N = 100$ and $K = 50$): SPARSE10 works on average the best. As for SA, it allows to compute exact NMF’s of 32-G (6/10) while all other initializations fail.

Table 14 Comparison of the different initialization strategies combined with RBR

Table 15 gives the results for several values of the parameters K and N. It is interesting to observe that when K gets larger, the heuristic performs rather poorly in some cases (e.g., for the UDISJ6 matrix). The reason is that when K increases, the heuristic tends to generate similar solutions: the ones obtained with Algorithm getRankPlusOne initialized with the best solution that can be obtained by combining the rank-$(k-1)$ solution with a rank-one one. In other words, the search domain that can be explored by RBR is reduced when K increases.

Table 15 Performance of the rank-by-rank heuristic for different values of K and N

Initialization for the hybridization

Again the best initialization strategy is SPARSE10. However, it is interesting to note that Hybrid is less sensitive to initialization than SA and RBR. In fact, except for 32-G with RNDCUBE and LEDM32 with SPARSE01, it was able to compute exact NMF’s in all situations. In other words, as shown in Table 16, Hybrid is a more robust strategy than RBR and SA although it is computationally more expensive on average.

Table 16 Comparison of the different initialization strategies combined with the hybridization between RBR and SA