Advertisement

Nonnegative Matrix Factorization

  • Ke-Lin DuEmail author
  • M. N. S. Swamy
Chapter

Abstract

Low-rank matrix factorization or factor analysis is an important task that is helpful in the analysis of high-dimensional real-world data such as dimension reduction, data compression, feature extraction, and information retrieval. Nonnegative matrix factorization is a special low-rank factorization technique for nonnegative data. This chapter is dedicated to nonnegative matrix factorization. Other matrix decomposition methods, such as Nystrom method and CUR matrix decomposition, are also introduced in this chapter.

References

  1. 1.
    Banerjee, A., Merugu, S., Dhillon, I. S., & Ghosh, J. (2005). Clustering with Bregman divergences. Journal of Machine Learning Research, 6, 1705–1749.MathSciNetzbMATHGoogle Scholar
  2. 2.
    Berry, M. W., Browne, M., Langville, A. N., Pauca, V. P., & Plemmons, R. J. (2007). Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics and Data Analysis, 52(1), 155–173.MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Blumensath, T. (2016). Directional clustering through matrix factorization. IEEE Transactions on Neural Networks and Learning Systems, 27(10), 2095–2107.MathSciNetCrossRefGoogle Scholar
  4. 4.
    Cai, D., He, X., & Han, J. (2011). Locally consistent concept factorization for document clustering. IEEE Transactions on Knowledge and Data Engineering, 23(6), 902–913.CrossRefGoogle Scholar
  5. 5.
    Cai, D., He, X., Han, J., & Huang, T. S. (2011). Graph regularized nonnegative matrix factorization for data representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(8), 1548–1560.CrossRefGoogle Scholar
  6. 6.
    Chen, Y., Rege, M., Dong, M., & Hua, J. (2008). Non-negative matrix factorization for semi-supervised data clustering. Knowledge and Information Systems, 17(3), 355–379.CrossRefGoogle Scholar
  7. 7.
    Cichocki, A., & Anh-Huy, P. (2009). Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 92(3), 708–721.CrossRefGoogle Scholar
  8. 8.
    Cichocki, A., & Zdunek, R. (2006). NMFLAB for signal and image processing. Technical Report, Laboratory for Advanced Brain Signal Processing, BSI RIKEN, Saitama, Japan.Google Scholar
  9. 9.
    Cichocki, A., & Zdunek, R. (2007). Multilayer nonnegative matrix factorization using projected gradient approaches. International Journal of Neural Systems, 17(6), 431–446.CrossRefGoogle Scholar
  10. 10.
    Cichocki, A., Zdunek, R., & Amari, S. (2006). New algorithms for non-negative matrix factorization in applications to blind source separation. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) (Vol. 5, pp. 621–624). Toulouse, France.Google Scholar
  11. 11.
    Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., & Harshman, R. A., (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 416, 391–407.Google Scholar
  12. 12.
    Dhillon, I. S., & Sra, S. (2006). Generalized nonnegative matrix approximations with Bregman divergences. In Advances in neural information processing systems (Vol. 18, pp. 283–290).Google Scholar
  13. 13.
    Ding, C., Li, T., Peng, W., & Park, H. (2006). Orthogonal nonnegative matrix tri-factorizations for clustering. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’06) (pp. 126–135). Philadelphia, PA.Google Scholar
  14. 14.
    Ding, C., Li, T., & Jordan, M. I. (2010). Convex and semi-nonnegative matrix factorizations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(1), 45–55.CrossRefGoogle Scholar
  15. 15.
    Ding, C. H., He, X., & Simon, H. D. (2005). On the equivalence of nonnegative matrix factorization and spectral clustering. In Proceedings of the SIAM International Conference on Data Mining (pp. 606–610). Newport Beach, CA.Google Scholar
  16. 16.
    Donoho, D., & Stodden, V. (2003). When does nonnegative matrix factorization give a correct decomposition into parts? In Advances in neural information processing systems (Vol. 16, pp. 1141–1148). Vancouver, Canada; Cambridge, MA: MIT Press.Google Scholar
  17. 17.
    Drineas, P., & Mahoney, M. W. (2005). On the Nystrom method for approximating a gram matrix for improved kernel-based learning. Journal of Machine Learning Research, 6, 2153–2175.MathSciNetzbMATHGoogle Scholar
  18. 18.
    Drineas, P., Kannan, R., & Mahoney, M. W. (2006). Fast Monte Carlo algorithms for matrices. III: Computing a compressed approximate matrix decomposition. SIAM Journal on Computing, 36(1), 184–206.Google Scholar
  19. 19.
    Drineas, P., Mahoney, M. W., & Muthukrishnan, S. (2006). Subspace sampling and relative-error matrix approximation: Column-based methods. In Proceedings of the 10th Annual International Workshop on Randomization and Computation (RANDOM), LNCS (Vol. 4110, pp. 316–326). Berlin: Springer.Google Scholar
  20. 20.
    Drineas, P., Mahoney, M. W., & Muthukrishnan, S. (2007). Relative-error CUR matrix decompositions. Technical Report, Department of Mathematics, Yale University.Google Scholar
  21. 21.
    Drineas, P., Mahoney, M. W., & Muthukrishnan, S. (2008). Relative-error CUR matrix decompositions. SIAM Journal on Matrix Analysis and Applications, 30(2), 844–881.MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Drineas, P., Magdon-Ismail, M., Mahoney, M. W., & Woodruff, D. P. (2012). Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research, 13, 3441–3472.MathSciNetzbMATHGoogle Scholar
  23. 23.
    Gillis, N. (2012). Sparse and unique nonnegative matrix factorization through data preprocessing. Journal of Machine Learning Research, 13, 3349–3386.MathSciNetzbMATHGoogle Scholar
  24. 24.
    Gillis, N., & Luce, R. (2014). Robust near-separable nonnegative matrix factorization using linear optimization. Journal of Machine Learning Research, 15, 1249–1280.MathSciNetzbMATHGoogle Scholar
  25. 25.
    Goreinov, S. A., Tyrtyshnikov, E. E., & Zamarashkin, N. L. (1997). A theory of pseudoskeleton approximations. Linear Algebra and Its Applications, 261, 1–21.MathSciNetzbMATHCrossRefGoogle Scholar
  26. 26.
    Guan, N., Tao, D., Luo, Z., & Yuan, B. (2012). Online nonnegative matrix factorization with robust stochastic approximation. IEEE Transactions on Neural Networks and Learning Systems, 23(7), 1087–1099.CrossRefGoogle Scholar
  27. 27.
    Guan, N., Liu, T., Zhang, Y., Tao, D., & Davis, L. S. (2019). Truncated Cauchy non-negative matrix factorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(1), 246–259.CrossRefGoogle Scholar
  28. 28.
    Halko, N., Martinsson, P. G., & Tropp, J. A. (2011). Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2), 217–288.MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    He, Y., Lu, H., Huang, L., & Xie, S. (2014). Pairwise constrained concept factorization for data representation. Neural Networks, 52, 1–17.zbMATHCrossRefGoogle Scholar
  30. 30.
    He, Z., Xie, S., Zdunek, R., Zhou, G., & Cichocki, A. (2011). Symmetric nonnegative matrix factorization: Algorithms and applications to probabilistic clustering. IEEE Transactions on Neural Networks, 22(12), 2117–2131.CrossRefGoogle Scholar
  31. 31.
    Hosoda, K., Watanabe, M., Wersing, H., Korner, E., Tsujino, H., Tamura, H., et al. (2009). A model for learning topographically organized parts-based representations of objects in visual cortex: Topographic nonnegative matrix factorization. Neural Computation, 21, 2605–2633.MathSciNetzbMATHCrossRefGoogle Scholar
  32. 32.
    Hoyer, P. O. (2004). Nonnegative matrix factorization with sparseness constraints. Journal of Machine Learning Research, 5, 1457–1469.zbMATHGoogle Scholar
  33. 33.
    Hsieh, C.-J., & Dhillon, I. S. (2011). Fast coordinate descent methods with variable selection for non-negative matrix factorization. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1064–1072).Google Scholar
  34. 34.
    Keprt, A., & Snasel, V. (2005). Binary factor analysis with genetic algorithms. In Proceedings of the 4th IEEE International Workshop on Soft Computing as Transdisciplinary Science and Technology (WSTST), AINSC (Vol. 29, pp. 1259–1268). Muroran, Japan; Berlin: Springer.Google Scholar
  35. 35.
    Kim, D., Sra, S., & Dhillon, I. S. (2008). Fast projection-based methods for the least squares nonnegative matrix approximation problem. Statistical Analysis and Data Mining, 1, 38–51.MathSciNetCrossRefGoogle Scholar
  36. 36.
    Kimura, K., Kudo, M., & Tanaka, Y. (2016). A column-wise update algorithm for nonnegative matrix factorization in Bregman divergence with an orthogonal constraint. Machine Learning, 103, 285–306.MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Kompass, R. (2007). A generalized divergence measure for nonnegative matrix factorization. Neural Computation, 19(3), 780–791.MathSciNetzbMATHCrossRefGoogle Scholar
  38. 38.
    Kong, D., Ding, C., & Huang, H. (2011). Robust nonnegative matrix factorization using \(l_{2,1}\)-norm. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (pp. 673–682). Glasgow, UK.Google Scholar
  39. 39.
    Kuang, D., Ding, C., & Park, H. (2012). Symmetric nonnegative matrix factorization for graph clustering. In Proceedings of the 12th SIAM International Conference on Data Mining (pp. 106–117). Anaheim, CA.Google Scholar
  40. 40.
    Laurberg, H., Christensen, M. G., Plumbley, M. D., Hansen, L. K., & Jensen, S. H. (2008). Theorems on positive data: on the uniqueness of NMF. Computational Intelligence and Neuroscience, 2008, Article ID 764206.Google Scholar
  41. 41.
    Lee, D. D., & Seung, H. S. (1999). Learning the parts of objects by nonnegative matrix factorization. Nature, 401(6755), 788–791.zbMATHCrossRefGoogle Scholar
  42. 42.
    Lee, H., Yoo, J., & Choi, S. (2010). Semi-supervised nonnegative matrix factorization. IEEE Signal Processing Letters, 17(1), 4–7.CrossRefGoogle Scholar
  43. 43.
    Li, L., Lebanon, G., & Park, H. (2012). Fast Bregman divergence NMF using Taylor expansion and coordinate descent. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 307–315).Google Scholar
  44. 44.
    Li, L.-X., Wu, L., Zhang, H.-S., & Wu, F.-X. (2014). A fast algorithm for nonnegative matrix factorization and its convergence. IEEE Transactions on Neural Networks and Learning Systems, 25(10), 1855–1863.CrossRefGoogle Scholar
  45. 45.
    Li, M., Bi, W., Kwok, J. T., & Lu, B.-L. (2015). Large-scale Nystrom kernel matrix approximation using randomized SVD. IEEE Transactions on Neural Networks and Learning Systems, 26(1), 152–164.MathSciNetCrossRefGoogle Scholar
  46. 46.
    Li, X., & Pang, Y. (2010). Deterministic column-based matrix decomposition. IEEE Transactions on Knowledge and Data Engineering, 22(1), 145–149.MathSciNetCrossRefGoogle Scholar
  47. 47.
    Lin, C.-J. (2007). Projected gradients for non-negative matrix factorization. Neural Computation, 19(10), 2756–2779.MathSciNetzbMATHCrossRefGoogle Scholar
  48. 48.
    Lin, C.-J. (2007). On the convergence of multiplicative update algorithms for non-negative matrix factorization. IEEE Transactions on Neural Networks, 18(6), 1589–1596.CrossRefGoogle Scholar
  49. 49.
    Liu, H., Wu, Z., Li, X., Cai, D., & Huang, T. S. (2012). Constrained nonnegative matrix factorization for image representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7), 1299–1311.CrossRefGoogle Scholar
  50. 50.
    Liu, H., Li, X., & Zheng, X. (2013). Solving non-negative matrix factorization by alternating least squares with a modified strategy. Data Mining and Knowledge Discovery, 26(3), 435–451.MathSciNetzbMATHCrossRefGoogle Scholar
  51. 51.
    Liu, T., Gong, M., & Tao, D. (2017). Large-cone nonnegative matrix factorization. IEEE Transactions on Neural Networks and Learning Systems, 28(9), 2129–2142.MathSciNetGoogle Scholar
  52. 52.
    Mirzal, A. (2014). A convergent algorithm for orthogonal nonnegative matrix factorization. Journal of Computational and Applied Mathematics, 260, 149–166.MathSciNetzbMATHCrossRefGoogle Scholar
  53. 53.
    Paatero, P., & Tapper, U. (1994). Positive matrix factorization: A nonnegative factor model with optimal utilization of error estimates of data values. Environmetrics, 5(2), 111–126.CrossRefGoogle Scholar
  54. 54.
    Pascual-Montano, A., Carazo, J. M., Kochi, K., Lehmann, D., & Pascual-Marqui, R. D. (2006). Nonsmooth nonnegative matrix factorization (nsNMF). EEE Transactions on Pattern Analysis and Machine Intelligence, 28(3), 403–415.CrossRefGoogle Scholar
  55. 55.
    Peharz, R., & Pernkopf, F. (2012). Sparse nonnegative matrix factorization with \(l^0\)-constraints. Neurocomputing, 80, 38–46.CrossRefGoogle Scholar
  56. 56.
    Sajda, P., Du, S., Brown, T. R., Stoyanova, R., Shungu, D. C., Mao, X., et al. (2004). Nonnegative matrix factorization for rapid recovery of constituent spectra in magnetic resonance chemical shift imaging of the brain. IEEE Transactions on Medical Imaging, 23(12), 1453–1465.CrossRefGoogle Scholar
  57. 57.
    Vavasis, S. A. (2009). On the complexity of nonnegative matrix factorization. SIAM Journal on Optimization, 20(3), 1364–1377.MathSciNetzbMATHCrossRefGoogle Scholar
  58. 58.
    Wang, D., & Lu, H. (2013). On-line learning parts-based representation via incremental orthogonal projective non-negative matrix factorization. Signal Processing, 93, 1608–1623.CrossRefGoogle Scholar
  59. 59.
    Wang, F., Li, T., & Zhang, C. (2008). Semi-supervised clustering via matrix factorization. In Proceedings of the SIAM International Conference on Data Mining (pp. 1–12). Atlanta, GA.Google Scholar
  60. 60.
    Wang, J. J.-Y., & Gao, X. (2015). Max–min distance nonnegative matrix factorization. Neural Networks, 61, 75–84.zbMATHCrossRefGoogle Scholar
  61. 61.
    Wang, S., & Zhang, Z. (2013). Improving CUR matrix decomposition and the Nystrom approximation via adaptive sampling. Journal of Machine Learning Research, 14, 2729–2769.MathSciNetzbMATHGoogle Scholar
  62. 62.
    Wang, S., Zhang, Z., & Zhang, T. (2016). Towards more efficient SPSD matrix approximation and CUR matrix decomposition. Journal of Machine Learning Research, 17, 1–49.MathSciNetzbMATHGoogle Scholar
  63. 63.
    Williams, C., & Seeger, M. (2001). Using the Nystrom method to speedup kernel machines. In T. Leen, T. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems (Vol. 13, pp. 682–690). Cambridge, MA, USA: MIT Press.Google Scholar
  64. 64.
    Wu, X., Kumar, V., Quinlan, J. R., Ghosh, J., Yang, Q., Motoda, H., et al. (2008). Top 10 algorithms in data mining. Knowledge and Information Systems, 14(1), 1–37.CrossRefGoogle Scholar
  65. 65.
    Xu, W., & Gong, Y. (2004). Document clustering by concept factorization. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 202–209).Google Scholar
  66. 66.
    Yang, S., & Yi, Z. (2010). Convergence analysis of non-negative matrix factorization for BSS algorithm. Neural Processing Letters, 31, 45–64.CrossRefGoogle Scholar
  67. 67.
    Yang, S., Yi, Z., Ye, M., & He, X. (2014). Convergence analysis of graph regularized non-negative matrix factorization. IEEE Transactions on Knowledge and Data Engineering, 26(9), 2151–2165.CrossRefGoogle Scholar
  68. 68.
    Yang, Z., & Laaksonen, J. (2007). Multiplicative updates for non-negative projections. Neurocomputing, 71, 363–373.CrossRefGoogle Scholar
  69. 69.
    Yang, Z., & Oja, E. (2010). Linear and nonlinear projective nonnegative matrix factorization. IEEE Transactions on Neural Networks, 21(5), 734–749.CrossRefGoogle Scholar
  70. 70.
    Yokota, T., Zdunek, R., Cichocki, A., & Yamashita, Y. (2015). Smooth nonnegative matrix and tensor factorizations for robust multi-way data analysis. Signal Processing, 113, 234–249.CrossRefGoogle Scholar
  71. 71.
    Zafeiriou, S., & Petrou, M. (2010). Nonlinear non-negative component analysis algorithms. IEEE Transactions on Image Processing, 19(4), 1050–1066.MathSciNetzbMATHCrossRefGoogle Scholar
  72. 72.
    Zafeiriou, S., Tefas, A., Buciu, I., & Pitas, I. (2006). Exploiting discriminant information in nonnegative matrix factorization with application to frontal face verification. IEEE Transactions on Neural Networks, 17(3), 683–695.CrossRefGoogle Scholar
  73. 73.
    Zdunek, R., & Cichocki, A. (2007). Nonnegative matrix factorization with constrained second-order optimization. Signal Processing, 87, 1904–1916.zbMATHCrossRefGoogle Scholar
  74. 74.
    Zdunek, R., & Cichocki, A. (2008). Fast nonnegative matrix factorization algorithms using projected gradient approaches for large-scale problems. Computational Intelligence and Neuroscience, 2008, Article ID 939567.Google Scholar
  75. 75.
    Zhang, J., Wei, L., Feng, X., Ma, Z., & Wang, Y. (2008). Pattern expression nonnegative matrix factorization: Algorithm and applications to blind source separation. Computational Intelligence and Neuroscience, 2008, Article ID 168769.Google Scholar
  76. 76.
    Zhang, X., Zong, L., Liu, X., & Luo, J. (2016). Constrained clustering with nonnegative matrix factorization. IEEE Transactions on Neural Networks and Learning Systems, 27(7), 1514–1526.MathSciNetCrossRefGoogle Scholar
  77. 77.
    Zhang, Z., & Zhao, K. (2013). Low-rank matrix approximation with manifold regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7), 1717–1729.CrossRefGoogle Scholar
  78. 78.
    Zhao, R., & Tan, V. Y. F. (2017). Online nonnegative matrix factorization with outliers. IEEE Transactions on Signal Processing, 65(3), 555–570.MathSciNetzbMATHCrossRefGoogle Scholar
  79. 79.
    Zhou, G., Xie, S., Yang, Z., Yang, J.-M., & He, Z. (2011). Minimum-volume-constrained nonnegative matrix factorization: Enhanced ability of learning parts. IEEE Transactions on Neural Networks, 22(10), 1626–1637.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringConcordia UniversityMontrealCanada
  2. 2.Xonlink Inc.HangzhouChina

Personalised recommendations