Abstract
In this paper, based on the alternating nonnegative least squares framework, we present a new efficient method for nonnegative matrix factorization that uses a quadratic regularization projected Barzilai–Borwein (QRPBB) method to solve the subproblems. At each iteration, the QRPBB method first generates a point by solving a strongly convex quadratic minimization problem, which has a simple closed-form solution that is inexpensive to calculate, and then applies a projected Barzilai–Borwein method to update the solution of NMF. Global convergence result is established under mild conditions. Numerical comparisons of methods on both synthetic and real-world datasets show that the proposed method is efficient.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Both Reuters-21578 corpus and TDT-2 corpus in MATLAB format are available at http://www.cad.zju.edu.cn/home/dengcai/Data/TextData.html.
References
Barzilai J, Borwein JM (1988) Two-point step size gradient methods. IMA J Numer Anal 8(1):141–148
Berry MW, Browne M, Langville AN, Pauca VP, Plemmons RJ (2007) Algorithms and applications for approximate nonnegative matrix factorization. Computat Statist Data Anal 52(1):155–173
Bertsekas DP (1999) Nonlinear programming, 2nd edn. Athena Scientific, Belmont
Birgin EG, Martínez JM, Raydan M (2000) Nonmonotone spectral projected gradient methods on convex sets. SIAM J Optim 10(4):1196–1211
Birgin EG, Martínez JM, Raydan M (2012) Spectral projected gradient methods: Review and perspectives. http://www.ime.usp.br/~egbirgin/
Bonettini S, Zanella R, Zanni L (2009) A scaled gradient projection method for constrained image deblurring. Inverse Probl 25(1):015002
Cai D, He X, Han J (2005) Document clustering using locality preserving indexing. IEEE Trans Knowl Data Eng 17(12):1624–1637
Cai D, He X, Han J, Huang TS (2011) Graph regularized nonnegative matrix factorization for data representation. IEEE Trans Pattern Anal Mach Intell 33(8):1548–1560
Cichocki A, Zdunek R, Amari SI (2006) New algorithms for non-negative matrix factorization in applications to blind source separation. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp 621–624
Cichocki A, Zdunek R, Phan AH, Amari SI (2009) Nonnegative matrix and tensor factorizations: applications to exploratory multi-way data analysis and blind source separation. Wiley, Chichester
Cieri C, Graff D, Liberman M, Martey N, Strassel S (1999) The TDT-2 text and speech corpus. In: Proceedings of the DARPA Broadcast News Workshop, pp 57–60
Cores D, Escalante R, González-Lima M, Jimenez O (2009) On the use of the spectral projected gradient method for support vector machines. Comp Optim Appl 28(3):327–364
Dai YH, Zhang H (2001) Adaptive two-point stepsize gradient algorithm. Numer Algor 27(4):377–385
Dai YH, Liao LZ (2002) \(R\)-linear convergence of the Barzilai and Borwein gradient method. IMA J Numer Anal 22(1):1–10
Dai YH, Fletcher R (2005) Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming. Numer Math 100(1):21–47
Dai YH, Hager WW, Schittkowski K, Zhang H (2006) The cyclic Barzilai–Borwein method for unconstrained optimization. IMA J Numer Anal 26(3):604–627
Figueiredo MA, Nowak RD, Wright SJ (2007) Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE J Sel Top Signal Process 1(4):586–597
Fletcher R (2005) On the Barzilai–Borwein method. Optimization and control with applications, applied optimization. Springer, New York
Gong P, Zhang C (2012) Efficient nonnegative matrix factorization via projected Newton method. Pattern Recognit 45(9):3557–3565
Grippo L, Sciandrone M (2000) On the convergence of the block nonlinear Gauss–Seidel method under convex constraints. Oper Res Lett 26(3):127–136
Grippo L, Sciandrone M (2002) Nonmonotone globalization techniques for the Barzilai–Borwein gradient method. Comput Optim Appl 23(2):143–169
Guan N, Tao D, Luo Z, Yuan B (2012) NeNMF: an optimal gradient method for nonnegative matrix factorization. IEEE Trans Signal Process 60(6):2882–2898
Han L, Neumann M, Prasad U (2009) Alternating projected Barzilai-Borwein methods for nonnegative matrix factorization. Electron Trans Numer Anal 36(6):54–82
Hoyer PO (2004) Non-negative matrix factorization with sparseness constraints. J Mach Learn Res 5:1457–1469
Huang Y, Liu H, Zhou S (2013) A Barzilai–Borwein type method for stochastic linear complementarity problems. Numer Algor. doi:10.1007/s11075-013-9803-y
Kim H, Park H (2008) Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method. SIAM J Matrix Anal Appl 30(2):713–730
Kim D, Sra S, Dhillon IS (2007) Fast Newton-type methods for the least squares nonnegative matrix approximation problem. In: Proceedings of the 2007 SIAM International Conference on Data Mining, pp 343–354
Lee DD, Seung HS (2001) Algorithms for nonnegative matrix factorization. In: Advances in Neural Information Processing Systems, pp 556–562
Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788–791
Lewis DD, Yang Y, Rose TG, Li F (2004) RCV1: a new benchmark collection for text categorization research. J Mach Learn Res 5:361–397
Li SZ, Hou XW, Zhang HJ, Cheng QS (2001) Learning spatially localized, parts-based representation. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 207–212
Lin CJ (2007) Projected gradient methods for nonnegative matrix factorization. Neural Comput 19(10):2756–2779
Nesterov Y (1983) A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). In: Soviet Mathematics Doklady, pp 372–376
Paatero P, Tapper U (1994) Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5(2):111–126
Pascual-Montano A, Carazo JM, Kochi K, Lehmann D, Pascual-Marqui R (2006) Nonsmooth nonnegative matrix factorization (nsNMF). IEEE Trans Pattern Anal Mach Intell 28(3):403–415
Pauca VP (2004) Text mining using nonnegative matrix factorizations. In: Proceedings of the 2004 SIAM International Conference on Data Mining, pp 22–24
Raydan M (1997) The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J Optim 7(1):26–33
Shahnaz F, Berry MW, Pauca VP, Plemmons RJ (2006) Document clustering using nonnegative matrix factorization. J Inf Process Manage 42(2):373–386
Vavasis S (2009) On the complexity of nonnegative matrix factorization. SIAM J Optim 20(3):1364–1377
Xu W, Liu X, Gong Y (2003) Document clustering based on non-negative matrix factorization. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 267–273
Zdunek R, Cichocki A (2006) Non-negative matrix factorization with quasi-Newton optimization. In: Proceedings of the 8th Artificial Intelligence and Soft Computing, pp 870–879
Zhang H, Hager WW (2004) A nonmonotone line search technique and its application to unconstrained optimization. SIAM J Optim 14(4):1043–1056
Acknowledgments
The authors would like to thank the handling associate editor Professor Kristian Kersting and the anonymous referees for their constructive comments and useful suggestions. This work is supported by the National Natural Science Foundation of China (NNSFC) under Grant No. 61072144 and No. 61179040 and the Fundamental Research Funds for the Central Universities No. K50513100007.
Author information
Authors and Affiliations
Corresponding author
Additional information
Responsible editor: Kristian Kersting.
Rights and permissions
About this article
Cite this article
Huang, Y., Liu, H. & Zhou, S. Quadratic regularization projected Barzilai–Borwein method for nonnegative matrix factorization. Data Min Knowl Disc 29, 1665–1684 (2015). https://doi.org/10.1007/s10618-014-0390-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10618-014-0390-x