Scalable algorithms for locally low-rank matrix modeling

  • Qilong GuEmail author
  • Joshua D. Trzasko
  • Arindam Banerjee
Regular Paper


We consider the problem of modeling data matrices with locally low-rank (LLR) structure, a generalization of the popular low-rank structure widely used in a variety of real-world application domains ranging from medical imaging to recommendation systems. While LLR modeling has been found to be promising in real-world application domains, limited progress has been made on the design of scalable algorithms for such structures. In this paper, we consider a convex relaxation of LLR structure and propose an efficient algorithm based on dual projected gradient descent (D-PGD) for computing the proximal operator. While the original problem is non-smooth, so that primal (sub)gradient algorithms will be slow, we show that the proposed D-PGD algorithm has geometrical convergence rate. We present several practical ways to further speed up the computations, including acceleration and approximate SVD computations. With experiments on both synthetic and real data from MRI (magnetic resonance imaging) denoising, we illustrate the superior performance of the proposed D-PGD algorithm compared to several baselines.


Locally low rank Projected gradient descent Geometrical convergence MRI 



The research was supported by NSF grants IIS-1563950, IIS-1447566, IIS-1447574, IIS-1422557, CCF-1451986, CNS- 1314560, IIS-0953274, IIS-1029711, CCF:CIF:Small:1318347, NASA grant NNX12AQ39A, “Mayo Clinic Discovery Translation Program”, and gifts from Adobe, IBM, and Yahoo.


  1. 1.
    Banerjee A, Chen S, Fazayeli F, Sivakumar V (2014) Estimation with norm regularization. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in neural information processing systems 27. Curran Associates, Inc., pp 1556–1564Google Scholar
  2. 2.
    Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci 2(1):183–202MathSciNetzbMATHGoogle Scholar
  3. 3.
    Bertsekas DP (1999) Nonlinear programming. Athena Scientific, BelmontzbMATHGoogle Scholar
  4. 4.
    Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2010) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn 3(1):1–122zbMATHGoogle Scholar
  5. 5.
    Cai J-F, Candès EJ, Shen Z (2010) A singular value thresholding algorithm for matrix completion. SIAM J Optim 20(4):1956–1982MathSciNetzbMATHGoogle Scholar
  6. 6.
    Candès EJ, Recht B (2009) Exact matrix completion via convex optimization. Found Comput Math 9(6):717–772MathSciNetzbMATHGoogle Scholar
  7. 7.
    Candès EJ, Sing-long CA, Trzasko JD (2013) Unbiased risk estimates for singular value thresholding and spectral estimators. IEEE Trans Signal Process 61(19):4643–4657MathSciNetzbMATHGoogle Scholar
  8. 8.
    Chen S, Banerjee A (2016) Structured matrix recovery via the generalized dantzig selector. In: Lee DD, Sugiyama M, Luxburg UV, Guyon I, Garnett R (eds) Advances in neural information processing systems 29. Curran Associates Inc., New York, pp 3252–3260Google Scholar
  9. 9.
    Chen X, Tseng P (2003) Non-interior continuation methods for solving semidefinite complementarity problems. Math Program 95(3):431–474MathSciNetzbMATHGoogle Scholar
  10. 10.
    Donoho D, Gavish M (2014) Minimax risk of matrix denoising by singular value thresholding. Ann Stat 42(6):2413–2440MathSciNetzbMATHGoogle Scholar
  11. 11.
    Goud S, Hu Y, Jacob M (2010) Real-time cardiac MRI using low-rank and sparsity penalties. In: ISBI, pp. 988–991Google Scholar
  12. 12.
    Gunasekar S, Banerjee A, Ghosh J (2015) Unified view of matrix completion under general structural constraints. Adv Neural Inf Process Syst 28:1180–1188Google Scholar
  13. 13.
    Haldar JP, Liang ZP (2010) Spatiotemporal imaging with partially separable functions: a matrix recovery approach. In: ISBI, pp. 716–719Google Scholar
  14. 14.
    Hoffman AJ (1952) On approximate solutions of systems of linear inequalities. J Res Natl Bur Stand 49(4):263–265MathSciNetGoogle Scholar
  15. 15.
    Horn RA, Johnson CR (1985) Matrix analysis, vol 169. Cambridge University Press, CambridgezbMATHGoogle Scholar
  16. 16.
    Koren Y (2008) Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 426–434Google Scholar
  17. 17.
    Lee J, Kim S, Lebanon G, Singer Y, Bengio S (2016) Llorma: local low-rank matrix approximation. J Mach Learn Res 17(15):1–24MathSciNetzbMATHGoogle Scholar
  18. 18.
    Lin Z, Liu R, Su Z (2011) Linearized alternating direction method with adaptive penalty for low-rank representation. Adv Neural Inf Process Syst 24(1):1–9Google Scholar
  19. 19.
    Luo ZQ, Tseng P (1993) Error bounds and convergence analysis of feasible descent methods: a general approach. Ann Oper Res 46–47(1):157–178MathSciNetzbMATHGoogle Scholar
  20. 20.
    Ma S, Goldfarb D, Chen L (2011) Fixed point and bregman iterative methods for matrix rank minimization. Math Program 128(1):321–353MathSciNetzbMATHGoogle Scholar
  21. 21.
    Moreau JJ (1962) Decomposition orthogonale d’un espace hilbertien selon deux cones mutuellement polaires. Comptes Rendus de l’Académie des Sci 255:238–240zbMATHGoogle Scholar
  22. 22.
    Negahban S, Wainwright MJ (2011) Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann Stat 39(2):1069–1097MathSciNetzbMATHGoogle Scholar
  23. 23.
    Nesterov Y (2004) Introductory lectures on convex optimization : a basic course, applied optimization. Kluwer Academic Publ, BostonzbMATHGoogle Scholar
  24. 24.
    O’Donoghue B, Candés E (2015) Adaptive restart for accelerated gradient schemes. Found Comput Math 15(3):715–732MathSciNetzbMATHGoogle Scholar
  25. 25.
    Parikh N, Boyd S (2014) Proximal algorithms. Found Trends Optim 1(3):127–239Google Scholar
  26. 26.
    Peng Z, Yan M, Yin W (2013) Parallel and distributed sparse optimization. In: 2013 Asilomar conference on signals, systems and computers, pp. 659–646Google Scholar
  27. 27.
    Recht B, Fazel M, Parrilo PA (2010) Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev 52(3):471–501MathSciNetzbMATHGoogle Scholar
  28. 28.
    Rockafellar RT (1970) Convex analysis. Princeton University Press, PrincetonzbMATHGoogle Scholar
  29. 29.
    Toh KC, Yun S (2010) An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pac J Optim 6:615–640MathSciNetzbMATHGoogle Scholar
  30. 30.
    Trzasko JD (2013) Exploiting local low-rank structure in higher-dimensional mri applications. In: Proceedings of SPIE, vol 8858, pp 885821–885828Google Scholar
  31. 31.
    Trzasko JD, Mostardi PM, Riederer SJ, Manduca A (2013) Estimating t1 from multichannel variable flip angle SPGR sequences. Magn Reson Med 69(6):1787–1794Google Scholar
  32. 32.
    Trzasko J, Manduca A (2011) Local versus global low-rank promotion in dynamic MRI series reconstruction. In: ISMRM, vol 24, p 4371Google Scholar
  33. 33.
    Watson GA (1992) Characterization of the subdifferential of some matrix norms. Linear Algebra Appl 170(C):33–45MathSciNetzbMATHGoogle Scholar
  34. 34.
    Yao Q, Kwok JT (2015) Colorization by patch-based local low-rank matrix completion. In: Proceedings of the twenty-ninth AAAI conference on artificial intelligence, AAAI’15. AAAI Press, pp. 1959–1965Google Scholar
  35. 35.
    Zhou Z, So AM-C (2015) A unified approach to error bounds for structured convex optimization problems, pp. 1–32. arXiv:1512.03518

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringUniversity of MinnesotaTwin CitiesUSA
  2. 2.Department of RadiologyMayo ClinicRochesterUSA

Personalised recommendations