Advertisement

MADMM: A Generic Algorithm for Non-smooth Optimization on Manifolds

  • Artiom KovnatskyEmail author
  • Klaus Glashoff
  • Michael M. Bronstein
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9909)

Abstract

Numerous problems in computer vision, pattern recognition, and machine learning are formulated as optimization with manifold constraints. In this paper, we propose the Manifold Alternating Directions Method of Multipliers (MADMM), an extension of the classical ADMM scheme for manifold-constrained non-smooth optimization problems. To our knowledge, MADMM is the first generic non-smooth manifold optimization method. We showcase our method on several challenging problems in dimensionality reduction, non-rigid correspondence, multi-modal clustering, and multidimensional scaling.

Keywords

Matrix Completion Manifold Learning Euclidean Distance Matrix Manifold Constraint Sparse Principal Component Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgement

This research was supported by the ERC Starting Grant No. 307047 (COMET).

References

  1. 1.
    Dong, X., Frossard, P., Vandergheynst, P., Nefedov, N.: Clustering on multi-layer graphs via subspace analysis on Grassmann manifolds. Trans. Sig. Process. 62(4), 905–918 (2014)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Keshavan, R.H., Oh, S.: A gradient descent algorithm on the Grassman manifold for matrix completion (2009). arXiv:0910.5260
  3. 3.
    Boufounos, P.T., Baraniuk, R.G.: 1-bit compressive sensing. In: Proceedings of CISS (2008)Google Scholar
  4. 4.
    Kovnatsky, A., Bronstein, M.M., Bronstein, A.M., Glashoff, K., Kimmel, R.: Coupled quasi-harmonic bases. Comput. Graph. Forum 32(2), 439–448 (2013)CrossRefGoogle Scholar
  5. 5.
    Eynard, D., Kovnatsky, A., Bronstein, M.M., Glashoff, K., Bronstein, A.M.: Multimodal manifold analysis using simultaneous diagonalization of Laplacians. Trans. PAMI 37(12), 2505–2517 (2015)CrossRefGoogle Scholar
  6. 6.
    Cucuringu, M., Lipman, Y., Singer, A.: Sensor network localization by eigenvector synchronization over the Euclidean group. ACM Trans. Sensor Netw. 8(3), 19 (2012)CrossRefGoogle Scholar
  7. 7.
    Cucuringu, M., Singer, A., Cowburn, D.: Eigenvector synchronization, graph rigidity and the molecule problem. Inf. Inference 1(1), 21–67 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Arie-Nachimson, M., Kovalsky, S.Z., Kemelmacher-Shlizerman, I., Singer, A., Basri, R.: Global motion estimation from point matches. In: Proceedings of 3DIMPVT (2012)Google Scholar
  9. 9.
    Zhang, M., Fletcher, P.T.: Probabilistic principal geodesic analysis. In: Proceedings of NIPS (2013)Google Scholar
  10. 10.
    Journée, M., Bach, F., Absil, P.A., Sepulchre, R.: Low-rank optimization on the cone of positive semidefinite matrices. SIAM J. Optimization 20(5), 2327–2351 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Meyer, G., Bonnabel, S., Sepulchre, R.: Linear regression under fixed-rank constraints: a Riemannian approach. In: Proceedings of ICML (2011)Google Scholar
  12. 12.
    Boumal, N., Absil, P.A.: RTRMC: A Riemannian trust-region method for low-rank matrix completion. In: Procedings of NIPS, pp. 406–414 (2011)Google Scholar
  13. 13.
    Tan, M., Tsang, I.W., Wang, L., Vandereycken, B., Pan, S.J.: Riemannian pursuit for big matrix recovery. In: Proceedings of ICML (2014)Google Scholar
  14. 14.
    Shalit, U., Weinshall, D., Chechik, G.: Online learning in the manifold of low-rank matrices. In: Proceedings of NIPS (2010)Google Scholar
  15. 15.
    Absil, P.A., Gallivan, K.A.: Joint diagonalization on the oblique manifold for independent component analysis. In: Proceedings of ICASSP (2006)Google Scholar
  16. 16.
    Kleinsteuber, M., Shen, H.: Blind source separation with compressively sensed linear mixtures. Sig. Process. Lett. 19(2), 107–110 (2012)CrossRefGoogle Scholar
  17. 17.
    Higham, N.J.: Computing the nearest correlation matrix - a problem from finance. IMA J. Numer. Anal. 22(3), 329–343 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Smith, S.T.: Optimization techniques on Riemannian manifolds. Fields Inst. Commun. 3(3), 113–135 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Absil, P.A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2009)zbMATHGoogle Scholar
  20. 20.
    Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20(2), 303–353 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Absil, P.A., Baker, C.G., Gallivan, K.A.: Trust-region methods on Riemannian manifolds. Found. Comput. Math. 7(3), 303–330 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Alvarez, F., Bolte, J., Munier, J.: A unifying local convergence result for Newton’s method in Riemannian manifolds. Found. Comput. Math. 8(2), 197–226 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Boumal, N., Mishra, B., Absil, P.A., Sepulchre, R.: Manopt, a Matlab toolbox for optimization on manifolds. JMLR 15(1), 1455–1459 (2014)zbMATHGoogle Scholar
  24. 24.
    Candès, E., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58(3), 11 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Ozoliņš, V., Lai, R., Caflisch, R., Osher, S.: Compressed modes for variational problems in mathematics and physics. PNAS 110(46), 18368–18373 (2013)Google Scholar
  26. 26.
    Neumann, T., Varanasi, K., Theobalt, C., Magnor, M., Wacker, M.: Compressed manifold modes for mesh processing. Comput. Graphics Forum 33(5), 35–44 (2014)CrossRefGoogle Scholar
  27. 27.
    Cayton, L., Dasgupta, S.: Robust Euclidean embedding. In: Proceedings of ICML (2006)Google Scholar
  28. 28.
    Wang, L., Singer, A.: Exact and stable recovery of rotations for robust synchronization. Information and Inference (2013)Google Scholar
  29. 29.
    Huang, Q., Wang, F., Guibas, L.: Functional map networks for analyzing and exploring large shape collections. ACM Trans. Graphics 33(4), 36 (2014)Google Scholar
  30. 30.
    Kovnatsky, A., Bronstein, M.M., Bresson, X., Vandergheynst, P.: Functional correspondence by matrix completion. In: Proceedings of CVPR (2015)Google Scholar
  31. 31.
    Chen, X.: Smoothing methods for nonsmooth, nonconvex minimization. Math. Program. 134(1), 71–99 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Ferreira, O.P., Oliveira, P.R.: Subgradient algorithm on Riemannian manifolds. J. Optimization Theory Appl. 97(1), 93–104 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Ledyaev, Y., Zhu, Q.: Nonsmooth analysis on smooth manifolds. Trans. AMS 359(8), 3687–3732 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Borckmans, P.B., Selvan, S.E., Boumal, N., Absil, P.A.: A Riemannian subgradient algorithm for economic dispatch with valve-point effect. J. Comp. Applied Math. 255, 848–866 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Lai, R., Osher, S.: A splitting method for orthogonality constrained problems. J. Scientific Comput. 58(2), 431–449 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Rosman, G., Wang, Y., Tai, X., Kimmel, R., Bruckstein, A.M.: Fast regularization of matrix-valued images. In: Proceedings of Efficient Algorithms for Global Optimization Methods in Computer Vision (2011)Google Scholar
  37. 37.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2(1), 17–40 (1976)CrossRefzbMATHGoogle Scholar
  38. 38.
    Boyd, S., Parikk, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2010)CrossRefzbMATHGoogle Scholar
  39. 39.
    Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4(5), 303–320 (1969)MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Optimization. Academic Press, London, New York (1969)Google Scholar
  41. 41.
    Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, New York (1982)zbMATHGoogle Scholar
  42. 42.
    Andreani, R., Birgin, E.G., Martínez, J.M., Schuverdt, M.L.: On augmented Lagrangian methods with general lower-level constraints. SIAM J. Optimization 18(4), 1286–1309 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Lojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Wang, Y., Wotao, Y., Jinshan, Z.: Global Convergence of ADMM in Nonconvex Nonsmooth Optimization (2015). arXiv:1511.06324
  45. 45.
    Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: Proceedings of NIPS (2002)Google Scholar
  46. 46.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps and spectral techniques for embedding and clustering. In: Proceedings of NIPS (2001)Google Scholar
  47. 47.
    Ovsjanikov, M., Ben-Chen, M., Solomon, J., Butscher, A., Guibas, L.J.: Functional maps: a flexible representation of maps between shapes. ACM Trans. Graphics 31(4), 1–11 (2012)CrossRefGoogle Scholar
  48. 48.
    Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Numerical Geometry of Non-rigid Shapes. Springer, New York (2008)zbMATHGoogle Scholar
  49. 49.
    Kim, V.G., Lipman, Y., Funkhouser, T.: Blended intrinsic maps. Trans. Graphics 30, 79 (2011)Google Scholar
  50. 50.
    Bekkerman, R., Jeon, J.: Multi-modal clustering for multimedia collections. In: Proceedings of CVPR (2007)Google Scholar
  51. 51.
    Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)CrossRefzbMATHGoogle Scholar
  52. 52.
    Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: Proceedings of NIPS (2001)Google Scholar
  53. 53.
    Grant, M., Boyd, S.: Graph implementations for nonsmooth convex programs. In: Blondel, V., Boyd, S., Kimura, H. (eds.) Recent Advances in Learning and Control. LNCIS, vol. 371, pp. 95–110. Springer, Heidelberg (2008)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Artiom Kovnatsky
    • 1
    Email author
  • Klaus Glashoff
    • 1
  • Michael M. Bronstein
    • 1
  1. 1.Institute of Computational Science, Faculty of InformaticsUSI Universitá della Svizzera ItalianaLuganoSwitzerland

Personalised recommendations