Nonmonotone inexact restoration approach for minimization with orthogonality constraints

Abstract

Minimizing a differentiable function restricted to the set of matrices with orthonormal columns finds applications in several fields of science and engineering. In this paper, we propose to solve this problem through a nonmonotone variation of the inexact restoration method consisting of two phases: restoration phase, aimed to improve feasibility, and minimization phase, aimed to decrease the function value in the tangent set of the constraints. For this, we give a suitable characterization of the tangent set of the orthogonality constraints, allowing us to (i) deal with the minimization phase efficiently and (ii) employ the Cayley transform to bring a point in the tangent set back to feasibility, leading to a SVD-free restoration phase. Based on previous global convergence results for the nonmonotone inexact restoration algorithm, it follows that any limit point of the sequence generated by the new algorithm is stationary. Moreover, numerical experiments on three different classes of the problem indicate that the proposed method is reliable and competitive with other recently developed methods.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. 1.

    Available at http://www.math.ucla.edu/~wotaoyin/papers/feasible_method_matrix_manifold.html

  2. 2.

    http://www.manopt.org

References

  1. 1.

    Abrudan, T., Eriksson, J., Koivunen, V.: Steepest descent algorithms for optimization under unitary matrix constraint. IEEE Trans. Signal Process. 56(3), 1134–1147 (2008)

    MathSciNet  MATH  Google Scholar 

  2. 2.

    Absil, P.-A., Baker, C.G., Gallivan, K.A.: Trust-region methods on Riemannian manifolds. Found. Comput. Math. 7(3), 303–330 (2007)

    MathSciNet  MATH  Google Scholar 

  3. 3.

    Absil, P.A., Mahony, R., Sepulchre, R.: Optimization algorithms on matrix manifolds. Princeton University Press, Princeton (2008)

  4. 4.

    Absil, P.A., Malick, J.: Projection-like retractions on matrix manifolds. SIAM J. Optim. 22(1), 135–158 (2012)

    MathSciNet  MATH  Google Scholar 

  5. 5.

    Arouxét, B., Echebest, N.E., Pilotta, E.A.: Inexact Restoration method for nonlinear optimization without derivatives. J. Comput. Appl. Math. 290(15), 26–43 (2015)

    MathSciNet  MATH  Google Scholar 

  6. 6.

    Barzilai, J., Borwein, J.M.: Two point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)

    MathSciNet  MATH  Google Scholar 

  7. 7.

    Birgin, E.G., Bueno, L.F., Martínez, J.M.: Assessing the reability of general-purpose inexact restoration methods. J. Comput. Appl. Math. 282, 1–16 (2015)

    MathSciNet  MATH  Google Scholar 

  8. 8.

    Birgin, E.G., Martínez, J.M., Raydan, Marcos: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim. 10(4), 1196–1211 (2000)

    MathSciNet  MATH  Google Scholar 

  9. 9.

    Birgin, E.G., Martínez, J.M.: Local convergence of an inexact-restoration method and numerical experiments. J. Optim. Theory Appl. 127(2), 229–247 (2005)

    MathSciNet  MATH  Google Scholar 

  10. 10.

    Bueno, L.F., Martínez, J.M.: On the complexity of an inexact restoration method for constrained optimization. SIAM J. Optim. 30(1), 80–101 (2020)

    MathSciNet  MATH  Google Scholar 

  11. 11.

    Boumal, N., Mishra, B., Absil, P.-A., Sepulchre, R.: MAnopt, a Matlab Toolbox for optimization on manifolds. J. Mach. Learn. Res. 15, 1455–1459 (2014)

    MATH  Google Scholar 

  12. 12.

    Bueno, L.F., Friedlander, A., Martínez, J.M., Sobral, F.N.C.: Inexact restoration method for derivative-free optimization with smooth constraints. SIAM J. Optim. 23(2), 1189–1213 (2013)

    MathSciNet  MATH  Google Scholar 

  13. 13.

    Bueno, L.F., Haeser, G., Martínez, J.M.: A flexible inexact-restoration method for constrained optimization. J. Optim. Theory Appl. 165, 188–208 (2015)

    MathSciNet  MATH  Google Scholar 

  14. 14.

    Cancès, E., Chakir, R., Maday, Y.: Numerical analysis of nonlinear eigenvalue problems. J. Sci. Comput. 45, 90–117 (2010)

    MathSciNet  MATH  Google Scholar 

  15. 15.

    Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)

    MathSciNet  MATH  Google Scholar 

  16. 16.

    Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20(2), 303–353 (1998)

    MathSciNet  MATH  Google Scholar 

  17. 17.

    Francisco, J.B., Gonçalves, D.S., Bazán, F.S.V., Paredes, L.L.T.: Nonmonotone Inexact Restoration Method for nonlinear programming, Computational Optimization and Applications to appear. https://doi.org/10.1007/s10589-019-00129-2(2019)

  18. 18.

    Francisco, J.B., Viloche Bazán, F.S.: Nonmonotone algorithm for minimization on closed sets with application to minimization on Stiefel manifolds. J. Comp. and Appl. Math. 236(10), 2717–2727 (2012)

    MathSciNet  MATH  Google Scholar 

  19. 19.

    Francisco, J.B., Bazán, F.S.V., Weber Mendonça, M.: Non-monotone algorithm for minimization on arbitrary domains with applications to large-scale orthogonal Procrustes problem, Appl. Num. Math. 112, 51–64 (2017)

    MATH  Google Scholar 

  20. 20.

    Francisco, J.B., Martínez, J.M., Martínez, L., Pisnitchenko, F.: Inexact restoration method for minimization problems arising in electronic structure calculations. Comput. Optim. Appl. 50, 555–590 (2011)

    MathSciNet  MATH  Google Scholar 

  21. 21.

    Fischer, A., Friedlander, A.: A new line search inexact restoration approach for nonlinear programming. Comput. Optim. Appl. 46, 333–346 (2010)

    MathSciNet  MATH  Google Scholar 

  22. 22.

    Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th ed. The Johns Hopkins University Press (2013)

  23. 23.

    Helgaker, T., JøRgensen, J., Olsen, J.: Electronic - Structure theory. Wiley, New York (2000)

  24. 24.

    Janin, R.: Directional derivative of the marginal function in non linear programming. In: Sensitivity, Stability and Parametric Analysis, Math. Program. Stud., pp. 110–126. Springer, Berlin (1984)

  25. 25.

    Jiang, B., Dai, Y.H.: A framework of constraint preserving update schemes for optimization on Stiefel manifold. Math. Program., Ser. A 153(2), 535–575 (2015)

    MathSciNet  MATH  Google Scholar 

  26. 26.

    Journée, M., Nesterov, Y., Richtárik, P., Sepulchre, R.: Generalized power method for sparse principal component analysis. J. Mach. Learn. Res. 11, 517–553 (2010)

    MathSciNet  MATH  Google Scholar 

  27. 27.

    Manton, J.H.: Optimization algorithms exploiting unitary constraints. IEEE Trans. Signal Process. 50(3), 635–650 (2002)

    MathSciNet  MATH  Google Scholar 

  28. 28.

    Martínez, J.M., Pilotta, E.A.: Inexact restoration algorithm for constrained optimization. J. Optim. Theory App. 104(1), 135–163 (2000)

    MathSciNet  MATH  Google Scholar 

  29. 29.

    Martínez, J.M., Svaiter, B.F.: A practical optimality condition without constraint qualifications for nonlinear programming. J. Optimiz. Theory App. 118 (1), 117–133 (2003)

    MathSciNet  MATH  Google Scholar 

  30. 30.

    Nishimori, Y., Akaho, S.: Learning algorithms utilizing quasi-geodesic flows on the Stiefel manifold. Neurocomputing 67, 106–135 (2005)

    Google Scholar 

  31. 31.

    Shariff, M.: A constrained conjugate gradient method and the solution of linear equations. Comp. Mathem. Appl. 30(11), 25–37 (1995)

    MathSciNet  MATH  Google Scholar 

  32. 32.

    Karas, E. W., Pilotta, E., Ribeiro, A.: Numerical comparison of merit function with filter criterion in inexact restoration algorithms using hard-spheres problems. Comput. Optim. Appl. 44, 427–441 (2009)

    MathSciNet  MATH  Google Scholar 

  33. 33.

    Kaya, C.Y.: Inexact restoration for Runge–Kutta discretization of optimal control problems. SIAM J. Numer. Anal. 48(4), 1492–1517 (2010)

    MathSciNet  MATH  Google Scholar 

  34. 34.

    Kohn, W.: Nobel lecture: electronic structure of matter–wave functions and density functionals. Rev. Modern Phys. 71(5), 1253–1266 (1999)

    Google Scholar 

  35. 35.

    Krejić, N., Martínez, J.M.: Inexact restoration approach for minimization with inexact evaluation of the objective function. Mathematics of Computation 85, 1775–1791 (2016)

    MathSciNet  MATH  Google Scholar 

  36. 36.

    Wen, Z., Yin, W.: A feasible method for optimization with orthogonality constraints. Math. Program. Ser. A, 142, 397–434 (2013)

  37. 37.

    Zhang, H., Hager, W.: A nonmonotone line search technique and its application to unconstrained optimization. SIAM J. Optimiz. 14(4), 1043–1056 (2004)

    MathSciNet  MATH  Google Scholar 

  38. 38.

    Zhao, Z., Bai, Z.-J., Jin, X.-Q.: A Riemannian Newton algorithm for nonlinear eigenvalue problems. SIAM J. Matrix Anal. Appl. 36(2), 752–774 (2015)

    MathSciNet  MATH  Google Scholar 

  39. 39.

    Zhu, X.: A feasible filter method for the nearest low-rank correlation matrix problem. Numer. Algorithm. 69, 763–784 (2015)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

D.S.G thanks CAPES/Print Processes 88881.310538/2018-01 and 88887.465828/2019-00 which allow him to present part of this work at ICCOPT 2019 and supported his period as visiting professor at École Polytechnique where part of this work was carried out. We also thank anonymous referees whose comments helped to improve the quality of this work.

Funding

J.B.F., D.S.G, and F.S.V.B are grateful to CNPq by the financial support (Grants n. 421386/2016-9 and 308523/2017-2). L.L.T.P would like to thank CAPES for the scholarship during her Ph.D. at Universidade Federal de Santa Catarina. This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Juliano B. Francisco.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix

A Tables of numerical results

This appendix gathers the tables with numerical results for the experiments discussed in Sections 6.36.4, and 6.5. The symbol “–” means that the solver could not achieve \(\|\tilde {P}_{S_{k}} (\nabla f(Y_{k})) \| < \varepsilon = 10^{-6}\), using less than IT\(_{\max \limits }=2,000\) iterations and FE\(_{\max \limits }=4,000\) function evaluations.

Only for test problems of Table 7, the maximum number of iterations was increased to IT\(_{\max \limits }=5,000\) and function evaluations to IT\(_{\max \limits }=10,000\). Moreover, for this last case, we set M = 0 and \(\delta _{CG} = 10^{-4} \| P_{S_{0}} (Y_{0}) \|\) in IRNOM.

Table 1 CPU time (seconds) in random instances of LEP
Table 2 CPU time (seconds) in random instances of NEP with α = 1
Table 3 CPU time (seconds) in random instances of NEP with α = 10
Table 4 CPU time (seconds) in random instances of NEP with α = 100
Table 5 CPU time in random instances of OPP (uniformly distributed singular values)
Table 6 CPU time in random instances of OPP (equally spaced singular values)
Table 7 CPU time in random instances of OPP (clustered eigenvalues)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Francisco, J.B., Gonçalves, D.S., Bazán, F.S.V. et al. Nonmonotone inexact restoration approach for minimization with orthogonality constraints. Numer Algor (2020). https://doi.org/10.1007/s11075-020-00948-z

Download citation

Keywords

  • Inexact restoration
  • Orthogonality constraints
  • Cayley transform
  • Conjugate gradient

Mathematics subject classification (2010)

  • 49Q99
  • 65K05
  • 90C22
  • 90C26
  • 90C30