Advertisement

Computational Optimization and Applications

, Volume 60, Issue 2, pp 343–376 | Cite as

Algebraic rules for quadratic regularization of Newton’s method

  • Elizabeth W. Karas
  • Sandra A. Santos
  • Benar F. Svaiter
Article

Abstract

In this work we propose a class of quasi-Newton methods to minimize a twice differentiable function with Lipschitz continuous Hessian. These methods are based on the quadratic regularization of Newton’s method, with algebraic explicit rules for computing the regularizing parameter. The convergence properties of this class of methods are analysed. We show that if the sequence generated by the algorithm converges then its limit point is stationary. We also establish local quadratic convergence in a neighborhood of a stationary point with positive definite Hessian. Encouraging numerical experiments are presented.

Keywords

Smooth unconstrained minimization Newton’s method   Regularization Global convergence Local convergence  Computational results 

Mathematics Subject Classification

90C30 90C53 49M15 

Notes

Acknowledgments

The authors are grateful to José Mario Martínez and Ernesto Birgin for their valuable comments and suggestions. We are also thankful to the anonymous referee whose suggestions led to improvements in the paper. Elizabeth W. Karas was partially supported by CNPq Grants 307714/2011-0 and 477611/2013-3. Sandra A. Santos was partially supported by CNPq Grant 304032/2010-7, FAPESP Grants 2013/05475-7, 2013/07375-0 and PRONEX Optimization. Benar F. Svaiter was partially supported by CNPq Grants 302962/2011-5, 474996/2013-1, FAPERJ Grant E-26/102.940/2011 and PRONEX Optimization.

References

  1. 1.
    Birgin, E.G., Gentil, J.M.: Evaluating bound-constrained minimization software. Comput. Optim. Appl. 53(2), 347–373 (2012)CrossRefMATHMathSciNetGoogle Scholar
  2. 2.
    Cartis, C., Gould, N.I.M., Toint, Ph.L.: Evaluation complexity of adaptive cubic regularization methods for convex unconstrained optimization. Optim. Methods Softw. 27(2), 197–219 (2012)Google Scholar
  3. 3.
    Cartis, C., Gould, N.I.M., Toint, Ph.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results. Math. Program. 127(2), 245–295 (2011)Google Scholar
  4. 4.
    Cartis, C., Gould, N.I.M., Toint, Ph.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity. Math. Program. 130(2), 295–319 (2011)Google Scholar
  5. 5.
    Cartis, C., Gould, N.I.M., Toint, Ph.L.: An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity. IMA J. Numer. Anal. 32(4), 1662–1695 (2012)Google Scholar
  6. 6.
    Conn, A.R., Gould, N.I.M., Toint, Ph.L.: Trust-Region Methods. MPS/SIAM Series on Optimization. SIAM, Philadelphia, PA (2000)Google Scholar
  7. 7.
    Dennis, J.E., Schnabel, R.B.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Classics in Applied Mathematics. SIAM, Philadelphia, PA (1996). (Corrected reprint of the 1983 original)CrossRefGoogle Scholar
  8. 8.
    Fletcher, R.: Practical Methods of Optimization, 2nd edn. John Wiley, Chichester (1987)MATHGoogle Scholar
  9. 9.
    Fuentes, M., Malick, J., Lemaréchal, C.: Descentwise inexact proximal algorithms for smooth optimization. Comput. Optim. Appl. 53(3), 755–769 (2012)CrossRefMATHMathSciNetGoogle Scholar
  10. 10.
    Gill, P.E., Murray, W., Wright, R.H.: Practical Optimization. Academic Press Inc., San Diego (1981)MATHGoogle Scholar
  11. 11.
    Goldfeld, S.M., Quandt, R.E., Trotter, H.F.: Maximization by quadratic hill-climbing. Econometrica 34, 541–551 (1966)CrossRefMATHMathSciNetGoogle Scholar
  12. 12.
    Gould, N.I.M., Porcelli, M., Toint, Ph.L.: Updating the regularization parameter in the adaptive cubic regularization algorithm. Comput. Optim. Appl. 53(1), 1–22 (2012)Google Scholar
  13. 13.
    Griewank, A.: The modification of Newton’s method for unconstrained optimization by bounding cubic terms. Technical Report NA/12, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1981)Google Scholar
  14. 14.
    Hager, W.W., Zhang, H.: Self-adaptive inexact proximal point methods. Comput. Optim. Appl. 39(2), 161–181 (2008)CrossRefMATHMathSciNetGoogle Scholar
  15. 15.
    Hebden, M. D.: An algorithm for minimization using exact second derivatives. Technical Report T.P. 515, AERE Harwell Laboratory, Harwell, Oxfordshire (1973)Google Scholar
  16. 16.
    Lehoucq, R.B., Sorensen, D.C., Yang, C.: ARPACK Users Guide, Software, Environments, and Tools. SIAM, Philadelphia, PA (1998)CrossRefGoogle Scholar
  17. 17.
    Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Quart. Appl. Math. 2, 164–168 (1944)MATHMathSciNetGoogle Scholar
  18. 18.
    Martinet, B.: Régularisation d’inéquations variationnelles par approximations successives. Rev. Française Info Recherche Opérationnelle, 4:154–158 (1970)Google Scholar
  19. 19.
    Moré, J. J.: Recent developments in algorithms and software for trust region methods. In: Proceedings of the Mathematical programming: the state of the art (Bonn, 1982), pp. 258–287. Springer, Berlin (1983)Google Scholar
  20. 20.
    Nesterov, Y., Polyak, B. T.: Cubic regularization of Newton method and its global performance. Math. Program. 108(1):177–205 (2006)Google Scholar
  21. 21.
    Nocedal, J., Wright, S.J.: Numerical Optimization. Operations Research and Financial Engineering, 2nd edn. Springer, New York (2006)Google Scholar
  22. 22.
    Schnabel, R.B., Koontz, J.E., Weiss, B.E.: A modular system of algorithms for unconstrained minimization. ACM Trans. Math. Softw. 11(4), 419–440 (1985)CrossRefMATHMathSciNetGoogle Scholar
  23. 23.
    Tikhonov, A. N.: On the stability of inverse problems (39, pp. 176–179). C. R. (Doklady) Acad. Sci. URSS (N.S), Moscow (1943)Google Scholar
  24. 24.
    Tukey, J.W.: Exploratory Data Analysis. Behavioral Science: Quantitative Methods. Addison-Wesley Publishing Company, Reading, MA (1977)Google Scholar
  25. 25.
    Weiser, M., Deuflhard, P., Erdmann, B.: Affine conjugate adaptive Newton methods for nonlinear elastomechanics. Optim. Methods Softw. 22(3), 413–431 (2007)CrossRefMATHMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Elizabeth W. Karas
    • 1
  • Sandra A. Santos
    • 2
  • Benar F. Svaiter
    • 3
  1. 1.Department of MathematicsFederal University of ParanáCuritibaBrazil
  2. 2.Department of Applied MathematicsUniversity of CampinasCampinasBrazil
  3. 3.IMPARio de Janeiro Brazil

Personalised recommendations