Mathematical Programming Computation

, Volume 10, Issue 4, pp 457–486 | Cite as

Cubic regularization in symmetric rank-1 quasi-Newton methods

  • Hande Y. BensonEmail author
  • David F. Shanno
Full Length Paper


Quasi-Newton methods based on the symmetric rank-one (SR1) update have been known to be fast and provide better approximations of the true Hessian than popular rank-two approaches, but these properties are guaranteed under certain conditions which frequently do not hold. Additionally, SR1 is plagued by the lack of guarantee of positive definiteness for the Hessian estimate. In this paper, we propose cubic regularization as a remedy to relax the conditions on the proofs of convergence for both speed and accuracy and to provide a positive definite approximation at each step. We show that the n-step convergence property for strictly convex quadratic programs is retained by the proposed approach. Extensive numerical results on unconstrained problems from the CUTEr test set are provided to demonstrate the computational efficiency and robustness of the approach.


Interior-point methods Nonlinear programming Cubic regularization Newton’s method 

Mathematics Subject Classification

90C30 90C53 90–08 



We would like to thank Daniel Bienstock and Andreas Waechter for their handling of the paper as Editor and Associate Editor, respectively, for MPC. We would also like to thank the two anonymous referees whose feedback and suggestions have greatly improved the paper.


  1. 1.
    Anandkumar, A., Ge, R.: Efficient approaches for escaping higher order saddle points in non-convex optimization. (2016). arXiv preprint arXiv:1602.05908
  2. 2.
    Bellavia, S., Morini, B.: Strong local convergence properties of adaptive regularized methods for nonlinear least squares. IMA J. Numer. Anal. 35, dru021 (2014)MathSciNetGoogle Scholar
  3. 3.
    Benson, H.Y., Shanno, D.F.: Interior-point methods for nonconvex nonlinear programming: cubic regularization. Comput. Optim. Appl. 58, 323 (2014)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bianconcini, T., Liuzzi, G., Morini, B., Sciandrone, M.: On the use of iterative methods in cubic regularization for unconstrained optimization. Comput. Optim. Appl. 60(1), 35–57 (2015)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Bianconcini, T., Sciandrone, M.: A cubic regularization algorithm for unconstrained optimization using line search and nonmonotone techniques. Optim. Methods Softw. 1–28 (2016)Google Scholar
  6. 6.
    Birgin, E.G., Gardenghi, J.L., Martınez, J.M., Santos, S.A., Toint, Ph. L.: Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models. Report naXys-05-2015, University of Namur, Belgium (2015)Google Scholar
  7. 7.
    Broyden, C.G.: Quasi-newton methods and their application to function minimisation. Math. Comput. 21(99), 368–381 (1967)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Broyden, G.C.: The convergence of a class of double-rank minimization algorithms 2. the new algorithm. IMA J. Appl. Math. 6(3), 222–231 (1970)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Byrd, R.H., Khalfan, H.F., Schnabel, R.B.: Analysis of a symmetric rank-one trust region method. SIAM J. Optim. 6, 1025–1039 (1996)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Cartis, C., Gould, N.I.M., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results. Math. Program. Ser. A 127, 245–295 (2011)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Cartis, C., Gould, N.I.M., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity. Math. Program. Ser. A 130, 295–319 (2011)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Conn, A.R., Gould, N.I.M., Toint, P.L.: Convergence of quasi-newton matrices generated by the symmetric rank one update. Math. Program. 50(1–3), 177–195 (1991)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Conn, A.R., Gould, N., Toint, Ph.L.: Constrained and unconstrained testing environment. Accessed 01 Feb 2018
  14. 14.
    Conn, A.R., Gould, N.I.M., Toint, P.L.: Convergence of quasi-newton matrices generated by the symmetric rank one update. Math. Program. 50, 177–195 (1991)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Davidon, W.C.: Variance algorithm for minimization. Comput. J. 10(4), 406–410 (1968)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Davidon, W.C.: Variable metric method for minimization. SIAM J. Optim. 1(1), 1–17 (1991)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Technical report, Argonne National Laboratory (2001)Google Scholar
  18. 18.
    Dussault, Jean-Pierre: Simple unified convergence proofs for the trust-region and a new arc variant. Technical report, Technical report, University of Sherbrooke, Sherbrooke, Canada (2015)Google Scholar
  19. 19.
    Feldman, S.I.: A fortran to c converter. In: ACM SIGPLAN Fortran Forum, vol. 9, pp. 21–22. ACM (1990)Google Scholar
  20. 20.
    Fiacco, A.V., McCormick, G.P.: Nonlinear programming: sequential unconstrained minimization techniques. Research Analysis Corporation, McLean Virginia. Republished in 1990 by SIAM, Philadelphia (1968)Google Scholar
  21. 21.
    Fletcher, R.: Practical Methods of Optimization. Wiley, Chichester (1987)zbMATHGoogle Scholar
  22. 22.
    Fletcher, R.: A new approach to variable metric algorithms. Comput. J. 13(3), 317–322 (1970)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Fletcher, R., Powell, M.J.D.: A rapidly convergent descent method for minimization. Comput. J. 6(2), 163–168 (1963)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Fourer, R., Gay, D.M., Kernighan, B.W.: AMPL: A Modeling Language for Mathematical Programming. Scientific Press, London (1993)zbMATHGoogle Scholar
  25. 25.
    Goldfarb, D.: Sufficient conditions for the convergence of a variable metric algorithm. In: Fletcher, R. (ed.) Optimization, pp. 273–281. Academic Press, New York (1969)Google Scholar
  26. 26.
    Goldfarb, D.: A family of variable-metric methods derived by variational means. Mathematics of computation 24(109), 23–26 (1970)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Göllner, T., Hess, W., Ulbrich, S.: Geometry optimization of branched sheet metal products. PAMM 12(1), 619–620 (2012)CrossRefGoogle Scholar
  28. 28.
    Gould, N.I.M., Porcelli, M., Toint, P.L.: Updating the regularization parameter in the adaptive cubic regularization algorithm. Comput. Optim. Appl. 53(1), 1–22 (2012)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Griewank, A.: The modification of Newton’s method for unconstrained optimization by bounding cubic terms. Technical Report NA/12, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1981)Google Scholar
  30. 30.
    Griewank, A., Fischer, J., Bosse, T.: Cubic overestimation and secant updating for unconstrained optimization of c 2, 1 functions. Optim. Methods Softw. 29(5), 1075–1089 (2014)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Hsia, Y., Sheu, R.-L., Yuan, Y.-X.: On the p-regularized trust region subproblem (2014). arXiv preprint arXiv:1409.4665
  32. 32.
    Huang, H.Y.: Unified approach to quadratically convergent algorithms for function minimization. J. Optim. Theory Appl. 5(6), 405–423 (1970)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Khalfan, H.F., Byrd, R.H., Schnabel, R.B.: A theoretical and experimental study of the symmetric rank-one update. SIAM J. Optim. 3(1), 1–24 (1993)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Liu, X., Sun, J.: Global convergence analysis of line search interior-point methods for nonlinear programming without regularity assumptions. J. Optim. Theory Appl. 125(3), 609–628 (2005)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Sha, L., Wei, Z., Li, L.: A trust region algorithm with adaptive cubic regularization methods for nonsmooth convex minimization. Comput. Optim. Appl. 51(2), 551–573 (2012)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Martınez, J.M., Raydan, M.: Cubic-regularization counterpart of a variable-norm trust-region method for unconstrained minimization. Technical report (2015)Google Scholar
  37. 37.
    Murtagh, B.A., Sargent, R.W.H.: A constrained minimization method with quadratic convergence. Optimization, pp. 215–246 (1969)Google Scholar
  38. 38.
    Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Math. Program. Ser. A 108, 177–205 (2006)MathSciNetCrossRefGoogle Scholar
  39. 39.
    Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research. Springer, Berlin (1999)Google Scholar
  40. 40.
    Oren, S.S., Luenberger, D.G.: Self-scaling variable metric (ssvm) algorithms. Manag. Sci. 20(5), 863–874 (1974)CrossRefGoogle Scholar
  41. 41.
    Oren, S.S., Luenberger, D.G.: Self-scaling variable metric (ssvm) algorithms: Part i: Criteria and sufficient conditions for scaling a class of algorithms. Manag. Sci. 20(5), 845–862 (1974)CrossRefGoogle Scholar
  42. 42.
    Oren, S.S., Spedicato, E.: Optimal conditioning of self-scaling variable metric algorithms. Math. Program. 10(1), 70–90 (1976)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Powell, M.J.D.: Recent advances in unconstrained optimization. Math. Program. 1, 26–57 (1971)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Schiela, A.: A flexible framework for cubic regularization algorithms for non-convex optimization in function space. Technical report (2014)Google Scholar
  45. 45.
    Shanno, D.D., Phua, K.H.: Remark on algorithm 500. minimization of unconstrained multivariate functions. Trans. Math. Softw. 6(4), 618–622 (1980)CrossRefGoogle Scholar
  46. 46.
    Shanno, D.F.: Conditioning of quasi-newton methods for function minimization. Math. Comput. 24(111), 647–656 (1970)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Shanno, D.F.: Conjugate gradient methods with inexact searches. Math. Oper. Res. 3(3), 244–256 (1978)MathSciNetCrossRefGoogle Scholar
  48. 48.
    Shanno, D.F., Phua, K.H.: Matrix conditioning and nonlinear optimization. Math. Program. 14(1), 149–160 (1978)MathSciNetCrossRefGoogle Scholar
  49. 49.
    Sherman, J., Morrison, W.J.: Adjustment of an inverse matrix corresponding to changes in the elements of a given column or a given row of the original matrix. In: Annals of Mathematical Statistics, volume 20(4), pp. 621–621. INST MATHEMATICAL STATISTICS IMS BUSINESS OFFICE-SUITE 7, 3401 INVESTMENT BLVD, HAYWARD, CA 94545 (1949)Google Scholar
  50. 50.
    Vanderbei, R.J.: AMPL models. Accessed 01 August 2016
  51. 51.
    Vanderbei, R.J., Shanno, D.F.: An interior-point algorithm for nonconvex nonlinear programming. Comput. Optim. Appl. 13, 231–252 (1999)MathSciNetCrossRefGoogle Scholar
  52. 52.
    Weiser, M., Deuflhard, P., Erdmann, B.: Affine conjugate adaptive Newton methods for nonlinear elastomechanics. Optim. Methods Softw. 22(3), 413–431 (2007)MathSciNetCrossRefGoogle Scholar
  53. 53.
    Wolfe, P.: Another variable metric method. Technical report, working paper (1967)Google Scholar
  54. 54.
    Wolfe, P.: Convergence conditions for ascent methods. SIAM Rev. 11(2), 226–235 (1969)MathSciNetCrossRefGoogle Scholar
  55. 55.
    Wolfe, P.: Convergence conditions for ascent methods. ii: Some corrections. SIAM Rev. 13(2), 185–188 (1971)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature and The Mathematical Programming Society 2018

Authors and Affiliations

  1. 1.Drexel UniversityPhiladelphiaUSA
  2. 2.EmeritusRutgers UniversityNew BrunswickUSA

Personalised recommendations