Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Hybrid method for nonlinear least-square problems without calculating derivatives

  • 167 Accesses

  • 6 Citations


This paper presents a no-derivative modification of the hybrid Gauss-Newton-BFGS method for nonlinear least-square problems suggested initially by Al-Baali and Fletcher and modified later by Fletcher and Xu. The modification is made in such a way that, in a Gauss-Newton step, the Broyden's rank-one updating formula is used to obtain an approximate Jacobian and, in a BFGS step, the Jacobian is estimated using difference formulas. A set of numerical comparisons among the new hybrid method, the Gauss-Newton-Broyden method, and the finite-difference BFGS method is made and shows that the new hybrid method combines the better features of the Gauss-Newton-Broyden method and the finite-difference BFGS method. This paper also extends to the least-square problem the finite-termination property of the Broyden method, proved for a nonsingular system of equations by Gay and for the full-rank rectangular system of equations by Gerber and Luk.

This is a preview of subscription content, log in to check access.


  1. 1.

    Powell, M. J. D.,Some Global Convergence Properties of a Variable Metric Algorithm for Minimization without Exact Line Searches, Nonlinear Programming, Edited by R. W. Cottle and C. E. Lemke, SIAM-AMS Proceedings, Vol. 9, Philadelphia, Pennsylvania, 1976.

  2. 2.

    Fletcher, R.,Practical Methods of Optimization, Vol. 1: Unconstrained Optimization, Wiley, Chichester, England, 1980.

  3. 3.

    Powell, M. J. D.,A Hybrid Method for Nonlinear Equations, Numerical Methods for Nonlinear Equations, Edited by P. Rabinowitz, Gordon and Breach, London, England, 1970.

  4. 4.

    Dennis, J. E., Jr., Gay, D. M., andWelsch, R. E.,An Adaptive Nonlinear Least-Squares Algorithm, TOMS, Vol. 7, pp. 348–368, 1981.

  5. 5.

    Dennis, J. E., Jr., Gay, D. M., andWelsch, R. E.,Algorithm 573, NL2SOL: An Adaptive Nonlinear Least-Square Algorithm], TOMS, Vol. 7, pp. 369–383, 1981.

  6. 6.

    Dennis, J. E., Jr., Martinez, H. J., andTapia, R. A.,A Convergence Theory for the Structured BFGS Secant Method with an Application to Nonlinear Least Squares, Rice University, Mathematical Sciences Department, Technical Report No. 87–15, 1987.

  7. 7.

    Al-Baali, M., andFletcher, R.,Variational Methods for Nonlinear Least Squares, Journal of the Operational Research Society, Vol. 36, pp. 405–421, 1985.

  8. 8.

    Fletcher, R., andXu, C.,Hybrid Methods for Nonlinear Least Squares, IMA Journal of Numerical Analysis, Vol. 7, pp. 371–389, 1987.

  9. 9.

    Dennis, J. E., Jr.,Algorithms for Nonlinear Problems Which Use Discrete Approximations to Derivatives, Proceedings of ACM 1971 National Conference, Chicago, Illinois, 1971.

  10. 10.

    Dennis, J. E., Jr.,Some Computational Techniques for Nonlinear Least-Square Problems, Numerical Solution of Systems of Nonlinear Equations, Edited by G. D. Byrne and C. A. Hall, Academic Press, New York, New York, 1973.

  11. 11.

    Dennis, J. E., Jr.,Nonlinear Least Squares and Equations, The State-of-the-Art in Numerical Analysis, Edited by D. Jacobs, Academic Press, New York, New York, 1977.

  12. 12.

    Boggs, P. T., andDennis, J. E., Jr.,A Stability Analysis for Perturbed Nonlinear Iterative Methods, Mathematics of Computation, Vol. 30, pp. 1–17, 1976.

  13. 13.

    Broyden, C. G.,A Class of Methods for Solving Nonlinear Simultaneous Equations, Mathematics of Computation, Vol. 19, pp. 577–593, 1965.

  14. 14.

    Barnes, J. G. P.,An Algorithm for Solving Nonlinear Equations Based on the Secant Method, Computer Journal, Vol. 8, pp. 66–72, 1965.

  15. 15.

    Gay, D. M.,Some Convergence Properties of Broyden's Method, SIAM Journal on Numerical Analysis, Vol. 16, pp. 623–630, 1979.

  16. 16.

    Gerber, R. R., andLuk, F. T.,A Generalized Broyden's Method for Solving Simultaneous Linear Equations, SIAM Journal on Numerical Analysis, Vol. 18, pp. 882–890, 1980.

  17. 17.

    Stewart, G. W.,A Modification of Davidon's Minimization Method to Accept Difference Approximation of Derivatives, Journal of the ACM, Vol. 14, pp. 72–83, 1967.

  18. 18.

    Gill, P. E., Murray, W., andPitfield, R. A.,The Implementation of Two Revised Quasi-Newton Algotithms for Unconstrained Optimization, National Physical Laboratory, Report No. NAC24, 1972.

  19. 19.

    Fletcher, R.,Fortran Subroutines for Minimization by Quasi-Newton Methods, AERE, Report No. R7125, 1972.

  20. 20.

    Brown, K. M., andDennis, J. E., Jr.,Derivative-Free Analogues of Levenberg-Marquardt and Gauss Algorithms for Nonlinear Least-Square Approximation, Numerische Mathematik, Vol. 18, pp. 289–297, 1972.

  21. 21.

    Gill, P. E., andMurray, W.,Quasi-Newton Methods for Unconstrained Optimization, Journal of Institute of Mathematics and Its Applications, Vol. 9, pp. 91–108, 1972.

  22. 22.

    McKeown, J. J.,Large Residual Nonlinear Least-Square Problems, Nonlinear Optimization: Theory and Algorithms, Edited by L. C. W. Dixon, E. Spedicato, and G. P. Szego, Birkhauser, Boston, Massachusetts, 1980.

  23. 23.

    Ortega, J. M., andRheinboldt, W. G.,Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York, New York, 1970.

  24. 24.

    Ben-Israel, A.,On Iterative Methods for Solving Nonlinear Least-Square Problems over Convex Sets, Israel Journal of Mathematics, Vol. 5, pp. 211–224, 1967.

  25. 25.

    Fletcher, R.,Generalized Inverse Methods for the Best Least-Square Solution of Systems of Nonlinear Equations, Computer Journal, Vol. 9, pp. 392–399, 1967.

  26. 26.

    Boggs, P. T.,The Convergence of the Ben-Israel Iteration for Nonlinear Least-Square Problem, Mathematics of Computation, Vol. 30, pp. 199–215, 1976.

  27. 27.

    Rao, C. R., andMitra, S. K.,Generalized Inverse of Matrices and Its Applications, John Wiley, New York, New York, 1971.

  28. 28.

    Dennis, J. E., Jr., andMore, J. J.,Quasi-Newton Methods, Motivation and Theory, SIAM Review, Vol. 19, pp. 46–89, 1975.

  29. 29.

    Xu, C.,Hybrid Methods for Nonlinear Least Squares and Related Problems, University of Dundee, PhD Thesis, 1986.

  30. 30.

    Gill, P. E., andMurray, W.,Safeguarded Steplength Algorithms for Optimization Using Descent Methods, National Physical Laboratory, Report No. NAC37, 1974.

  31. 31.

    Al-Baali, M., andFletcher, R.,An Efficient Line Search for Nonlinear Least Squares, University of Dundee, Numerical Analysis Report No. NA78, 1984.

Download references

Author information

Additional information

The author would like to acknowledge the support of Xian Jiaotong University, Xian, China, and the award of a United Kingdom ORS studentship. The author wishes to express his gratitute to Professor R. Fletcher for his encouragement and to thank Dr. G. A. Watson and Dr. M. C. Bartholomew-Biggs for their useful comments during the preparation of this paper. The author also wishes to acknowledge Professor R. A. Tapia for his valuable suggestions.

Communicated by R. A. Tapia

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Xu, C.X. Hybrid method for nonlinear least-square problems without calculating derivatives. J Optim Theory Appl 65, 555–574 (1990). https://doi.org/10.1007/BF00939566

Download citation

Key Words

  • Nonlinear least squares
  • hybrid method
  • Gauss-Newton method
  • BFGS method
  • finite-termination property