A Dynamical-System Analysis of the Optimum s-Gradient Algorithm

  • L. Pronzato
  • H.P. Wynn
  • A. Zhigljavsky
Part of the Springer Optimization and Its Applications book series (SOIA, volume 28)


We study the asymptotic behaviour of Forsythe's s-optimum gradient algorithm for the minimization of a quadratic function in \({\mathbb R}^d\) using a renormalization that converts the algorithm into iterations applied to a probability measure. Bounds on the performance of the algorithm (rate of convergence) are obtained through optimum design theory and the limiting behaviour of the algorithm for s = 2 is investigated into details. Algorithms that switch periodically between s = 1 and s = 2 are shown to converge much faster than when s is fixed at 2.


Limit Point Steep Descent Global Rate Support Point Orthogonality Property 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Akaike, H. (1959). On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method. Annals of the Institute Statistical Mathematics Tokyo, 11, 1–16.MATHCrossRefMathSciNetGoogle Scholar
  2. Elaydi, S. (2005). An Introduction to Difference Equations. Springer, Berlin. Third Edition.Google Scholar
  3. Fedorov, V. (1972). Theory of Optimal Experiments. Academic Press, New York.Google Scholar
  4. Forsythe, G. (1968). On the asymptotic directions of the s-dimensional optimum gradient method. Numerische Mathematik, 11, 57–76.MATHCrossRefMathSciNetGoogle Scholar
  5. Hoel, P. and Levine, A. (1964). Optimal spacing and weighting in polynomial prediction. Annals of Mathematical Statistics, 35(4), 1553–1560.MATHCrossRefMathSciNetGoogle Scholar
  6. Kantorovich, L. and Akilov, G. (1982). Functional Analysis. Pergamon Press, London. Second edition.Google Scholar
  7. Kiefer, J. and Wolfowitz, J. (1959). Optimum designs in regression problems. Annals of Mathematical Statistics, 30, 271–294.MATHCrossRefMathSciNetGoogle Scholar
  8. Luenberger, D. (1973). Introduction to Linear and Nonlinear Programming. Addison-Wesley, Reading, Massachusetts.Google Scholar
  9. Meinardus, G. (1963). Über eine Verallgemeinerung einer Ungleichung von L.V. Kantorowitsch. Numerische Mathematik, 5, 14–23.MATHCrossRefMathSciNetGoogle Scholar
  10. Molchanov, I. and Zuyev, S. (2001). Variational calculus in the space of measures and optimal design. In A. Atkinson, B. Bogacka, and A. Zhigljavsky, editors, Optimum Design 2000, chapter 8, pages 79–90. Kluwer, Dordrecht.Google Scholar
  11. Nocedal, J., Sartenaer, A., and Zhu, C. (1998). On the accuracy of nonlinear optimization algorithms. Technical Report Nov. 1998, ECE Department, Northwestern Univ., Evanston, Il 60208.Google Scholar
  12. Nocedal, J., Sartenaer, A., and Zhu, C. (2002). On the behavior of the gradient norm in the steepest descent method. Computational Optimization and Applications, 22, 5–35.MATHCrossRefMathSciNetGoogle Scholar
  13. Pronzato, L., Wynn, H., and Zhigljavsky, A. (2000). Dynamical Search. Chapman & Hall/CRC, Boca Raton.Google Scholar
  14. Pronzato, L., Wynn, H., and Zhigljavsky, A. (2001). Renormalised steepest descent in Hilbert space converges to a two-point attractor. Acta Applicandae Mathematicae, 67, 1–18.MATHCrossRefMathSciNetGoogle Scholar
  15. Pronzato, L., Wynn, H., and Zhigljavsky, A. (2005). Kantorovich-type inequalities for operators via D-optimal design theory. Linear Algebra and its Applications (Special Issue on Linear Algebra and Statistics), 410, 160–169.MATHMathSciNetGoogle Scholar
  16. Pronzato, L., Wynn, H., and Zhigljavsky, A. (2006). Asymptotic behaviour of a family of gradient algorithms in \({\mathbb R}^d\) and Hilbert spaces. Mathematical Programming, A107, 409–438.CrossRefMathSciNetGoogle Scholar
  17. Sahm, M. (1998). Optimal designs for estimating individual coefficients in polynomial regression. Ph.D. Thesis, Fakultät für Mathematik, Ruhr Universität Bochum.Google Scholar
  18. Silvey, S. (1980). Optimal Design. Chapman & Hall, London.Google Scholar

Copyright information

© Springer Science+Business Media LLC 2009

Authors and Affiliations

  1. 1.Laboratoire I3S, CNRS - UNSALes Algorithmes – Bât. Euclide BSophia AntipolisFrance
  2. 2.London School of Economics and Political ScienceLondonUK
  3. 3.Cardiff University, School of MathematicsCardiffUK

Personalised recommendations