Abstract
Numerical optimization is introduced as the mathematical foundation for this book, focusing on two basic unconstrained optimization algorithms: line search and trust-region methods. Line search optimization methods are relatively simple and commonly used gradient descent based methods. Their strength lies in their simplicity and ease of implementation, but their convergence properties degrade as the nonlinearity and complexity of the function to be optimized increases. Trust-region, or “restricted step” methods are presented as an often more practical alternative to line search that involved the construction of an approximation, or “model,” of the function to be minimized, together with a dynamic estimate of the region where this model is sufficiently valid. There are a large number of optimization methods, and the interested reader is referred to the bibliographical citations presented in this chapter. In this book we restrict our attention mainly to line search and trust-region methods, since we will use them in the context of their application to extremum seeking control and stability proofs thereof.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Audet, C., Dennis, J.E.: Analysis of generalized pattern searches. SIAM J. Optim. 13(3), 889–903 (2002)
Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1995)
Blowey, J.F., Craig, A.W., Shardlow, T.: Frontiers in Numerical Analysis. Springer, Berlin (2002)
Conn, A.R., Scheinberg, K., Toint, P.L.: On the convergence of derivative-free methods for unconstrained optimization. Approximation Theory and Optimization: Tributes to M.J.D. Powell, pp. 83–108 (1997)
Conn, A.R., Scheinberg, K., Toint, P.L.: Recent progress in unconstrained nonlinear optimization without derivatives. Math. Program. 79, 397–414 (1997)
Davis, C.: Theory of positive linear dependence. Am. J. Math. 76(4), 733–746 (1954)
Fletcher, R.: Practical Methods of Optimization. Wiley, New York (1987)
Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cambridge (1985)
Kolda, T.G., Lewis, R.M., Torczon, V.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45(3), 385–482 (2003)
Nelder, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7(4), 308–313 (1965)
Nocedal, J., Wright, S.: Numerical Optimization. Springer, Berlin (1999)
Pardalos, P.M., Resende, M.G.C.: Handbook of Applied Optimization. Oxford University Press, New York (2002)
Powell, M.J.D.: UOBYQA: Unconstrained optimization by quadratic approximation. Technical Report NA2000/14, DAMTP. University of Cambridge (2000)
Spall, J.C.: Introduction to Stochastic Search and Optimization. Wiley-Interscience, New Jersey (2003)
Torczon, V.: On the convergence of pattern search algorithms. SIAM J. Optim. 7(1), 1–25 (1997)
Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67–82 (1997)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2012 Springer-Verlag London Limited
About this chapter
Cite this chapter
Zhang, C., Ordóñez, R. (2012). Numerical Optimization. In: Extremum-Seeking Control and Applications. Advances in Industrial Control. Springer, London. https://doi.org/10.1007/978-1-4471-2224-1_2
Download citation
DOI: https://doi.org/10.1007/978-1-4471-2224-1_2
Publisher Name: Springer, London
Print ISBN: 978-1-4471-2223-4
Online ISBN: 978-1-4471-2224-1
eBook Packages: EngineeringEngineering (R0)