Abstract
We consider the problem of minimizing a function whose derivatives are not available. This paper first presents an algorithm for solving problems of this class using interpolation polynomials and trust-region techniques. We then show how both the data structure and the procedure allowing to build the interpolating polynomials may be adapted in a suitable way to consider problems for which the Hessian matrix is known to be sparse with a general sparsity pattern. The favourable behaviour of the resulting algorithm is confirmed with numerical experiments illustrating the advantages of the method in terms of storage, speed and function evaluations, the latter criterion being particularly important in the framework of derivative-free optimization.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
N. M. Alexandrov, J.E. Dennis, R. M. Lewis, and V. Torczon. A trust region framework for managing the use of approximation models. Structural Optimization, 15 (1): 16–23, 1998
C. Audet, A. Booker, J. E. Dennis, P. Frank, and D. W. Moore. A surrogatemodel-based method for constrained optimization. AIAA paper 2000–4891, AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, September 2000.
G. E. P. Box. Evolutionary operation: a method for increasing industrial productivity. Applied statistics, 6: 81–101, 1957.
R. P. Brent. Algorithms for minimization without derivatives. Prentice-Hall, Englewood Cliffs, New Jersey, USA, 1973.
A. G. Buckley. A derivative-free algorithm for parallel and sequential optimization. presentation at the NATO ASI on Algorithms for Continuous Optimization, Il Ciocco, 1994.
F. M. Callier and Ph. L. Toint. Recent results on the accelerating property of an algorithm for function minimization without calculating derivatives. In A. Prekopa, editor, Survey of Mathematical Programming, pages 369–376. Publishing House of the Hungarian Academy of Sciences, 1977.
I. G. Campey and D. G. Nickols. Simplex minimization. Program specification, Imperial Chemical Industries Ltd, UK, 1961.
B. Colson and Ph. L. Toint. Exploiting band structure in unconstrained optimization without derivatives Technical Report TR01/03, Department of Mathematics, University of Namur, Namur, Belgium, 2001.
A. R. Conn, N. I. M. Gould, and Ph. L. Toint. Trust-Region Methods. Number 0l in MPS-SIAM Series on Optimization. SIAM, Philadelphia, USA, 2000.
A. R. Conn, K. Scheinberg, and Ph. L. Toint. On the convergence of derivative-free methods for unconstrained optimization. In A. Iserles and M. Buhmann, editors, Approximation Theory and Optimization: Tributes to M. J. D. Powell, pages 83–108, Cambridge, England, 1997. Cambridge University Press.
A. R. Conn, K. Scheinberg, and Ph. L. Toint. Recent progress in unconstrained nonlinear optimization without derivatives. Mathematical Programming, Series B, 79 (3): 397–414, 1997.
A. R. Conn, K. Scheinberg, and Ph. L. Toint. A derivative free optimization algorithm in practice. Technical Report TR98/11, Department of Mathematics, University of Namur, Namur, Belgium, 1998.
A. R. Conn and Ph. L. Toint. An algorithm using quadratic interpolation for unconstrained derivative free optimization. In G. Di Pillo and F. Gianessi, editors, Nonlinear Optimization and Applications,pages 27–47, New York, 1996. Plenum Publishing. Also available as Report 95/6, Dept of Mathematics, FUNDP, Namur, Belgium.
J. E. Dennis and R. B. Schnabel. Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Prentice-Hall, Englewood Cliffs, New Jersey, USA, 1983. Reprinted as Classics in Applied Mathematics 16, SIAM, Philadelphia, USA, 1996.
J. E. Dennis and V. Torczon. Direct search methods on parallel machines. SIAM Journal on Optimization, 1 (4): 448–474, 1991.
J. E. Dennis and V. Torczon. Managing approximation models in optimization. In N. M. Alexandrov and M. Y. Hussaini, editors, Multidisciplinary Design Optimization, pages 330–347, Philadelphia, USA, 1997. SIAM.
L. C. W. Dixon. Nonlinear Optimisation. The English Universities Press Ltd, London, 1972.
P. E. Gill, W. Murray, and M. H. Wright. Practical Optimization. Academic Press, London, 1981.
R. Hooke and T. A. Jeeves. Direct search solution of numerical and statistical problems. Journal of the ACM, 8: 212–229, 1961.
S. Lucidi and M. Sciandrone. A coordinate descent method without derivatives. Technical Report 10–95 (in preparation), University of Rome “La Sapienza”, Rome, 1995.
M. Marazzi and J. Nocedal. Wedge trust region methods for derivative free optimization. Technical Report OTC 2000/10, Optimization Technology Center, Argonnne National Laboratory, Argonne, Illinois, USA, 2000.
J. A. Neider and R. Mead. A simplex method for function minimization. Computer Journal, 7: 308–313, 1965.
J. Nocedal and S. J. Wright. Numerical Optimization. Springer Verlag, Heidelberg, Berlin, New York, 1999.
M. J. D. Powell. An efficient method for finding the minimum of a function of several variables without calculating derivatives. Computer Journal, 17: 155–162, 1964.
M. J. D. Powell. A method for minimizing a sum of squares of nonlinear functions without calculating derivatives. Computer Journal, 7: 303–307, 1965.
M. J. D. Powell. A Fortran subroutine for unconstrained minimization requiring first derivatives of the objective function. Technical Report R-6469, AERE Harwell Laboratory, Harwell, Oxfordshire, England, 1970.
M. J. D. Powell. A new algorithm for unconstrained optimization. In J. B. Rosen, O. L. Mangasarian, and K. Ritter, editors, Nonlinear Programming, London, 1970. Academic Press.
M. J. D. Powell. A direct search optimization method that models the objective and constraint functions by linear interpolation. In S. Gomez and J. P. Hennart, editors, Advances in Optimization and Numerical Analysis, Proceedings of the Sixth Workshop on Optimization and Numerical Analysis Oaxaca, Mexico,volume 275, pages 51–67, Dordrecht, The Netherlands, 1994. Kluwer Academic Publishers.
M. J. D. Powell. Trust region methods that employ quadratic interpolation to the objective function. Presentation at the 5th SIAM Conference on Optimization, Victoria, 1996.
M. J. D. Powell. On the Lagrange functions of quadratic models defined by interpolation. Technical Report NA10, Department of Applied Mathematics and Theoretical Physics, Cambridge University, Cambridge, England, 2000.
M. J. D. Powell. UOBYQA: unconstrained optimization by quadratic interpolation. Technical Report NA14, Department of Applied Mathematics and Theoretical Physics, Cambridge University, Cambridge, England, 2000.
Th. Sauer and Yuan Xu. On multivariate Lagrange interpolation. Mathematics of Computation, 64: 1147–1170, 1995.
W. Spendley, G. R. Hext, and F. R. Himsworth. Sequential application of simplex designs in optimization and evolutionary operation. Technometrics, 4, 1962.
G. W. Stewart. A modification of Davidon’s minimization method to accept difference approximations of derivatives. Journal of the ACM, 14, 1967.
V. Torczon. On the convergence of pattern search algorithms. SIAM Journal on Optimization, 7 (1): 1–25, 1997.
D. Winfield. Function and functional optimization by interpolation in data tables. PhD thesis, Harvard University, Cambridge, USA, 1969.
D. Winfield. Function minimization by interpolation in a data table. Journal of the Institute of Mathematics and its Applications, 12: 339–347, 1973.
M. H. Wright. Direct search methods: once scorned, now respectable. In D. F. Griffiths and G. A. Watson, editors, Proceedings of the 1995 Dundee Biennal Conference in Numerical Analysis, Reading, Massachusetts, USA, 1996. Addison-Wesley Publishing Company.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Kluwer Academic Publishers
About this chapter
Cite this chapter
Colson, B., Toint, P.L. (2002). A Derivative-Free Algorithm for Sparse Unconstrained Optimization Problems. In: Siddiqi, A.H., Kočvara, M. (eds) Trends in Industrial and Applied Mathematics. Applied Optimization, vol 72. Springer, Boston, MA. https://doi.org/10.1007/978-1-4613-0263-6_7
Download citation
DOI: https://doi.org/10.1007/978-1-4613-0263-6_7
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4613-7967-6
Online ISBN: 978-1-4613-0263-6
eBook Packages: Springer Book Archive