Advertisement

Computational Optimization and Applications

, Volume 72, Issue 3, pp 561–588 | Cite as

Efficient calculation of regular simplex gradients

  • Ian Coope
  • Rachael TappendenEmail author
Article

Abstract

Simplex gradients are an essential feature of many derivative free optimization algorithms, and can be employed, for example, as part of the process of defining a direction of search, or as part of a termination criterion. The calculation of a general simplex gradient in \(\mathbf {R}^n\) can be computationally expensive, and often requires an overhead operation count of \(\mathcal {O}(n^3)\) and in some algorithms a storage overhead of \(\mathcal {O}(n^2)\). In this work we demonstrate that the linear algebra overhead and storage costs can be reduced, both to \(\mathcal {O}(n)\), when the simplex employed is regular and appropriately aligned. We also demonstrate that a gradient approximation that is second order accurate can be obtained cheaply from a combination of two, first order accurate (appropriately aligned) regular simplex gradients. Moreover, we show that, for an arbitrarily aligned regular simplex, the gradient can be computed in \(\mathcal {O}(n^2)\) operations.

Keywords

Positive bases Numerical optimization Derivative free optimization Regular simplex Simplex gradient Least squares Well poised 

Mathematics Subject Classification

52B12 65F20 65F35 90C56 

Notes

Acknowledgements

The authors thank Luis Vicente, and the anonymous referees for their helpful comments and suggestions, leading to improvements in an earlier version of this work.

References

  1. 1.
    Alberto, P., Nogueira, F., Rocha, H.: Pattern search methods for user-provided points: application to molecular geometry problems. SIAM J. Optim. 14(4), 1216–1236 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Audet, C., Hare, W.: Derivative-Free and Blackbox Optimization. Springer Series in Operations Research and Financial Engineering. Springer, Cham (2017)CrossRefzbMATHGoogle Scholar
  3. 3.
    Belegundu, A., Chandrupatla, T. (eds.): Optimization Concepts and Applications in Engineering (second printing), 1st edn. Prentice Hall, Upper Saddle River (1999)Google Scholar
  4. 4.
    Bortz, D.M., Kelley, C.T.: The simplex gradient and noisy optimization problems. In: Borggaard, J.T., Burns, J., Cliff, E., Schreck, S. (eds.) Computational Methods in Optimal Design and Control, Progress in Systems and Control Theory, vol. 24, pp. 77–90. Birkhäuser, Boston (1998)CrossRefGoogle Scholar
  5. 5.
    Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  6. 6.
    Conn, A., Scheinberg, K., Vicente, L.: Introduction to Derivative-Free Optimization, MPS–SIAM Series on Optimization. SIAM, Philadelphia (2009)CrossRefzbMATHGoogle Scholar
  7. 7.
    Custódio, A.L., Dennis Jr., J.E., Vicente, L.N.: Using simplex gradients of nonsmooth functions in direct search methods. IMA J. Numer. Anal. 28(4), 770–784 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Custódio, A.L., Vicente, L.N.: Using sampling and simplex derivatives in pattern search methods. SIAM J. Optim. 18(2), 537–555 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Davis, C.: Theory of positive linear dependence. Am. J. Math. 76(4), 733–746 (1954).  https://doi.org/10.2307/2372648 MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Dennis, J., Torczon, V.: Direct search methods on parallel machines. SIAM J. Optim. 1, 448–474 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    El-Gebeily, M.A., Fiagbedzi, Y.A.: On certain properties of the regular \(n\)-simplex. Int. J. Math. Educ. Sci. Technol. 35(4), 617–629 (2004)CrossRefGoogle Scholar
  12. 12.
    Golub, G., Loan, C.V.: Matrix Computations, 4th edn. JHU Press, Baltimore (2016)zbMATHGoogle Scholar
  13. 13.
    Hooke, R., Jeeves, T.: Direct search solution of numerical and statistical problems. J. Assoc. Comput. Mach. (ACM) 8, 212–219 (1961)CrossRefzbMATHGoogle Scholar
  14. 14.
    Jacoby, S.L.S., Kowalik, J.S., Pizzo, J.T.: Iterative Methods for Nonlinear Optimization Problems. Prentice Hall, Englewood Cliffs (1972)zbMATHGoogle Scholar
  15. 15.
    Kelley, C.: Iterative Methods for Optimization. Frontiers in Applied Mathematics. SIAM, Philadelphia (1999)CrossRefGoogle Scholar
  16. 16.
    Lazebnik, F.: On a Regular Simplex in \({\mathbf{R}}^n\). Technical report, Department of Mathematical Sciences, University of Delaware (2004). http://www.math.udel.edu/~lazebnik/papers/simplex.pdf
  17. 17.
    Lewis, R.M., Torczon, V.: Rank Ordering and Positive Bases in Pattern Search Algorithms. Technical Report TR96-71, ICASE, NASA Langley Research Center, Hampton (1999)Google Scholar
  18. 18.
    Liu, Q.: Two minimal positive basis based direct search conjugate gradient methods for computationally expensive functions. Numer. Algorithms 58, 461–474 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Nelder, J., Mead, R.: A simplex method for function minimization. Comput. J. 7(4), 308–313 (1965)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Parkinson, J., Hutchinson, D.: An investigation into the efficiency of variants on the simplex method. In: Lootsma, F.A. (ed.) Numerical Methods for Non-linear Optimization, University of Dundee in Scotland, UK, 28 June–1 July 1971. London-New York Academic Press (1972)Google Scholar
  21. 21.
    Price, C., Coope, I., Byatt, D.: A convergent variant of the Nelder–Mead algorithm. J. Optim. Theory Appl. 113(1), 5–19 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Regis, R.G.: The calculus of simplex gradients. Optim. Lett. 9, 845–865 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Richardson, L., Gaunt, J.: The deferred approach to the limit. Philos. Trans. R. Soc. Lond. 226, 299–361 (1927)CrossRefzbMATHGoogle Scholar
  24. 24.
    Schoenberg, I.: Regular simplices and quadratic forms. J. Lond. Math. Soc. 12, 48–55 (1937)CrossRefzbMATHGoogle Scholar
  25. 25.
    Spendley, W., Hext, G., Himsworth, F.: Sequential application of simplex designs in optimisation and evolutionary operation. Technometrics 4, 441–461 (1962)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Torczon, V.: Multi-directional Search: A Direct Search Algorithm for Parallel Machines. Ph.D. thesis, Rice University, Houston, TX (1989)Google Scholar
  27. 27.
    Torczon, V.: On the convergence of the multidirectional search algorithm. SIAM J. Optim. 1, 123–145 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Watkins, D.S.: Fundamentals of Matrix Computations, Pure and Applied Mathematics, 3rd edn. Wiley, Hoboken (2010)Google Scholar
  29. 29.
    Zlatev, Z., Dimov, I., Faragó, I., Havasi, A.: Richardson Extrapolation: Practical Aspects and Applications, 1st edn. Series in Applied and Numerical Mathematics 2. De Gruyter, Berlin (2018)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Mathematics and StatisticsUniversity of CanterburyChristchurchNew Zealand

Personalised recommendations