Skip to main content
SpringerLink
Log in

Introduction: Tools and Challenges in Derivative-Free and Blackbox Optimization

  • Chapter
  • First Online:
Derivative-Free and Blackbox Optimization

Abstract

In this introductory chapter, we present a high-level description of optimization, blackbox optimization, and derivative-free optimization. We introduce some basic optimization notation used throughout this book, and some of the standard classifications of optimization problems. We end with three examples where blackbox optimization problems have arisen in practice. The first of these examples will be used throughout this book as a case study for how various optimization algorithms behave.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 99.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. M.A. Abramson, Mixed variable optimization of a Load-Bearing thermal insulation system using a filter pattern search algorithm. Optim. Eng. 5(2), 157–177 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  2. M.A. Abramson, Second-order behavior of pattern search. SIAM J. Optim. 16(2), 315–330 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  3. M.A. Abramson, C. Audet, Convergence of mesh adaptive direct search to second-order stationary points. SIAM J. Optim. 17(2), 606–619 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  4. M.A. Abramson, C. Audet, J.W. Chrissis, J.G. Walston, Mesh adaptive direct search algorithms for mixed variable optimization. Optim. Lett. 3(1), 35–47 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  5. M.A. Abramson, C. Audet, G. Couture, J.E. Dennis Jr., S. Le Digabel, C. Tribes, The NOMAD project. Software available at https://www.gerad.ca/nomad

  6. M.A. Abramson, C. Audet, J.E. Dennis Jr., S. Le Digabel, OrthoMADS: a deterministic MADS instance with orthogonal directions. SIAM J. Optim. 20(2), 948–966 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  7. H.L. Anderson, W.C. Davidon, M.G. Glicksman, U.E. Kruse, Scattering of positive pions by hydrogen at 189 MeV. Phys. Rev. 100, 279–287 (1955)

    Article  Google Scholar 

  8. C. Audet, Convergence results for generalized pattern search algorithms are tight. Optim. Eng. 5(2), 101–122 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  9. C. Audet, A short proof on the cardinality of maximal positive bases. Optim. Lett. 5(1), 191–194 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  10. C. Audet, A survey on direct search methods for blackbox optimization and their applications, in Mathematics Without Boundaries: Surveys in Interdisciplinary Research, ed. by P.M. Pardalos, T.M. Rassias, Chap. 2, pp. 31–56 (Springer, Berlin, 2014)

    Google Scholar 

  11. C. Audet, Tuning Runge-Kutta parameters on a family of ordinary differential equations. Int. J. Math. Model. Numer. Optim. (in press 2018)

    Google Scholar 

  12. C. Audet, J.E. Dennis Jr., Pattern search algorithms for mixed variable programming. SIAM J. Optim. 11(3), 573–594 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  13. C. Audet, J.E. Dennis Jr., Analysis of generalized pattern searches. SIAM J. Optim. 13(3), 889–903 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  14. C. Audet, J.E. Dennis Jr., A pattern search filter method for nonlinear programming without derivatives. SIAM J. Optim. 14(4), 980–1010 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  15. C. Audet, J.E. Dennis Jr., Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17(1), 188–217 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  16. C. Audet, J.E. Dennis Jr., Nonlinear programming by mesh adaptive direct searches. SIAG/Optim. Views-and-News 17(1), 2–11 (2006)

    Google Scholar 

  17. C. Audet, J.E. Dennis Jr., A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20(1), 445–472 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  18. C. Audet, V. Béchard, J. Chaouki, Spent potliner treatment process optimization using a MADS algorithm. Optim. Eng. 9(2), 143–160 (2008)

    Article  MATH  Google Scholar 

  19. C. Audet, G. Savard, W. Zghal, Multiobjective optimization through a series of single-objective formulations. SIAM J. Optim. 19(1), 188–210 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  20. C. Audet, S. Le Digabel, C. Tribes, NOMAD user guide. Technical Report G-2009-37, Les cahiers du GERAD (2009)

    Google Scholar 

  21. C. Audet, G. Savard, W. Zghal, A mesh adaptive direct search algorithm for multiobjective optimization. Eur. J. Oper. Res. 204(3), 545–556 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  22. C. Audet, J.E. Dennis Jr., S. Le Digabel, Globalization strategies for mesh adaptive direct search. Comput. Optim. Appl. 46(2), 193–215 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  23. C. Audet, C.-K. Dang, D. Orban, Optimization of algorithms with OPAL. Math. Program. Comput. 6(3), 233–254 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  24. C. Audet, S. Le Digabel, M. Peyrega, Linear equalities in blackbox optimization. Comput. Optim. Appl. 61(1), 1–23 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  25. P. Balaprakash, S.M. Wild, B. Norris, Spapt: search problems in automatic performance tuning. Procedia Comput. Sci. 9, 1959–1968 (2012). Proceedings of the International Conference on Computational Science, ICCS 2012

    Google Scholar 

  26. T. Begin, B. Baynat, F. Sourd, A. Brandwajn, A DFO technique to calibrate queueing models. Comput. Oper. Res. 37(2), 273–281 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  27. V. Beiranvand, W. Hare, Y. Lucet, Benchmarking of single-objective optimization algorithms. Eng. Optim. (to appear)

    Google Scholar 

  28. H.-G. Beyer, H.-P. Schwefel, Evolution strategies – a comprehensive introduction. Nat. Comput. 1(1), 3–52 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  29. K. Bigdeli, W. Hare, S. Tesfamariam, Configuration optimization of dampers for adjacent buildings under seismic excitations. Eng. Optim. 44(12), 1491–1509 (2012)

    Article  Google Scholar 

  30. S.C. Billups, J. Larson, P. Graf, Derivative-free optimization of expensive functions with computational error using weighted regression. SIAM J. Optim. 23(1), 27–53 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  31. M. Björkman, K. Holmström, Global optimization of costly nonconvex functions using radial basis functions. Optim. Eng. 1, 373–397 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  32. A.J. Booker, Well-conditioned Kriging models for optimization of computer simulations. Technical Report M&CT-TECH-00-002, Boeing Computer Services, Research and Technology, M/S 7L–68, Seattle, Washington 98124 (2000)

    Google Scholar 

  33. A.J. Booker, J.E. Dennis Jr., P.D. Frank, D.W. Moore, D.B. Serafini, Managing surrogate objectives to optimize a helicopter rotor design – further experiments, in AIAA Paper 1998–4717, Presented at the 8th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, St. Louis (1998)

    Google Scholar 

  34. A.J. Booker, J.E. Dennis Jr., P.D. Frank, D.B. Serafini, V. Torczon, Optimization using surrogate objectives on a helicopter test example, in Optimal Design and Control, ed. by J. Borggaard, J. Burns, E. Cliff, S. Schreck. Progress in Systems and Control Theory (Birkhäuser, Cambridge, MA, 1998), pp. 49–58

    Google Scholar 

  35. A.J. Booker, J.E. Dennis Jr., P.D. Frank, D.B. Serafini, V. Torczon, M.W. Trosset, A rigorous framework for optimization of expensive functions by surrogates. Struct. Multidiscip. Optim. 17(1), 1–13 (1999)

    Article  Google Scholar 

  36. G.E.P. Box, Evolutionary operation: a method for increasing industrial productivity. Appl. Stat. 6(2), 81–101 (1957)

    Article  Google Scholar 

  37. Á. B\(\mathop{\mathrm{u}}\limits^{\textquotedblright }\) rmen, J. Puhan, T. Tuma, Grid restrained Nelder-Mead algorithm. Comput. Optim. Appl. 34(3), 359–375 (2006)

    Google Scholar 

  38. P.J. Carreau, D. De Kee, R.P. Chhabra, Rheology of Polymeric Systems (PWS Kent, Boston, 1993)

    Google Scholar 

  39. X. Chen, C.T. Kelley, Optimization with hidden constraints and embedded Monte Carlo computations. Optim. Eng. 17(1), 157–175 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  40. T.D. Choi, O.J. Eslinger, C.T. Kelley, J.W. David, M. Etheridge, Optimization of automotive valve train components with implicit filtering. Optim. Eng. 1(1), 9–27 (2000)

    Article  MATH  Google Scholar 

  41. F.H. Clarke, Optimization and Nonsmooth Analysis (Wiley, New York, 1983). Reissued in 1990 by SIAM Publications, Philadelphia, as vol. 5 in the series Classics in Applied Mathematics

    Google Scholar 

  42. A.R. Conn, S. Le Digabel, Use of quadratic models with mesh-adaptive direct search for constrained black box optimization. Optim. Methods Softw. 28(1), 139–158 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  43. A.R. Conn, P.L. Toint, Nonlinear optimization and applications, in An Algorithm using Quadratic Interpolation for Unconstrained Derivative Free Optimization (Springer, Berlin, 1996), pp. 27–47

    MATH  Google Scholar 

  44. A.R. Conn, N.I.M. Gould, P.L. Toint, Trust-Region Methods. MPS-SIAM Series on Optimization (SIAM, Providence, 2000)

    Google Scholar 

  45. A.R. Conn, K. Scheinberg, L.N. Vicente, Geometry of interpolation sets in derivative free optimization. Math. Program. 111(1–2), 141–172 (2008)

    MathSciNet  MATH  Google Scholar 

  46. A.R. Conn, K. Scheinberg, L.N. Vicente, Geometry of sample sets in derivative free optimization: polynomial regression and underdetermined interpolation. IMA J. Numer. Anal. 28(4), 721–749 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  47. A.R. Conn, K. Scheinberg, L.N. Vicente, Introduction to Derivative-Free Optimization. MOS-SIAM Series on Optimization (SIAM, Philadelphia, 2009)

    Google Scholar 

  48. I.D. Coope, C.J. Price, Frame-based methods for unconstrained optimization. J. Optim. Theory Appl. 107(2), 261–274 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  49. I.D. Coope, C.J. Price, Positive bases in numerical optimization. Comput. Optim. Appl. 21(2), 169–175 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  50. E.J. Cramer, J.M. Gablonsky, Effective parallel optimization of complex computer simulations, in Proceedings of the 10th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference (August 2004)

    Google Scholar 

  51. A.L. Custódio, J.F.A. Madeira, Glods: global and local optimization using direct search. J. Glob. Optim. 62(1), 1–28 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  52. A.L. Custódio, H. Rocha, L.N. Vicente, Incorporating minimum Frobenius norm models in direct search. Comput. Optim. Appl. 46(2), 265–278 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  53. A.L. Custódio, J.F.A. Madeira, A.I.F. Vaz, L.N. Vicente, Direct multisearch for multiobjective optimization. SIAM J. Optim. 21(3), 1109–1140 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  54. A.L. Custódio, K. Scheinberg, L.N. Vicente, Methodologies and software for derivative-free optimization, in Advances and Trends in Optimization with Engineering Applications, ed. by T. Terlaky, M.F. Anjos, S. Ahmed. MOS-SIAM Book Series on Optimization, Chap. 37 (SIAM, Philadelphia, 2017)

    Google Scholar 

  55. G.B. Dantzig, Linear Programming and Extensions (Princeton University Press, Princeton, 1963)

    Book  MATH  Google Scholar 

  56. I. Das, J.E. Dennis Jr., Normal-boundary intersection: a new method for generating the Pareto surface in nonlinear multicriteria optimization problems. SIAM J. Optim. 8(3), 631–657 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  57. C. Davis, Theory of positive linear dependence. Am. J. Math. 76, 733–746 (1954)

    Article  MathSciNet  MATH  Google Scholar 

  58. J.E. Dennis Jr., V. Torczon, Direct search methods on parallel machines. SIAM J. Optim. 1(4), 448–474 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  59. J.E. Dennis Jr., V. Torczon, Managing approximation models in optimization, in Multidisciplinary Design Optimization: State of the Art, ed. by N.M. Alexandrov, M.Y. Hussaini (SIAM, Philadelphia, 1997), pp. 330–347

    Google Scholar 

  60. J.E. Dennis Jr., D.J. Woods, Optimization on microcomputers: the Nelder–Mead simplex algorithm, New Computing Environments: Microcomputers in Large-Scale Computing, ed. by A. Wouk (Society for Industrial and Applied Mathematics, Philadelphia, 1987), pp. 116–122

    Google Scholar 

  61. Y. Diouane, S. Gratton, X. Vasseur, L.N. Vicente, H. Calandra, A parallel evolution strategy for an earth imaging problem in geophysics. Optim. Eng. 17(1), 3–26 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  62. M. Dodangeh, L.N. Vicente, Z. Zhang, On the optimal order of worst case complexity of direct search. Optim. Lett. 10(4), 699–708 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  63. E.D. Dolan, J.J. Moré, Benchmarking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  64. E.D. Dolan, R.M. Lewis, V. Torczon, On the local convergence of pattern search. SIAM J. Optim. 14(2), 567–583 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  65. D.W. Dreisigmeyer, Direct search methods over Riemannian manifolds. Technical Report LA-UR-06-7416, Los Alamos National Laboratory, Los Alamos (2006)

    Google Scholar 

  66. D.W. Dreisigmeyer, Equality constraints, Riemannian manifolds and direct search methods. Technical Report LA-UR-06-7406, Los Alamos National Laboratory, Los Alamos (2006)

    Google Scholar 

  67. D.W. Dreisigmeyer, Direct search algorithms over Lipschitz manifolds. Technical Report LA-UR-07-1073, Los Alamos National Laboratory, Los Alamos (2007)

    Google Scholar 

  68. E. Fermi, N. Metropolis, Numerical solution of a minimum problem. Los Alamos Unclassified Report LA–1492, Los Alamos National Laboratory, Los Alamos (1952)

    Google Scholar 

  69. D.E. Finkel, C.T. Kelley, Convergence analysis of the DIRECT algorithm. Technical Report CRSC-TR04-28, Center for Research in Scientific Computation (2004)

    Google Scholar 

  70. D.E. Finkel, C.T. Kelley, Additive scaling and the DIRECT algorithm. J. Glob. Optim. 36, 597–608 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  71. D.E. Finkel, C.T. Kelley, Convergence analysis of sampling methods for perturbed Lipschitz functions. Pac. J. Optim. 5(2), 339–350 (2009)

    MathSciNet  MATH  Google Scholar 

  72. P.J. Fleming, R.C. Purshouse, Evolutionary algorithms in control systems engineering: a survey. Control Eng. Pract. 10(11), 1223–1241 (2002)

    Article  Google Scholar 

  73. R. Fletcher, S. Leyffer, Nonlinear programming without a penalty function. Math. Program. Ser. A 91, 239–269 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  74. R. Fletcher, N.I.M. Gould, S. Leyffer, P.L. Toint, A. Wächter, On the global convergence of trust-region SQP-filter algorithms for general nonlinear programming. SIAM J. Optim. 13(3), 635–659 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  75. R. Fletcher, S. Leyffer, P.L. Toint, On the global convergence of a filter—SQP algorithm. SIAM J. Optim. 13(1), 44–59 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  76. A. Fortin, Analyse numérique pour ingénieurs, deuxième édition. Éditions de l’École Polytechnique de Montréal (2001)

    Google Scholar 

  77. K.R. Fowler, C.T. Kelley, C.T. Miller, C.E. Kees, R.W. Darwin, J.P. Reese, M.W. Farthing, M.S.C. Reed, Solution of a well-field design problem with implicit filtering. Optim. Eng. 5(2), 207–234 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  78. A.F. Freitas, A survey of evolutionary algorithms for data mining and knowledge discovery, in Advances in Evolutionary Computing: Theory and Applications, ed. by A. Ghosh, S. Tsutsui (Springer, New York, 2003), pp. 819–845

    Chapter  Google Scholar 

  79. J.M. Gablonsky, C.T. Kelley, A Locally-biased form of the DIRECT algorithm. J. Glob. Optim. 21(1), 27–37 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  80. U.M. García-Palomares, J.F. Rodríguez, New sequential and parallel derivative-free algorithms for unconstrained optimization. SIAM J. Optim. 13(1), 79–96 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  81. R. Garmanjani, L.N. Vicente, Smoothing and worst-case complexity for direct-search methods in nonsmooth optimization. IMA J. Numer. Anal. 33, 1008–1028 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  82. R. Garmanjani, D. Júdice, L.N. Vicente, Trust-region methods without using derivatives: worst case complexity and the nonsmooth case. SIAM J. Optim. 26(4), 1987–2011 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  83. S. Gill, A process for the step-by-step integration of differential equations in an automatic digital computing machine. Proc. Camb. Philos. Soc. 47, 95–108 (1951)

    Article  MathSciNet  MATH  Google Scholar 

  84. P. Gilmore, T.D. Choi, O. Eslinger, C.T. Kelley, H.A. Patrick, J.M. Gablonsky, IFFCO (implicit filtering for constrained optimization). Software available at http://www4.ncsu.edu/~ctk/iffco.html

  85. P. Gilmore, C.T. Kelly, C.T. Miller, G.A. Williams, Implicit filtering and optimal design problems, in Optimal Design and Control, ed. by J. Borggaard, J. Burkhardt, M. Gunzberger, J. Peterson. Progress in Systems and Control Theory, vol. 19 (Birkhäuser, Cambridge, 1995), pp. 159–176

    Google Scholar 

  86. R.B. Gramacy, S. Le Digabel, The mesh adaptive direct search algorithm with treed Gaussian process surrogates. Pac. J. Optim. 11(3), 419–447 (2015)

    MathSciNet  MATH  Google Scholar 

  87. S. Gratton, C.W. Royer, L.N. Vicente, A second-order globally convergent direct-search method and its worst-case complexity. Optimization 65(6), 1105–1128 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  88. G.A. Gray, T.G. Kolda, Algorithm 856: APPSPACK 4.0: asynchronous parallel pattern search for derivative-free optimization. ACM Trans. Math. Softw. 32(3), 485–507 (2006)

    Google Scholar 

  89. J.D. Griffin, K.R. Fowler, G.A. Gray, T. Hemker, M.D. Parno, Derivative-free optimization via evolutionary algorithms guiding local search (EAGLS) for MINLP. Pac. J. Optim. 7(3), 425–442 (2011)

    MathSciNet  MATH  Google Scholar 

  90. W.L. Hare, Using derivative free optimization for constrained parameter selection in a home and community care forecasting model, in International Perspectives on Operations Research and Health Care. Proceedings of the 34th Meeting of the EURO Working Group on Operational Research Applied to Health Sciences, pp. 61–73 (2010)

    Google Scholar 

  91. W.L. Hare, Y. Lucet, Derivative-free optimization via proximal point methods. J. Optim. Theory Appl. 160(1), 204–220 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  92. W. Hare, M. Macklem, Derivative-free optimization methods for finite minimax problems. Optim. Methods Softw. 28(2), 300–312 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  93. W. Hare, J. Nutini, A derivative-free approximate gradient sampling algorithm for finite minimax problems. Comput. Optim. Appl. 56(1), 1–38 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  94. W. Hare, H. Song, On the cardinality of positively linearly independent sets. Optim. Lett. 10(4), 649–654 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  95. W. Hare, J. Nutini, S. Tesfamariam, A survey of non-gradient optimization methods in structural engineering. Adv. Eng. Softw. 59, 19–28 (2013)

    Article  Google Scholar 

  96. R.E. Hayes, F.H. Bertrand, C. Audet, S.T. Kolaczkowski, Catalytic combustion kinetics: using a direct search algorithm to evaluate kinetic parameters from light-off curves. Can. J. Chem. Eng. 81(6), 1192–1199 (2003)

    Article  Google Scholar 

  97. J.-B. Hiriart-Urruty, C. Lemaréchal, Convex Analysis and Minimization Algorithms (Springer, Berlin, 1993)

    MATH  Google Scholar 

  98. J.H. Holland, Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence (University of Michigan Press, Ann Arbor, 1975)

    MATH  Google Scholar 

  99. R. Hooke, T.A. Jeeves, “Direct search” solution of numerical and statistical problems. J. Assoc. Comput. Mach. 8(2), 212–229 (1961)

    Article  MATH  Google Scholar 

  100. H.H. Hoos, Automated algorithm configuration and parameter tuning, in Autonomous Search, ed. by Y. Hamadi, E. Monfroy, F. Saubion (Springer, Berlin, 2012), pp. 37–71

    Google Scholar 

  101. E.R. Hruschka, R.J.G.B. Campello, A.A. Freitas, A.C. Ponce Leon F. de Carvalho, A survey of evolutionary algorithms for clustering. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 39(2), 133–155 (2009)

    Google Scholar 

  102. J. Jahn, Introduction to the Theory of Nonlinear Optimization (Springer, Berlin, 1994)

    Book  MATH  Google Scholar 

  103. D.R. Jones, M. Schonlau, W.J. Welch, Efficient global optimization of expensive black box functions. J. Glob. Optim. 13(4), 455–492 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  104. C.T. Kelley, Detection and remediation of stagnation in the Nelder–Mead algorithm using a sufficient decrease condition. SIAM J. Optim. 10(1), 43–55 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  105. C.T. Kelley, Implicit Filtering (Society for Industrial and Applied Mathematics, Philadelphia, 2011)

    Book  MATH  Google Scholar 

  106. J. Kennedy, R. Eberhart, Particle swarm optimization, in Proceedings of the 1995 IEEE International Conference on Neural Networks, pp. 1942–1948, Perth (IEEE Service Center, Piscataway, 1995)

    Google Scholar 

  107. S. Kirkpatrick, C.D. Gelatt Jr., M.P. Vecchi, Optimization by simulated annealing. Science 220(4598), 671–680 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  108. M. Kokkolaras, C. Audet, J.E. Dennis Jr., Mixed variable optimization of the number and composition of heat intercepts in a thermal insulation system. Optim. Eng. 2(1), 5–29 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  109. T.G. Kolda, R.M. Lewis, V. Torczon, Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45(3), 385–482 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  110. T.G. Kolda, R.M. Lewis, V. Torczon, Stationarity results for generating set search for linearly constrained optimization. SIAM J. Optim. 17(4), 943–968 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  111. J. Larson, M. Menickelly, S.M. Wild, Manifold sampling for ℓ 1 nonconvex optimization. SIAM J. Optim. 26(4), 2540–2563 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  112. S. Le Digabel, Algorithm 909: NOMAD: nonlinear optimization with the MADS algorithm. ACM Trans. Math. Softw. 37(4), 44:1–44:15 (2011)

    Google Scholar 

  113. S. Le Digabel, S.M. Wild, A taxonomy of constraints in simulation-based optimization. Technical Report G-2015-57, Les cahiers du GERAD (2015)

    Google Scholar 

  114. E.B. Leach, A note on inverse function theorem, in Proceedings of the American Mathematical Society, vol. 12, pp. 694–697 (1961)

    Article  MathSciNet  MATH  Google Scholar 

  115. R.M. Lewis, V. Torczon, Rank ordering and positive bases in pattern search algorithms. Technical Report 96–71, Institute for Computer Applications in Science and Engineering, Mail Stop 132C, NASA Langley Research Center, Hampton, Virginia 23681–2199 (1996)

    Google Scholar 

  116. R.M. Lewis, V. Torczon, Rank ordering and positive bases in pattern search algorithms. Technical Report TR96-71, ICASE, NASA Langley Research Center (1999)

    Google Scholar 

  117. R.M. Lewis, V. Torczon, M.W. Trosset, Why pattern search works. Optima 59, 1–7 (1998). Also available as ICASE Technical Report 98–57. ICASE, Mail Stop 132C, NASA Langley Research Center, Hampton, Virginia 23681–2199

    Google Scholar 

  118. R.M. Lewis, V. Torczon, M.W. Trosset, Direct search methods: then and now. J. Comput. Appl. Math. 124(1–2), 191–207 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  119. Q. Liu, J. Zeng, G. Yang, MrDIRECT: a multilevel robust DIRECT algorithm for global optimization problems. J. Glob. Optim. 62(2), 205–227 (2015)

    MathSciNet  MATH  Google Scholar 

  120. G. Liuzzi, S. Lucidi, F. Rinaldi, Derivative-free methods for mixed-integer constrained optimization problems. J. Optim. Theory Appl. 164(3), 933–965 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  121. S. Lophaven, H. Nielsen, J. Søondergaard, Dace: a matlab Kriging toolbox version 2.0. Technical Report IMM-REP-2002-12, Informatics and Mathematical Modelling, Technical University of Denmark (2002)

    Google Scholar 

  122. S. Lucidi, M. Sciandrone, On the global convergence of derivative-free methods for unconstrained optimization. SIAM J. Optim. 13(1), 97–116 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  123. S. Lucidi, V. Piccialli, M. Sciandrone, An algorithm model for mixed variable programming. SIAM J. Optim. 15(4), 1057–1084 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  124. J.M. Martínez, F.N.C. Sobral, Constrained derivative-free optimization on thin domains. J. Glob. Optim. 56(3), 1217–1232 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  125. K.I.M. McKinnon, Convergence of the Nelder-Mead simplex method to a nonstationary point. SIAM J. Optim. 9, 148–158 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  126. J.C. Meza, M.L. Martinez, On the use of direct search methods for the molecular conformation problem. J. Comput. Chem. 15, 627–632 (1994)

    Article  Google Scholar 

  127. J.C. Meza, R.S. Judson, T.R. Faulkner, A.M. Treasurywala, A comparison of a direct search method and a genetic algorithm for conformational searching. J. Comput. Chem. 17(9), 1142–1151 (1996)

    Article  Google Scholar 

  128. J.J. Moré, S.M. Wild, Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  129. J. Müller, MISO: mixed-integer surrogate optimization framework. Optim. Eng. 17(1), 177–203 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  130. J. Müller, C.A. Shoemaker, R. Piché, SO-MI: a surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems. Comput. Oper. Res. 40(5), 1383–1400 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  131. L. Nazareth, P. Tseng, Gilding the lily: a variant of the Nelder–Mead algorithm based on golden-section search. Comput. Optim. Appl. 22, 133–144 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  132. J.A. Nelder, R. Mead, A simplex method for function minimization. Comput. J. 7(4), 308–313 (1965)

    Article  MathSciNet  MATH  Google Scholar 

  133. J. Nocedal, S.J. Wright, Numerical Optimization. Springer Series in Operations Research (Springer, New York, 1999)

    Google Scholar 

  134. R. Oeuvray, M. Bierlaire, Boosters: a derivative-free algorithm based on radial basis functions. Int. J. Model. Simul. 29(1), 26–36 (2009)

    Article  Google Scholar 

  135. E. Polak, M. Wetter, Precision control for generalized pattern search algorithms with adaptive precision function evaluations. SIAM J. Optim. 16(3), 650–669 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  136. M. Porcelli, P.L. Toint, BFO, a trainable derivative-free Brute Force Optimizer for nonlinear bound-constrained optimization and equilibrium computations with continuous and discrete variables. ACM Trans. Math. Softw. Article 6, 44:1, pp. 25 (2017)

    Google Scholar 

  137. M.J.D. Powell, A view of unconstrained minimization algorithms that do not require derivatives. ACM Trans. Math. Softw. 1(2), 97–107 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  138. M.J.D. Powell, A direct search optimization method that models the objective and constraint functions by linear interpolation, in Advances in Optimization and Numerical Analysis. Proceedings of the 6th Workshop on Optimization and Numerical Analysis, Oaxaca, Mexico, ed. by S. Gomez, J.-P. Hennart, vol. 275 (Kluwer Academic Publishers, Dordrecht, 1994), pp. 51–67

    Google Scholar 

  139. M.J.D. Powell, UOBYQA: unconstrained optimization by quadratic approximation. Math. Program. 92(3), 555–582 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  140. M.J.D. Powell, The NEWUOA software for unconstrained optimization without derivatives, in Large-Scale Nonlinear Optimization, ed. by P. Pardalos, G. Pillo, M. Roma. Nonconvex Optimization and Its Applications, vol. 83 (Springer, Berlin, 2006), pp. 255–297

    Google Scholar 

  141. M.J.D. Powell, The BOBYQA algorithm for bound constrained optimization without derivatives. Technical report, Department of Applied Mathematics and Theoretical Physics, Cambridge University (2009)

    Google Scholar 

  142. C.J. Price, I.D. Coope, Frames and grids in unconstrained and linearly constrained optimization: a nonsmooth approach. SIAM J. Optim. 14, 415–438 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  143. A. Ralston, Runge-Kutta methods with minimum error bounds. Math. Comput. 16, 431–437 (1962)

    Article  MathSciNet  MATH  Google Scholar 

  144. I. Rechenberg, Evolutionsstrategie: optimierung technischer systeme nach prinzipien der biologischen evolution. PhD thesis, Technische Universität Berlin, Berlin (1971)

    Google Scholar 

  145. R.G. Regis, The calculus of simplex gradients. Optim. Lett. 9(5), 845–865 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  146. R.G. Regis, Multi-objective constrained black-box optimization using radial basis function surrogates. J. Comput. Sci. 16, 140–155 (2016)

    Article  MathSciNet  Google Scholar 

  147. R.G. Regis, On the convergence of adaptive stochastic search methods for constrained and multi-objective black-box optimization. J. Optim. Theory Appl. 170(3), 932–959 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  148. R.G. Regis, On the properties of positive spanning sets and positive bases. Optim. Eng. 17(1), 229–262 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  149. R.G. Regis, C.A. Shoemaker, Constrained global optimization of expensive black box functions using radial basis functions. J. Glob. Optim. 31, 153–171 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  150. R.T. Rockafellar, Convex Analysis (Princeton University Press, Princeton, 1970)

    Book  MATH  Google Scholar 

  151. R.T. Rockafellar, Generalized directional derivatives and subgradients of nonconvex functions. Can. J. Math. 32(2), 257–280 (1980)

    Article  MathSciNet  MATH  Google Scholar 

  152. P.R. Sampaio, P.L. Toint, A derivative-free trust-funnel method for equality-constrained nonlinear optimization. Comput. Optim. Appl. 61(1), 25–49 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  153. S.E. Selvan, P.B. Borckmans, A. Chattopadhyay, P.-A. Absil, Spherical mesh adaptive direct search for separating quasi-uncorrelated sources by range-based independent component analysis. Neural Comput. 25(9), 2486–2522 (2013)

    Article  MathSciNet  Google Scholar 

  154. J. Søndergaard, Optimization using surrogate models — by the space mapping technique. PhD thesis, Informatics and Mathematical Modelling, Technical University of Denmark (2003)

    Google Scholar 

  155. M.C. Spillane, E. Gica, V.V. Titov, Tsunameter network design for the U.S. DART array. AGU Fall Meeting Abstracts, p. A1368 (December 2009)

    Google Scholar 

  156. P. Stein, Classroom notes: a note on the volume of a simplex. Am. Math. Mon. 73(3), 299–301 (1966)

    Article  MathSciNet  Google Scholar 

  157. M. Strasser, Übertrangung des Optimierungsverfahrens von Nelder und Mead auf restringierte Probleme. Diploma thesis, Numerical Mathematics Group, Technical University of Darmstadt (1994)

    Google Scholar 

  158. The CUTEr/st test problem set. http://www.cuter.rl.ac.uk/Problems/mastsif.shtml

  159. V. Torczon, On the convergence of pattern search algorithms. SIAM J. Optim. 7(1), 1–25 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  160. V. Torczon, M.W. Trosset, From evolutionary operation to parallel direct search: pattern search algorithms for numerical optimization. Comput. Sci. Stat. 29, 396–401 (1998)

    Google Scholar 

  161. M.W. Trosset, I know it when I see it: toward a definition of direct search methods. SIAG/OPT Views-and-News: Forum SIAM Activity Group Optim. 9, 7–10 (1997)

    Google Scholar 

  162. P. Tseng, Fortified-descent simplicial search method: a general approach. SIAM J. Optim. 10, 269–288 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  163. B. Van Dyke, T.J. Asaki, Using QR decomposition to obtain a new instance of mesh adaptive direct search with uniformly distributed polling directions. J. Optim. Theory Appl. 159(3), 805–821 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  164. L.N. Vicente, Worst case complexity of direct search. EURO J. Comput. Optim. 1(1), 143–153 (2013)

    Article  MATH  Google Scholar 

  165. L.N. Vicente, A.L. Custódio, Analysis of direct searches for discontinuous functions. Math. Program. 133(1–2), 299–325 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  166. S.M. Wild, C.A. Shoemaker, Global convergence of radial basis function trust region derivative-free algorithms. SIAM J. Optim. 21(3), 761–781 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  167. S.M. Wild, R.G. Regis, C.A. Shoemaker, ORBIT: optimization by radial basis function interpolation in trust-regions. SIAM J. Sci. Comput. 30(6), 3197–3219 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  168. D. Winfield, Function and functional optimization by interpolation in data tables. PhD thesis, Harvard University (1969)

    Google Scholar 

  169. D. Winfield, Function minimization by interpolation in a data table. J. Inst. Math. Appl. 12, 339–347 (1973)

    Article  MathSciNet  MATH  Google Scholar 

  170. T.A. Winslow, R.J. Trew, P. Gilmore, C.T. Kelley, Doping profiles for optimum class B performance of GaAs MESFET amplifiers, in Proceedings IEEE/Cornell Conference on Advanced Concepts in High Speed Devices and Circuits, pp. 188–197 (1991)

    Google Scholar 

  171. M.H. Wright, Direct search methods: once scorned, now respectable, in Numerical Analysis 1995 (Proceedings of the 1995 Dundee Biennial Conference in Numerical Analysis), ed. by D.F. Griffiths, G.A. Watson. Pitman Research Notes in Mathematics, vol. 344 (CRC Press, Boca Raton, 1996), pp. 191–208

    Google Scholar 

  172. M.H. Wright, Nelder, Mead, and the other simplex method. Documenta Math. Extra Volume: Optimization Stories, 271–276 (2012)

    Google Scholar 

  173. W.-C. Yu, Positive basis and a class of direct search techniques. Sci. Sinica, Special Issue of Mathematics 1, 53–67 (1979)

    Google Scholar 

  174. A. Zhou, B.-Y. Qu, H. Li, S.-Z. Zhao, P.N. Suganthan, Q. Zhang, Multiobjective evolutionary algorithms: a survey of the state of the art. Swarm Evol. Comput. 1(1), 32–49 (2011)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dépt. Mathématiques et Génie Industriel, Ecole Polytechnique de Montréal, Montréal, Québec, Canada

    Charles Audet

  2. Department of Mathematics, University of British Columbia, Kelowna, BC, Canada

    Warren Hare

Authors
  1. Charles Audet
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Warren Hare
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Audet, C., Hare, W. (2017). Introduction: Tools and Challenges in Derivative-Free and Blackbox Optimization. In: Derivative-Free and Blackbox Optimization. Springer Series in Operations Research and Financial Engineering. Springer, Cham. https://doi.org/10.1007/978-3-319-68913-5_1

Download citation

Publish with us

Policies and ethics

Search

Navigation