Download PDF

Derivative-free optimization: a review of algorithms and comparison of software implementations

Journal of Global Optimization volume 56, pages1247–1293(2013)Cite this article

Abstract

This paper addresses the solution of bound-constrained optimization problems using algorithms that require only the availability of objective function values but no derivative information. We refer to these algorithms as derivative-free algorithms. Fueled by a growing number of applications in science and engineering, the development of derivative-free optimization algorithms has long been studied, and it has found renewed interest in recent time. Along with many derivative-free algorithms, many software implementations have also appeared. The paper presents a review of derivative-free algorithms, followed by a systematic comparison of 22 related implementations using a test set of 502 problems. The test bed includes convex and nonconvex problems, smooth as well as nonsmooth problems. The algorithms were tested under the same conditions and ranked under several criteria, including their ability to find near-global solutions for nonconvex problems, improve a given starting point, and refine a near-optimal solution. A total of 112,448 problem instances were solved. We find that the ability of all these solvers to obtain good solutions diminishes with increasing problem size. For the problems used in this study, TOMLAB/MULTIMIN, TOMLAB/GLCCLUSTER, MCS and TOMLAB/LGO are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations. These global solvers outperform local solvers even for convex problems. Finally, TOMLAB/OQNLP, NEWUOA, and TOMLAB/MULTIMIN show superior performance in terms of refining a near-optimal solution.

References

  1. 1

    Aarts E.H.L., van Laarhoven P.J.M.: Statistical cooling: a general approach to combinatorial optimization problems. Phillips J. Res. 40, 193–226 (1985)

    Google Scholar 

  2. 2

    Abramson, M.A.: Pattern Search Algorithms for Mixed Variable General Constrained Optimization Problems. PhD thesis, Department of Computational and Applied Mathematics, Rice University, Houston (2002, Aug)

  3. 3

    Abramson, M.A.: NOMADm Version 4.5 User’s Guide. Air Force Institute of Technology, Wright-Patterson AFB, OH (2007)

  4. 4

    Abramson M.A., Asaki T.J., Dennis J.E. Jr., O’Reilly K.R., Pingel R.L.: Quantitative object reconstruction via Abel-based X-ray tomography and mixed variable optimization. SIAM J. Imaging Sci. 1, 322–342 (2008)

    Article  Google Scholar 

  5. 5

    Abramson M.A., Audet C.: Convergence of mesh adaptive direct search to second-order stationary points. SIAM J. Optim. 17, 606–609 (2006)

    Article  Google Scholar 

  6. 6

    Abramson, M.A., Audet, C., Couture, G., Dennis, J.E. Jr., LeDigabel, S.: The Nomad project. http://www.gerad.ca/nomad/

  7. 7

    Abramson M.A., Audet C., Dennis J.E. Jr: Filter pattern search algorithms for mixed variable constrained optimization problems. Pac. J. Optim. 3, 477–500 (2007)

    Google Scholar 

  8. 8

    Abramson M.A., Audet C., Dennis J.E. Jr, Le Digabel S.: OrthoMADS: a deterministic MADS instance with orthogonal directions. SIAM J. Optim. 20, 948–966 (2009)

    Article  Google Scholar 

  9. 9

    Audet C.: Convergence results for generalized pattern search algorithms are tight. Optim. Eng. 5, 101–122 (2004)

    Article  Google Scholar 

  10. 10

    Audet C., Béchard V., Chaouki J.: Spent potliner treatment process optimization using a MADS algorithm. Optim. Eng. 9, 143–160 (2008)

    Article  Google Scholar 

  11. 11

    Audet C., Dennis J.E. Jr: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17, 188–217 (2006)

    Article  Google Scholar 

  12. 12

    Audet C., Dennis J.E. Jr: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20, 445–472 (2009)

    Article  Google Scholar 

  13. 13

    Awasthi, S.: Molecular Docking by Derivative-Free Optimization Solver. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2008)

  14. 14

    Barros P.A. Jr, Kirby M.R., Mavris D.N.: Impact of sampling techniques selection on the creation of response surface models. SAE Trans. J. Aerosp. 113, 1682–1693 (2004)

    Google Scholar 

  15. 15

    Bartholomew-Biggs M.C., Parkhurst S.C., Wilson S.P.: Using DIRECT to solve an aircraft routing problem. Comput. Optim. Appl. 21, 311–323 (2002)

    Article  Google Scholar 

  16. 16

    Barton, R.R.: Metamodeling: A state of the art review. In: Proceedings of the 1994 Winter Simulation Conference, pp. 237–244 (1994)

  17. 17

    Bélisle C.J., Romeijn H.E., Smith R.L.: Hit-and-run algorithms for generating multivariate distributions. Math. Oper. Res. 18, 255–266 (1993)

    Article  Google Scholar 

  18. 18

    Bethke, J.D.: Genetic Algorithms as Function Optimizers. PhD thesis, Department of Computer and Communication Sciences, University of Michigan, Ann Arbor (1980)

  19. 19

    Björkman M., Holmström K.: Global optimization of costly nonconvex functions using radial basis functions. Optim. Eng. 1, 373–397 (2000)

    Article  Google Scholar 

  20. 20

    Boender C.G.E., Rinnooy Kan A.H.G., Timmer G.T.: A stochastic method for global optimization. Math. Program. 22, 125–140 (1982)

    Article  Google Scholar 

  21. 21

    Boneh, A., Golan, A.: Constraints’ redundancy and feasible region boundedness by random feasible point generator (RFPG). In: 3rd European Congress on Operations Research (EURO III), Amsterdam (1979)

  22. 22

    Booker, A.J., Dennis, J.E., Jr., Frank, P.D., Serafini, D.B., Torczon, V.J., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. In: ICASE Report, pp. 1–24 (1998)

  23. 23

    Booker A.J., Dennis J.E. Jr, Frank P.D., Serafini D.B., Torczon V.J., Trosset M.W.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Optim. 17, 1–13 (1999)

    Article  Google Scholar 

  24. 24

    Booker A.J., Meckesheimer M., Torng T.: Reliability based design optimization using design explorer. Optim. Eng. 5, 179–205 (2004)

    Article  Google Scholar 

  25. 25

    Brent R.P.: Algorithms for Minimization without Derivatives. Prentice-Hall, Englewood Cliffs (1973)

    Google Scholar 

  26. 26

    Chang, K.-F.: Modeling and Optimization of Polymerase Chain Reaction Using Derivative-Free Optimization. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)

  27. 27

    Chiang T., Chow Y.: A limit theorem for a class of inhomogeneous Markov processes. Ann. Probab. 17, 1483–1502 (1989)

    Article  Google Scholar 

  28. 28

    COIN-OR Project. Derivative Free Optimization. http://projects.coin-or.org/Dfo

  29. 29

    COIN-OR Project. IPOPT 2.3.x A software package for large-scale nonlinear optimization. http://www.coin-or.org/Ipopt/ipopt-fortran.html

  30. 30

    Conn A.R., Gould N., Lescrenier M., Toint Ph.L.: Performance of a multifrontal scheme for partially separable optimization. In: Gomez, S., Hennart, J.-P. (eds) Advances in Optimization and Numerical Analysis, pp. 79–96. Kluwer, Dordrecht (1994)

    Google Scholar 

  31. 31

    Conn A.R., Scheinberg K., Toint P.L.: On the convergence of derivative-free methods for unconstrained optimization. In: Buhmann, M.D., Iserles, A. (eds) Approximation Theory and Optimization, Tribute to M. J. D. Powell, pp. 83–108. Cambridge University Press, Cambridge (1996)

    Google Scholar 

  32. 32

    Conn A.R., Scheinberg K., Toint P.L.: Recent progress in unconstrained nonlinear optimization without derivatives. Math. Program. 79, 397–414 (1997)

    Google Scholar 

  33. 33

    Conn, A.R., Scheinberg, K., Toint, P.L.: A derivative free optimization algorithm in practice. In: Proceedings of AIAA St Louis Conference, pp. 1–11 (1998)

  34. 34

    Conn A.R., Scheinberg K., Vicente L.N.: Global convergence of general derivative-free trust-region algorithms to first and second order critical points. SIAM J. Optim. 20, 387–415 (2009)

    Article  Google Scholar 

  35. 35

    Conn A.R., Scheinberg K., Vicente L.N.: Introduction to derivative-free optimization. SIAM, Philadelphia (2009)

    Book  Google Scholar 

  36. 36

    Cox, D.D., John, S.: SDO: A statistical method for global optimization. In: Multidisciplinary Design Optimization (Hampton, VA, 1995), pp. 315–329. SIAM, Philadelphia (1997)

  37. 37

    Csendes T., Pál L., Sendín J.O.H., Banga J.R.: The GLOBAL optimization method revisited. Optim. Lett. 2, 445–454 (2008)

    Article  Google Scholar 

  38. 38

    Custódio A.L., Dennis J.E. Jr, Vicente L.N.: Using simplex gradients of nonsmooth functions in direct search methods. IMA J. Numer. Anal. 28, 770–784 (2008)

    Article  Google Scholar 

  39. 39

    Custódio, A.L., Rocha, H., Vicente, L.N.: Incorporating minimum Frobenius norm models in direct search. Comput. Optim. Appl. (to appear)

  40. 40

    Custódio, A.L., Vicente, L.N.: Using sampling and simplex derivatives in pattern search methods. SIAM J. Optim. 18, 537–555 (2007)

    Google Scholar 

  41. 41

    Custódio, A.L., Vicente, L.N.: SID-PSM: A Pattern Search Method Guided by Simplex Derivatives for Use in Derivative-Free Optimization. Departamento de Matemática, Universidade de Coimbra, Coimbra (2008)

  42. 42

    Deming S.N., Parker L.R. Jr, Denton M.B.: A review of simplex optimization in analytical chemistry. Crit. Rev. Anal. Chem. 7, 187–202 (1974)

    Article  Google Scholar 

  43. 43

    Desai, R.: A Comparison of Algorithms for Optimizing the Omega Function. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2010)

  44. 44

    Eberhart, R.,Kennedy, J.: A new optimizer using particle swarm theory. In: Proceedings of the 6th International Symposium on Micro Machine and Human Science, Nagoya, pp. 39–43 (1995)

  45. 45

    Eldred, M.S., Adams, B.M., Gay, D.M., Swiler, L.P., Haskell, K., Bohnhoff, W.J., Eddy, J.P., Hart, W.E., Watson, J-P, Hough, P.D., Kolda, T.G., Williams, P.J., Martinez-Canales, M.L., DAKOTA, A.: Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 4.2 User’s Manual. Sandia National Laboratories, Albuquerque (2008)

  46. 46

    Fan S.S., Zahara E.: A hybrid simplex search and particle swarm optimization for unconstrained optimization. Eur. J. Oper. Res. 181, 527–548 (2007)

    Article  Google Scholar 

  47. 47

    Finkel D.E., Kelley C.T.: Additive scaling and the DIRECT algorithm. J. Glob. Optim. 36, 597–608 (2006)

    Article  Google Scholar 

  48. 48

    Fowler K.R., Reese J.P., Kees C.E., Dennis J.E. Jr, Kelley C.T., Miller C.T., Audet C., Booker A.J., Couture G., Darwin R.W., Farthing M.W., Finkel D.E., Gablonsky J.M., Gray G., Kolda T.G.: A comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems. Adv. Water Resour. 31, 743–757 (2008)

    Article  Google Scholar 

  49. 49

    Gablonsky, J.M.: Modifications of the DIRECT Algorithm. PhD thesis, Department of Mathematics, North Carolina State University, Raleigh (2001)

  50. 50

    Gilmore P., Kelley C.T.: An implicit filtering algorithm for optimization of functions with many local minima. SIAM J. Optim. 5, 269–285 (1995)

    Article  Google Scholar 

  51. 51

    GLOBAL Library. http://www.gamsworld.org/global/globallib.htm

  52. 52

    Gray G., Kolda T., Sale K., Young M.: Optimizing an empirical scoring function for transmembrane protein structure determination. INFORMS J. Comput. 16, 406–418 (2004)

    Article  Google Scholar 

  53. 53

    Gutmann H.-M.: A radial basis function method for global optimization. J. Glob. Optim. 19, 201–227 (2001)

    Article  Google Scholar 

  54. 54

    Han J., Kokkolaras M., Papalambros P.Y.: Optimal design of hybrid fuel cell vehicles. J. Fuel Cell Sci. Technol. 5, 041014 (2008)

    Article  Google Scholar 

  55. 55

    Hansen, N.: The CMA Evolution Strategy: A tutorial. http://www.lri.fr/hansen/cmaesintro.html

  56. 56

    Hansen N.: The CMA evolution strategy: a comparing review. In: Lozano, J.A., Larranaga, P., Inza, I., Bengoetxea, E. (eds) Towards a New Evolutionary Computation. Advances on Estimation of Distribution Algorithms, pp. 75–102. Springer, Berlin (2006)

    Google Scholar 

  57. 57

    Hayes R.E., Bertrand F.H., Audet C., Kolaczkowski S.T.: Catalytic combustion kinetics: using a direct search algorithm to evaluate kinetic parameters from light-off curves. Can. J. Chem. Eng. 81, 1192–1199 (2003)

    Article  Google Scholar 

  58. 58

    Holland J.H.: Adaptation in Natural and Artificial Systems. The University of Michigan Press, Ann Arbor (1975)

    Google Scholar 

  59. 59

    Holmström, K.: Private Communication (2009)

  60. 60

    Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB 7. Tomlab Optimization. http://tomopt.com

  61. 61

    Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/CGO. Tomlab Optimization (2007). http://tomopt.com

  62. 62

    Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/OQNLP. Tomlab Optimization (2007). http://tomopt.com

  63. 63

    Holmström K., Quttineh N.-H., Edvall M.M.: An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer constrained global optimization. Optim. Eng. 9, 311–339 (2008)

    Article  Google Scholar 

  64. 64

    Hooke R., Jeeves T.A.: Direct search solution of numerical and statistical problems. J. Assoc. Comput. Mach. 8, 212–219 (1961)

    Article  Google Scholar 

  65. 65

    Huyer W., Neumaier A.: Global optimization by multilevel coordinate search. J. Glob. Optim. 14, 331–355 (1999)

    Article  Google Scholar 

  66. 66

    Huyer W., Neumaier A.: SNOBFIT—Stable noisy optimization by branch and fit. ACM Trans. Math. Softw. 35, 1–25 (2008)

    Article  Google Scholar 

  67. 67

    Hvattum L.M., Glover F.: Finding local optima of high-dimensional functions using direct search methods. Eur. J. Oper. Res. 195, 31–45 (2009)

    Article  Google Scholar 

  68. 68

    Ingber, L.: Adaptive Simulated Annealing (ASA). http://www.ingber.com/#ASA

  69. 69

    Järvi, T.: A Random Search Optimizer with an Application to a Max–Min Problem. Technical report, Pulications of the Institute for Applied Mathematics, University of Turku (1973)

  70. 70

    Jones D.R.: A taxonomy of global optimization methods based on response surfaces. J. Glob. Optim. 21, 345–383 (2001)

    Article  Google Scholar 

  71. 71

    Jones, D.R.: The DIRECT global optimization algorithm. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization, vol. 1, pp. 431–440. Kluwer, Boston (2001)

  72. 72

    Jones D.R., Perttunen C.D., Stuckman B.E.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl. 79, 157–181 (1993)

    Article  Google Scholar 

  73. 73

    Jones D.R., Schonlau M., Welch W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13, 455–492 (1998)

    Article  Google Scholar 

  74. 74

    Kelley, C.T.: Users Guide for IMFIL version 1.0. http://www4.ncsu.edu/ctk/imfil.html

  75. 75

    Kelley C.T.: Detection and remediation of stagnation in the Nelder–Mead algorithm using a sufficient decrease condition. SIAM J. Optim. 10, 43–55 (1999)

    Article  Google Scholar 

  76. 76

    Kelley C.T.: Iterative Methods for Optimization. SIAM, Philadelphia (1999)

    Book  Google Scholar 

  77. 77

    Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks, Piscataway, pp. 1942–1948 (1995)

  78. 78

    Kirkpatrick S., Gelatt C.D., Vecchi M.P.: Optimization by simulated annealing. Science 220, 671–680 (1983)

    Article  Google Scholar 

  79. 79

    Kolda T.G., Lewis R.M., Torczon V.J.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45, 385–482 (2003)

    Article  Google Scholar 

  80. 80

    Kolda T.G., Torczon V.J.: On the convergence of asynchronous parallel pattern search. SIAM J. Optim. 14, 939–964 (2004)

    Article  Google Scholar 

  81. 81

    Lagarias J.C., Reeds J.A., Wright M.H., Wright P.E.: Convergence properties of the Nelder–Mead simplex method in low dimensions. SIAM J. Optim. 9, 112–147 (1998)

    Article  Google Scholar 

  82. 82

    LeDigabel, S.: NOMAD User Guide Version 3.3. Technical report, Les Cahiers du GERAD (2009)

  83. 83

    Lewis R.M., Torczon V.J.: Pattern search algorithms for bound constrained minimization. SIAM J. Optim. 9, 1082–1099 (1999)

    Article  Google Scholar 

  84. 84

    Lewis R.M., Torczon V.J.: Pattern search algorithms for linearly constrained minimization. SIAM J. Optim. 10, 917–941 (2000)

    Article  Google Scholar 

  85. 85

    Lewis R.M., Torczon V.J.: A globally convergent augmented lagrangian pattern search algorithm for optimization with general constraints and simple bounds. SIAM J. Optim. 12, 1075–1089 (2002)

    Article  Google Scholar 

  86. 86

    Liepins G.E., Hilliard M.R.: Genetic algorithms: foundations and applications. Ann. Oper. Res. 21, 31–58 (1989)

    Article  Google Scholar 

  87. 87

    Lin Y., Schrage L.: The global solver in the LINDO API. Optim. Methods Softw. 24, 657–668 (2009)

    Article  Google Scholar 

  88. 88

    Lucidi S., Sciandrone M.: On the global convergence of derivative-free methods for unconstrained minimization. SIAM J. Optim. 13, 97–116 (2002)

    Article  Google Scholar 

  89. 89

    Lukšan, L., Vlček, J.: Test Problems for Nonsmooth Unconstrained and Linearly Constrained Optimization. Technical report, Institute of Computer Science, Academy of Sciences of the Czech Republic (2000). http://www3.cs.cas.cz/ics/reports/v798-00.ps

  90. 90

    Marsden A.L., Feinstein J.A., Taylor C.A.: A computational framework for derivative-free optimization of cardiovascular geometries. Comput. Methods Appl. Mech. Eng. 197, 1890–1905 (2008)

    Article  Google Scholar 

  91. 91

    Marsden A.L., Wang M., Dennis J.E. Jr, Moin P.: Optimal aeroacustic shape design using the surrogate management framework. Optim. Eng. 5, 235–262 (2004)

    Article  Google Scholar 

  92. 92

    Marsden A.L., Wang M., Dennis J.E. Jr, Moin P.: Trailing-edge noise reduction using derivative-free optimization and large-eddy simulation. J. Fluid Mech. 5, 235–262 (2007)

    Google Scholar 

  93. 93

    Matheron G.: Principles of geostatistics. Econ. Geol. 58, 1246–1266 (1967)

    Article  Google Scholar 

  94. 94

    McKinnon K.I.M.: Convergence of the Nelder–Mead simplex method to a nonstationary point. SIAM J. Optim. 9, 148–158 (1998)

    Article  Google Scholar 

  95. 95

    Metropolis N., Rosenbluth A.W., Rosenbluth M.N., Teller A.H., Teller E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–1092 (1953)

    Article  Google Scholar 

  96. 96

    Mongeau M., Karsenty H., Rouzé V., Hiriart-Urruty J.B.: Comparison of public-domain software for black box global optimization. Optim. Methods Softw. 13, 203–226 (2000)

    Article  Google Scholar 

  97. 97

    Moré, J., Wild, S.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20, 172–191 (2009)

    Google Scholar 

  98. 98

    Mugunthan P., Shoemaker C.A., Regis R.G.: Comparison of function approximation, heuristic, and derivative-based methods for automatic calibration of computationally expensive groundwater bioremediation models. Water Resour. Res. 41, W11427 (2005)

    Article  Google Scholar 

  99. 99

    Nelder J.A., Mead R.: A simplex method for function minimization. Comput. J. 7, 308–313 (1965)

    Article  Google Scholar 

  100. 100

    Nesterov, Y.: Gradient methods for minimizing composite objective function. CORE Discussion Paper 2007/76 (2007)

  101. 101

    Neumaier, A.: MCS: Global Optimization by Multilevel Coordinate Search. http://www.mat.univie.ac.at/neum/software/mcs/

  102. 102

    Neumaier A., Shcherbina O., Huyer W., Vinkó T.: A comparison of complete global optimization solvers. Math. Program. 103, 335–356 (2005)

    Article  Google Scholar 

  103. 103

    Oeuvray, R.: Trust-Region Methods Based on Radial Basis Functions with Application to Biomedical Imaging. PhD thesis, Institute of Mathematics, Swiss Federal Institute of Technology, Lausanne (2005, March)

  104. 104

    Orosz J.E., Jacobson S.H.: Finite-time performance analysis of static simulated annealing algorithms. Comput. Optim. Appl. 21, 21–53 (2002)

    Article  Google Scholar 

  105. 105

    Pintér, J.: Homepage of Pintér Consulting Services. http://www.pinterconsulting.com/

  106. 106

    Pintér J.D.: Global Optimization in Action: Continuous and Lipschitz Optimization. Algorithms, Implementations and Applications. Nonconvex Optimization and its Applications. Kluwer, Dordrecht (1995)

    Google Scholar 

  107. 107

    Pintér, J.D., Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/LGO. Tomlab Optimization (2006). http://tomopt.com

  108. 108

    Plantenga, T.D.: HOPSPACK 2.0 User Manual. Technical Report SAND2009-6265, Sandia National Laboratories, Albuquerque (2009)

  109. 109

    Powell M.J.D.: A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Gomez, S., Hennart, J-P. (eds) (eds.) Advances in Optimization and Numerical Analysis, pp. 51–67. Kluwer, Dordrecht (1994)

    Google Scholar 

  110. 110

    Powell, M.J.D.: A direct search optimization method that models the objective by quadratic interpolation. In: Presentation at the 5th Stockholm Optimization Days (1994)

  111. 111

    Powell, M.J.D.: Recent Research at Cambridge on Radial Basis Functions. Technical report, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1998)

  112. 112

    Powell M.J.D.: UOBYQA: unconstrained optimization by quadratic approximation. Math. Program. 92, 555–582 (2002)

    Article  Google Scholar 

  113. 113

    Powell M.J.D.: The NEWUOA software for unconstrained optimization without derivatives. In: Di Pillo, G., Roma, M. (eds) Large-Scale Nonlinear Optimization, pp. 255–297. Springer, New York (2006)

    Google Scholar 

  114. 114

    Powell M.J.D.: Developments of NEWUOA for minimization without derivatives. IMA J. Numer. Anal. 28, 649–664 (2008)

    Article  Google Scholar 

  115. 115

    Powell, M.J.D.: The BOBYQA Algorithm for Bound Constrained Optimization Without Derivatives. Technical report, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (2009)

  116. 116

    Princeton Library. http://www.gamsworld.org/performance/princetonlib/princetonlib.htm

  117. 117

    Regis R.G., Shoemaker C.A.: Constrained global optimization of expensive black box functions using radial basis functions. J. Glob. Optim. 31, 153–171 (2005)

    Article  Google Scholar 

  118. 118

    Regis R.G., Shoemaker C.A.: Improved strategies for radial basis function methods for global optimization. J. Glob. Optim. 37, 113–135 (2007)

    Article  Google Scholar 

  119. 119

    Richtarik, P.: Improved algorithms for convex minimization in relative scale. SIAM J. Optim. (2010, to appear). http://www.optimization-online.org/DB_FILE/2009/02/2226.pdf

  120. 120

    Rios, L.M.: Algorithms for Derivative-Free Optimization. PhD thesis, Department of Industrial and Enterprise Systems Engineering, University of Illinois, Urbana (2009, May)

  121. 121

    Romeo F., Sangiovanni-Vincentelli A.: A theoretical framework for simulated annealing. Algorithmica 6, 302–345 (1991)

    Article  Google Scholar 

  122. 122

    Sacks J., Welch W.J., Mitchell T.J., Wynn H.P.: Design and analysis of computer experiments. Stat. Sci. 4, 409–423 (1989)

    Article  Google Scholar 

  123. 123

    Sahinidis, N.V., Tawarmalani, M.: BARON 7.5: Global Optimization of Mixed-Integer Nonlinear Programs, User’s Manual (2005)

  124. 124

    Sandia National Laboratories: The Coliny Project. https://software.sandia.gov/trac/acro/wiki/Overview/Projects

  125. 125

    Scheinberg, K.: Manual for Fortran Software Package DFO v2.0 (2003)

  126. 126

    Schonlau, M.: Computer Experiments and Global Optimization. PhD thesis, Department of Statistics, University of Waterloo, Waterloo (1997)

  127. 127

    Serafini, D.B.: A Framework for Managing Models in Nonlinear Optimization of Computationally Expensive Functions. PhD thesis, Department of Computational and Applied Mathematics, Rice University, Houston (1998, Nov)

  128. 128

    Shah, S.B., Sahinidis, N.V.: SAS-Pro: Simultaneous residue assignment and structure superposition for protein structure alignment. PLoS ONE 7(5), e37493 (2012)

  129. 129

    Shubert B.O.: A sequential method seeking the global maximum of a function. SIAM J. Numer. Anal. 9, 379–388 (1972)

    Article  Google Scholar 

  130. 130

    Smith R.L.: Efficient Monte Carlo procedures for generating points uniformly distributed over bounded regions. Oper. Res. 32, 1296–1308 (1984)

    Article  Google Scholar 

  131. 131

    Søndergaard, J.: Optimization Using Surrogate Models—by the Space Mapping Technique. PhD thesis, Technical University of Denmark, Department of Mathematical Modelling, Lingby (2003)

  132. 132

    Spendley W., Hext G.R., Himsworth F.R.: Sequential application for simplex designs in optimisation and evolutionary operation. Technometrics 4, 441–461 (1962)

    Article  Google Scholar 

  133. 133

    Tawarmalani M., Sahinidis N.V.: A polyhedral branch-and-cut approach to global optimization. Math. Program. 103, 225–249 (2005)

    Article  Google Scholar 

  134. 134

    Torczon V.J.: On the convergence of multidirectional search algorithms. SIAM J. Optim. 1, 123–145 (1991)

    Article  Google Scholar 

  135. 135

    Torczon V.J.: On the convergence of pattern search algorithms. SIAM J. Optim. 7, 1–25 (1997)

    Article  Google Scholar 

  136. 136

    Tseng P.: Fortified-descent simplicial search method: a general approach. SIAM J. Optim. 10, 269–288 (1999)

    Article  Google Scholar 

  137. 137

    Vaz, A.I.F.: PSwarm Home Page. http://www.norg.uminho.pt/aivaz/pswarm/

  138. 138

    Vaz A.I.F., Vicente L.N.: A particle swarm pattern search method for bound constrained global optimization. J. Glob. Optim. 39, 197–219 (2007)

    Article  Google Scholar 

  139. 139

    Wang, H.: Application of Derivative-Free Algorithms in Powder Diffraction. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)

  140. 140

    Wild S.M., Regis R.G., Shoemaker C.A.: ORBIT: Optimization by radial basis function interpolation in trust-regions. SIAM J. Sci. Comput. 30, 3197–3219 (2008)

    Article  Google Scholar 

  141. 141

    Winfield, D.: Function and Functional Optimization by Interpolation in Data Tables. PhD thesis, Harvard University, Cambridge (1969)

  142. 142

    Winslow, T.A., Trew, R.J., Gilmore, P., Kelley, C.T.: Simulated performance optimization of gaas mesfet amplifiers. In: IEEE/Cornell Conference on Advanced Concepts in High Speed Semiconductor Devices and Circuits, Piscataway, pp. 393–402 (1991)

  143. 143

    Zhao Z., Meza J.C., Van Hove M.: Using pattern search methods for surface structure determination of nanomaterials. J. Phys. Condens. Matter 18, 8693–8706 (2006)

    Article  Google Scholar 

  144. 144

    Zheng, Y.: Pairs Trading and Portfolio Optimization. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)

Download references

Author information

Affiliations

  1. Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Luis Miguel Rios & Nikolaos V. Sahinidis

Authors
  1. Luis Miguel Rios
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikolaos V. Sahinidis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nikolaos V. Sahinidis.

Electronic Supplementary Material

The Below is the Electronic Supplementary Material.

ESM 1 (PDF 159 kb)

Rights and permissions

This article is published under an open access license. Please check the 'Copyright Information' section either on this page or in the PDF for details of this license and what re-use is permitted. If your intended use exceeds what is permitted by the license or if you are unable to locate the licence and re-use information, please contact the Rights and Permissions team.

About this article

Cite this article

Rios, L.M., Sahinidis, N.V. Derivative-free optimization: a review of algorithms and comparison of software implementations. J Glob Optim 56, 1247–1293 (2013). https://doi.org/10.1007/s10898-012-9951-y

Download citation

Keywords

  • Derivative-free algorithms
  • Direct search methods
  • Surrogate models