Skip to main content
Download PDF

Derivative-free optimization: a review of algorithms and comparison of software implementations

Journal of Global Optimization volume 56, pages 1247–1293 (2013)Cite this article

Abstract

This paper addresses the solution of bound-constrained optimization problems using algorithms that require only the availability of objective function values but no derivative information. We refer to these algorithms as derivative-free algorithms. Fueled by a growing number of applications in science and engineering, the development of derivative-free optimization algorithms has long been studied, and it has found renewed interest in recent time. Along with many derivative-free algorithms, many software implementations have also appeared. The paper presents a review of derivative-free algorithms, followed by a systematic comparison of 22 related implementations using a test set of 502 problems. The test bed includes convex and nonconvex problems, smooth as well as nonsmooth problems. The algorithms were tested under the same conditions and ranked under several criteria, including their ability to find near-global solutions for nonconvex problems, improve a given starting point, and refine a near-optimal solution. A total of 112,448 problem instances were solved. We find that the ability of all these solvers to obtain good solutions diminishes with increasing problem size. For the problems used in this study, TOMLAB/MULTIMIN, TOMLAB/GLCCLUSTER, MCS and TOMLAB/LGO are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations. These global solvers outperform local solvers even for convex problems. Finally, TOMLAB/OQNLP, NEWUOA, and TOMLAB/MULTIMIN show superior performance in terms of refining a near-optimal solution.

References

  1. Aarts E.H.L., van Laarhoven P.J.M.: Statistical cooling: a general approach to combinatorial optimization problems. Phillips J. Res. 40, 193–226 (1985)

    Google Scholar 

  2. Abramson, M.A.: Pattern Search Algorithms for Mixed Variable General Constrained Optimization Problems. PhD thesis, Department of Computational and Applied Mathematics, Rice University, Houston (2002, Aug)

  3. Abramson, M.A.: NOMADm Version 4.5 User’s Guide. Air Force Institute of Technology, Wright-Patterson AFB, OH (2007)

  4. Abramson M.A., Asaki T.J., Dennis J.E. Jr., O’Reilly K.R., Pingel R.L.: Quantitative object reconstruction via Abel-based X-ray tomography and mixed variable optimization. SIAM J. Imaging Sci. 1, 322–342 (2008)

    Article  Google Scholar 

  5. Abramson M.A., Audet C.: Convergence of mesh adaptive direct search to second-order stationary points. SIAM J. Optim. 17, 606–609 (2006)

    Article  Google Scholar 

  6. Abramson, M.A., Audet, C., Couture, G., Dennis, J.E. Jr., LeDigabel, S.: The Nomad project. http://www.gerad.ca/nomad/

  7. Abramson M.A., Audet C., Dennis J.E. Jr: Filter pattern search algorithms for mixed variable constrained optimization problems. Pac. J. Optim. 3, 477–500 (2007)

    Google Scholar 

  8. Abramson M.A., Audet C., Dennis J.E. Jr, Le Digabel S.: OrthoMADS: a deterministic MADS instance with orthogonal directions. SIAM J. Optim. 20, 948–966 (2009)

    Article  Google Scholar 

  9. Audet C.: Convergence results for generalized pattern search algorithms are tight. Optim. Eng. 5, 101–122 (2004)

    Article  Google Scholar 

  10. Audet C., Béchard V., Chaouki J.: Spent potliner treatment process optimization using a MADS algorithm. Optim. Eng. 9, 143–160 (2008)

    Article  Google Scholar 

  11. Audet C., Dennis J.E. Jr: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17, 188–217 (2006)

    Article  Google Scholar 

  12. Audet C., Dennis J.E. Jr: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20, 445–472 (2009)

    Article  Google Scholar 

  13. Awasthi, S.: Molecular Docking by Derivative-Free Optimization Solver. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2008)

  14. Barros P.A. Jr, Kirby M.R., Mavris D.N.: Impact of sampling techniques selection on the creation of response surface models. SAE Trans. J. Aerosp. 113, 1682–1693 (2004)

    Google Scholar 

  15. Bartholomew-Biggs M.C., Parkhurst S.C., Wilson S.P.: Using DIRECT to solve an aircraft routing problem. Comput. Optim. Appl. 21, 311–323 (2002)

    Article  Google Scholar 

  16. Barton, R.R.: Metamodeling: A state of the art review. In: Proceedings of the 1994 Winter Simulation Conference, pp. 237–244 (1994)

  17. Bélisle C.J., Romeijn H.E., Smith R.L.: Hit-and-run algorithms for generating multivariate distributions. Math. Oper. Res. 18, 255–266 (1993)

    Article  Google Scholar 

  18. Bethke, J.D.: Genetic Algorithms as Function Optimizers. PhD thesis, Department of Computer and Communication Sciences, University of Michigan, Ann Arbor (1980)

  19. Björkman M., Holmström K.: Global optimization of costly nonconvex functions using radial basis functions. Optim. Eng. 1, 373–397 (2000)

    Article  Google Scholar 

  20. Boender C.G.E., Rinnooy Kan A.H.G., Timmer G.T.: A stochastic method for global optimization. Math. Program. 22, 125–140 (1982)

    Article  Google Scholar 

  21. Boneh, A., Golan, A.: Constraints’ redundancy and feasible region boundedness by random feasible point generator (RFPG). In: 3rd European Congress on Operations Research (EURO III), Amsterdam (1979)

  22. Booker, A.J., Dennis, J.E., Jr., Frank, P.D., Serafini, D.B., Torczon, V.J., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. In: ICASE Report, pp. 1–24 (1998)

  23. Booker A.J., Dennis J.E. Jr, Frank P.D., Serafini D.B., Torczon V.J., Trosset M.W.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Optim. 17, 1–13 (1999)

    Article  Google Scholar 

  24. Booker A.J., Meckesheimer M., Torng T.: Reliability based design optimization using design explorer. Optim. Eng. 5, 179–205 (2004)

    Article  Google Scholar 

  25. Brent R.P.: Algorithms for Minimization without Derivatives. Prentice-Hall, Englewood Cliffs (1973)

    Google Scholar 

  26. Chang, K.-F.: Modeling and Optimization of Polymerase Chain Reaction Using Derivative-Free Optimization. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)

  27. Chiang T., Chow Y.: A limit theorem for a class of inhomogeneous Markov processes. Ann. Probab. 17, 1483–1502 (1989)

    Article  Google Scholar 

  28. COIN-OR Project. Derivative Free Optimization. http://projects.coin-or.org/Dfo

  29. COIN-OR Project. IPOPT 2.3.x A software package for large-scale nonlinear optimization. http://www.coin-or.org/Ipopt/ipopt-fortran.html

  30. Conn A.R., Gould N., Lescrenier M., Toint Ph.L.: Performance of a multifrontal scheme for partially separable optimization. In: Gomez, S., Hennart, J.-P. (eds) Advances in Optimization and Numerical Analysis, pp. 79–96. Kluwer, Dordrecht (1994)

    Chapter  Google Scholar 

  31. Conn A.R., Scheinberg K., Toint P.L.: On the convergence of derivative-free methods for unconstrained optimization. In: Buhmann, M.D., Iserles, A. (eds) Approximation Theory and Optimization, Tribute to M. J. D. Powell, pp. 83–108. Cambridge University Press, Cambridge (1996)

    Google Scholar 

  32. Conn A.R., Scheinberg K., Toint P.L.: Recent progress in unconstrained nonlinear optimization without derivatives. Math. Program. 79, 397–414 (1997)

    Google Scholar 

  33. Conn, A.R., Scheinberg, K., Toint, P.L.: A derivative free optimization algorithm in practice. In: Proceedings of AIAA St Louis Conference, pp. 1–11 (1998)

  34. Conn A.R., Scheinberg K., Vicente L.N.: Global convergence of general derivative-free trust-region algorithms to first and second order critical points. SIAM J. Optim. 20, 387–415 (2009)

    Article  Google Scholar 

  35. Conn A.R., Scheinberg K., Vicente L.N.: Introduction to derivative-free optimization. SIAM, Philadelphia (2009)

    Book  Google Scholar 

  36. Cox, D.D., John, S.: SDO: A statistical method for global optimization. In: Multidisciplinary Design Optimization (Hampton, VA, 1995), pp. 315–329. SIAM, Philadelphia (1997)

  37. Csendes T., Pál L., Sendín J.O.H., Banga J.R.: The GLOBAL optimization method revisited. Optim. Lett. 2, 445–454 (2008)

    Article  Google Scholar 

  38. Custódio A.L., Dennis J.E. Jr, Vicente L.N.: Using simplex gradients of nonsmooth functions in direct search methods. IMA J. Numer. Anal. 28, 770–784 (2008)

    Article  Google Scholar 

  39. Custódio, A.L., Rocha, H., Vicente, L.N.: Incorporating minimum Frobenius norm models in direct search. Comput. Optim. Appl. (to appear)

  40. Custódio, A.L., Vicente, L.N.: Using sampling and simplex derivatives in pattern search methods. SIAM J. Optim. 18, 537–555 (2007)

    Google Scholar 

  41. Custódio, A.L., Vicente, L.N.: SID-PSM: A Pattern Search Method Guided by Simplex Derivatives for Use in Derivative-Free Optimization. Departamento de Matemática, Universidade de Coimbra, Coimbra (2008)

  42. Deming S.N., Parker L.R. Jr, Denton M.B.: A review of simplex optimization in analytical chemistry. Crit. Rev. Anal. Chem. 7, 187–202 (1974)

    Article  Google Scholar 

  43. Desai, R.: A Comparison of Algorithms for Optimizing the Omega Function. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2010)

  44. Eberhart, R.,Kennedy, J.: A new optimizer using particle swarm theory. In: Proceedings of the 6th International Symposium on Micro Machine and Human Science, Nagoya, pp. 39–43 (1995)

  45. Eldred, M.S., Adams, B.M., Gay, D.M., Swiler, L.P., Haskell, K., Bohnhoff, W.J., Eddy, J.P., Hart, W.E., Watson, J-P, Hough, P.D., Kolda, T.G., Williams, P.J., Martinez-Canales, M.L., DAKOTA, A.: Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 4.2 User’s Manual. Sandia National Laboratories, Albuquerque (2008)

  46. Fan S.S., Zahara E.: A hybrid simplex search and particle swarm optimization for unconstrained optimization. Eur. J. Oper. Res. 181, 527–548 (2007)

    Article  Google Scholar 

  47. Finkel D.E., Kelley C.T.: Additive scaling and the DIRECT algorithm. J. Glob. Optim. 36, 597–608 (2006)

    Article  Google Scholar 

  48. Fowler K.R., Reese J.P., Kees C.E., Dennis J.E. Jr, Kelley C.T., Miller C.T., Audet C., Booker A.J., Couture G., Darwin R.W., Farthing M.W., Finkel D.E., Gablonsky J.M., Gray G., Kolda T.G.: A comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems. Adv. Water Resour. 31, 743–757 (2008)

    Article  Google Scholar 

  49. Gablonsky, J.M.: Modifications of the DIRECT Algorithm. PhD thesis, Department of Mathematics, North Carolina State University, Raleigh (2001)

  50. Gilmore P., Kelley C.T.: An implicit filtering algorithm for optimization of functions with many local minima. SIAM J. Optim. 5, 269–285 (1995)

    Article  Google Scholar 

  51. GLOBAL Library. http://www.gamsworld.org/global/globallib.htm

  52. Gray G., Kolda T., Sale K., Young M.: Optimizing an empirical scoring function for transmembrane protein structure determination. INFORMS J. Comput. 16, 406–418 (2004)

    Article  Google Scholar 

  53. Gutmann H.-M.: A radial basis function method for global optimization. J. Glob. Optim. 19, 201–227 (2001)

    Article  Google Scholar 

  54. Han J., Kokkolaras M., Papalambros P.Y.: Optimal design of hybrid fuel cell vehicles. J. Fuel Cell Sci. Technol. 5, 041014 (2008)

    Article  Google Scholar 

  55. Hansen, N.: The CMA Evolution Strategy: A tutorial. http://www.lri.fr/hansen/cmaesintro.html

  56. Hansen N.: The CMA evolution strategy: a comparing review. In: Lozano, J.A., Larranaga, P., Inza, I., Bengoetxea, E. (eds) Towards a New Evolutionary Computation. Advances on Estimation of Distribution Algorithms, pp. 75–102. Springer, Berlin (2006)

    Chapter  Google Scholar 

  57. Hayes R.E., Bertrand F.H., Audet C., Kolaczkowski S.T.: Catalytic combustion kinetics: using a direct search algorithm to evaluate kinetic parameters from light-off curves. Can. J. Chem. Eng. 81, 1192–1199 (2003)

    Article  Google Scholar 

  58. Holland J.H.: Adaptation in Natural and Artificial Systems. The University of Michigan Press, Ann Arbor (1975)

    Google Scholar 

  59. Holmström, K.: Private Communication (2009)

  60. Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB 7. Tomlab Optimization. http://tomopt.com

  61. Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/CGO. Tomlab Optimization (2007). http://tomopt.com

  62. Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/OQNLP. Tomlab Optimization (2007). http://tomopt.com

  63. Holmström K., Quttineh N.-H., Edvall M.M.: An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer constrained global optimization. Optim. Eng. 9, 311–339 (2008)

    Article  Google Scholar 

  64. Hooke R., Jeeves T.A.: Direct search solution of numerical and statistical problems. J. Assoc. Comput. Mach. 8, 212–219 (1961)

    Article  Google Scholar 

  65. Huyer W., Neumaier A.: Global optimization by multilevel coordinate search. J. Glob. Optim. 14, 331–355 (1999)

    Article  Google Scholar 

  66. Huyer W., Neumaier A.: SNOBFIT—Stable noisy optimization by branch and fit. ACM Trans. Math. Softw. 35, 1–25 (2008)

    Article  Google Scholar 

  67. Hvattum L.M., Glover F.: Finding local optima of high-dimensional functions using direct search methods. Eur. J. Oper. Res. 195, 31–45 (2009)

    Article  Google Scholar 

  68. Ingber, L.: Adaptive Simulated Annealing (ASA). http://www.ingber.com/#ASA

  69. Järvi, T.: A Random Search Optimizer with an Application to a Max–Min Problem. Technical report, Pulications of the Institute for Applied Mathematics, University of Turku (1973)

  70. Jones D.R.: A taxonomy of global optimization methods based on response surfaces. J. Glob. Optim. 21, 345–383 (2001)

    Article  Google Scholar 

  71. Jones, D.R.: The DIRECT global optimization algorithm. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization, vol. 1, pp. 431–440. Kluwer, Boston (2001)

  72. Jones D.R., Perttunen C.D., Stuckman B.E.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl. 79, 157–181 (1993)

    Article  Google Scholar 

  73. Jones D.R., Schonlau M., Welch W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13, 455–492 (1998)

    Article  Google Scholar 

  74. Kelley, C.T.: Users Guide for IMFIL version 1.0. http://www4.ncsu.edu/ctk/imfil.html

  75. Kelley C.T.: Detection and remediation of stagnation in the Nelder–Mead algorithm using a sufficient decrease condition. SIAM J. Optim. 10, 43–55 (1999)

    Article  Google Scholar 

  76. Kelley C.T.: Iterative Methods for Optimization. SIAM, Philadelphia (1999)

    Book  Google Scholar 

  77. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks, Piscataway, pp. 1942–1948 (1995)

  78. Kirkpatrick S., Gelatt C.D., Vecchi M.P.: Optimization by simulated annealing. Science 220, 671–680 (1983)

    Article  Google Scholar 

  79. Kolda T.G., Lewis R.M., Torczon V.J.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45, 385–482 (2003)

    Article  Google Scholar 

  80. Kolda T.G., Torczon V.J.: On the convergence of asynchronous parallel pattern search. SIAM J. Optim. 14, 939–964 (2004)

    Article  Google Scholar 

  81. Lagarias J.C., Reeds J.A., Wright M.H., Wright P.E.: Convergence properties of the Nelder–Mead simplex method in low dimensions. SIAM J. Optim. 9, 112–147 (1998)

    Article  Google Scholar 

  82. LeDigabel, S.: NOMAD User Guide Version 3.3. Technical report, Les Cahiers du GERAD (2009)

  83. Lewis R.M., Torczon V.J.: Pattern search algorithms for bound constrained minimization. SIAM J. Optim. 9, 1082–1099 (1999)

    Article  Google Scholar 

  84. Lewis R.M., Torczon V.J.: Pattern search algorithms for linearly constrained minimization. SIAM J. Optim. 10, 917–941 (2000)

    Article  Google Scholar 

  85. Lewis R.M., Torczon V.J.: A globally convergent augmented lagrangian pattern search algorithm for optimization with general constraints and simple bounds. SIAM J. Optim. 12, 1075–1089 (2002)

    Article  Google Scholar 

  86. Liepins G.E., Hilliard M.R.: Genetic algorithms: foundations and applications. Ann. Oper. Res. 21, 31–58 (1989)

    Article  Google Scholar 

  87. Lin Y., Schrage L.: The global solver in the LINDO API. Optim. Methods Softw. 24, 657–668 (2009)

    Article  Google Scholar 

  88. Lucidi S., Sciandrone M.: On the global convergence of derivative-free methods for unconstrained minimization. SIAM J. Optim. 13, 97–116 (2002)

    Article  Google Scholar 

  89. Lukšan, L., Vlček, J.: Test Problems for Nonsmooth Unconstrained and Linearly Constrained Optimization. Technical report, Institute of Computer Science, Academy of Sciences of the Czech Republic (2000). http://www3.cs.cas.cz/ics/reports/v798-00.ps

  90. Marsden A.L., Feinstein J.A., Taylor C.A.: A computational framework for derivative-free optimization of cardiovascular geometries. Comput. Methods Appl. Mech. Eng. 197, 1890–1905 (2008)

    Article  Google Scholar 

  91. Marsden A.L., Wang M., Dennis J.E. Jr, Moin P.: Optimal aeroacustic shape design using the surrogate management framework. Optim. Eng. 5, 235–262 (2004)

    Article  Google Scholar 

  92. Marsden A.L., Wang M., Dennis J.E. Jr, Moin P.: Trailing-edge noise reduction using derivative-free optimization and large-eddy simulation. J. Fluid Mech. 5, 235–262 (2007)

    Google Scholar 

  93. Matheron G.: Principles of geostatistics. Econ. Geol. 58, 1246–1266 (1967)

    Article  Google Scholar 

  94. McKinnon K.I.M.: Convergence of the Nelder–Mead simplex method to a nonstationary point. SIAM J. Optim. 9, 148–158 (1998)

    Article  Google Scholar 

  95. Metropolis N., Rosenbluth A.W., Rosenbluth M.N., Teller A.H., Teller E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–1092 (1953)

    Article  Google Scholar 

  96. Mongeau M., Karsenty H., Rouzé V., Hiriart-Urruty J.B.: Comparison of public-domain software for black box global optimization. Optim. Methods Softw. 13, 203–226 (2000)

    Article  Google Scholar 

  97. Moré, J., Wild, S.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20, 172–191 (2009)

    Google Scholar 

  98. Mugunthan P., Shoemaker C.A., Regis R.G.: Comparison of function approximation, heuristic, and derivative-based methods for automatic calibration of computationally expensive groundwater bioremediation models. Water Resour. Res. 41, W11427 (2005)

    Article  Google Scholar 

  99. Nelder J.A., Mead R.: A simplex method for function minimization. Comput. J. 7, 308–313 (1965)

    Article  Google Scholar 

  100. Nesterov, Y.: Gradient methods for minimizing composite objective function. CORE Discussion Paper 2007/76 (2007)

  101. Neumaier, A.: MCS: Global Optimization by Multilevel Coordinate Search. http://www.mat.univie.ac.at/neum/software/mcs/

  102. Neumaier A., Shcherbina O., Huyer W., Vinkó T.: A comparison of complete global optimization solvers. Math. Program. 103, 335–356 (2005)

    Article  Google Scholar 

  103. Oeuvray, R.: Trust-Region Methods Based on Radial Basis Functions with Application to Biomedical Imaging. PhD thesis, Institute of Mathematics, Swiss Federal Institute of Technology, Lausanne (2005, March)

  104. Orosz J.E., Jacobson S.H.: Finite-time performance analysis of static simulated annealing algorithms. Comput. Optim. Appl. 21, 21–53 (2002)

    Article  Google Scholar 

  105. Pintér, J.: Homepage of Pintér Consulting Services. http://www.pinterconsulting.com/

  106. Pintér J.D.: Global Optimization in Action: Continuous and Lipschitz Optimization. Algorithms, Implementations and Applications. Nonconvex Optimization and its Applications. Kluwer, Dordrecht (1995)

    Google Scholar 

  107. Pintér, J.D., Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/LGO. Tomlab Optimization (2006). http://tomopt.com

  108. Plantenga, T.D.: HOPSPACK 2.0 User Manual. Technical Report SAND2009-6265, Sandia National Laboratories, Albuquerque (2009)

  109. Powell M.J.D.: A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Gomez, S., Hennart, J-P. (eds) (eds.) Advances in Optimization and Numerical Analysis, pp. 51–67. Kluwer, Dordrecht (1994)

    Chapter  Google Scholar 

  110. Powell, M.J.D.: A direct search optimization method that models the objective by quadratic interpolation. In: Presentation at the 5th Stockholm Optimization Days (1994)

  111. Powell, M.J.D.: Recent Research at Cambridge on Radial Basis Functions. Technical report, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1998)

  112. Powell M.J.D.: UOBYQA: unconstrained optimization by quadratic approximation. Math. Program. 92, 555–582 (2002)

    Article  Google Scholar 

  113. Powell M.J.D.: The NEWUOA software for unconstrained optimization without derivatives. In: Di Pillo, G., Roma, M. (eds) Large-Scale Nonlinear Optimization, pp. 255–297. Springer, New York (2006)

    Chapter  Google Scholar 

  114. Powell M.J.D.: Developments of NEWUOA for minimization without derivatives. IMA J. Numer. Anal. 28, 649–664 (2008)

    Article  Google Scholar 

  115. Powell, M.J.D.: The BOBYQA Algorithm for Bound Constrained Optimization Without Derivatives. Technical report, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (2009)

  116. Princeton Library. http://www.gamsworld.org/performance/princetonlib/princetonlib.htm

  117. Regis R.G., Shoemaker C.A.: Constrained global optimization of expensive black box functions using radial basis functions. J. Glob. Optim. 31, 153–171 (2005)

    Article  Google Scholar 

  118. Regis R.G., Shoemaker C.A.: Improved strategies for radial basis function methods for global optimization. J. Glob. Optim. 37, 113–135 (2007)

    Article  Google Scholar 

  119. Richtarik, P.: Improved algorithms for convex minimization in relative scale. SIAM J. Optim. (2010, to appear). http://www.optimization-online.org/DB_FILE/2009/02/2226.pdf

  120. Rios, L.M.: Algorithms for Derivative-Free Optimization. PhD thesis, Department of Industrial and Enterprise Systems Engineering, University of Illinois, Urbana (2009, May)

  121. Romeo F., Sangiovanni-Vincentelli A.: A theoretical framework for simulated annealing. Algorithmica 6, 302–345 (1991)

    Article  Google Scholar 

  122. Sacks J., Welch W.J., Mitchell T.J., Wynn H.P.: Design and analysis of computer experiments. Stat. Sci. 4, 409–423 (1989)

    Article  Google Scholar 

  123. Sahinidis, N.V., Tawarmalani, M.: BARON 7.5: Global Optimization of Mixed-Integer Nonlinear Programs, User’s Manual (2005)

  124. Sandia National Laboratories: The Coliny Project. https://software.sandia.gov/trac/acro/wiki/Overview/Projects

  125. Scheinberg, K.: Manual for Fortran Software Package DFO v2.0 (2003)

  126. Schonlau, M.: Computer Experiments and Global Optimization. PhD thesis, Department of Statistics, University of Waterloo, Waterloo (1997)

  127. Serafini, D.B.: A Framework for Managing Models in Nonlinear Optimization of Computationally Expensive Functions. PhD thesis, Department of Computational and Applied Mathematics, Rice University, Houston (1998, Nov)

  128. Shah, S.B., Sahinidis, N.V.: SAS-Pro: Simultaneous residue assignment and structure superposition for protein structure alignment. PLoS ONE 7(5), e37493 (2012)

  129. Shubert B.O.: A sequential method seeking the global maximum of a function. SIAM J. Numer. Anal. 9, 379–388 (1972)

    Article  Google Scholar 

  130. Smith R.L.: Efficient Monte Carlo procedures for generating points uniformly distributed over bounded regions. Oper. Res. 32, 1296–1308 (1984)

    Article  Google Scholar 

  131. Søndergaard, J.: Optimization Using Surrogate Models—by the Space Mapping Technique. PhD thesis, Technical University of Denmark, Department of Mathematical Modelling, Lingby (2003)

  132. Spendley W., Hext G.R., Himsworth F.R.: Sequential application for simplex designs in optimisation and evolutionary operation. Technometrics 4, 441–461 (1962)

    Article  Google Scholar 

  133. Tawarmalani M., Sahinidis N.V.: A polyhedral branch-and-cut approach to global optimization. Math. Program. 103, 225–249 (2005)

    Article  Google Scholar 

  134. Torczon V.J.: On the convergence of multidirectional search algorithms. SIAM J. Optim. 1, 123–145 (1991)

    Article  Google Scholar 

  135. Torczon V.J.: On the convergence of pattern search algorithms. SIAM J. Optim. 7, 1–25 (1997)

    Article  Google Scholar 

  136. Tseng P.: Fortified-descent simplicial search method: a general approach. SIAM J. Optim. 10, 269–288 (1999)

    Article  Google Scholar 

  137. Vaz, A.I.F.: PSwarm Home Page. http://www.norg.uminho.pt/aivaz/pswarm/

  138. Vaz A.I.F., Vicente L.N.: A particle swarm pattern search method for bound constrained global optimization. J. Glob. Optim. 39, 197–219 (2007)

    Article  Google Scholar 

  139. Wang, H.: Application of Derivative-Free Algorithms in Powder Diffraction. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)

  140. Wild S.M., Regis R.G., Shoemaker C.A.: ORBIT: Optimization by radial basis function interpolation in trust-regions. SIAM J. Sci. Comput. 30, 3197–3219 (2008)

    Article  Google Scholar 

  141. Winfield, D.: Function and Functional Optimization by Interpolation in Data Tables. PhD thesis, Harvard University, Cambridge (1969)

  142. Winslow, T.A., Trew, R.J., Gilmore, P., Kelley, C.T.: Simulated performance optimization of gaas mesfet amplifiers. In: IEEE/Cornell Conference on Advanced Concepts in High Speed Semiconductor Devices and Circuits, Piscataway, pp. 393–402 (1991)

  143. Zhao Z., Meza J.C., Van Hove M.: Using pattern search methods for surface structure determination of nanomaterials. J. Phys. Condens. Matter 18, 8693–8706 (2006)

    Article  Google Scholar 

  144. Zheng, Y.: Pairs Trading and Portfolio Optimization. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)

Download references

Author information

Authors and Affiliations

  1. Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Luis Miguel Rios & Nikolaos V. Sahinidis

Authors
  1. Luis Miguel Rios
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikolaos V. Sahinidis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nikolaos V. Sahinidis.

Electronic Supplementary Material

The Below is the Electronic Supplementary Material.

ESM 1 (PDF 159 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Rios, L.M., Sahinidis, N.V. Derivative-free optimization: a review of algorithms and comparison of software implementations. J Glob Optim 56, 1247–1293 (2013). https://doi.org/10.1007/s10898-012-9951-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-012-9951-y

Keywords

  • Derivative-free algorithms
  • Direct search methods
  • Surrogate models