Download PDF

Journal of Global Optimization

, Volume 56, Issue 3, pp 1247–1293 | Cite as

Derivative-free optimization: a review of algorithms and comparison of software implementations

Open Access
Article
First Online:
Received:
Accepted:

Abstract

This paper addresses the solution of bound-constrained optimization problems using algorithms that require only the availability of objective function values but no derivative information. We refer to these algorithms as derivative-free algorithms. Fueled by a growing number of applications in science and engineering, the development of derivative-free optimization algorithms has long been studied, and it has found renewed interest in recent time. Along with many derivative-free algorithms, many software implementations have also appeared. The paper presents a review of derivative-free algorithms, followed by a systematic comparison of 22 related implementations using a test set of 502 problems. The test bed includes convex and nonconvex problems, smooth as well as nonsmooth problems. The algorithms were tested under the same conditions and ranked under several criteria, including their ability to find near-global solutions for nonconvex problems, improve a given starting point, and refine a near-optimal solution. A total of 112,448 problem instances were solved. We find that the ability of all these solvers to obtain good solutions diminishes with increasing problem size. For the problems used in this study, TOMLAB/MULTIMIN, TOMLAB/GLCCLUSTER, MCS and TOMLAB/LGO are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations. These global solvers outperform local solvers even for convex problems. Finally, TOMLAB/OQNLP, NEWUOA, and TOMLAB/MULTIMIN show superior performance in terms of refining a near-optimal solution.

Keywords

Derivative-free algorithms Direct search methods Surrogate models 
Download to read the full article text

Supplementary material

10898_2012_9951_MOESM1_ESM.pdf (159 kb)
ESM 1 (PDF 159 kb)

References

  1. 1.
    Aarts E.H.L., van Laarhoven P.J.M.: Statistical cooling: a general approach to combinatorial optimization problems. Phillips J. Res. 40, 193–226 (1985)Google Scholar
  2. 2.
    Abramson, M.A.: Pattern Search Algorithms for Mixed Variable General Constrained Optimization Problems. PhD thesis, Department of Computational and Applied Mathematics, Rice University, Houston (2002, Aug)Google Scholar
  3. 3.
    Abramson, M.A.: NOMADm Version 4.5 User’s Guide. Air Force Institute of Technology, Wright-Patterson AFB, OH (2007)Google Scholar
  4. 4.
    Abramson M.A., Asaki T.J., Dennis J.E. Jr., O’Reilly K.R., Pingel R.L.: Quantitative object reconstruction via Abel-based X-ray tomography and mixed variable optimization. SIAM J. Imaging Sci. 1, 322–342 (2008)CrossRefGoogle Scholar
  5. 5.
    Abramson M.A., Audet C.: Convergence of mesh adaptive direct search to second-order stationary points. SIAM J. Optim. 17, 606–609 (2006)CrossRefGoogle Scholar
  6. 6.
    Abramson, M.A., Audet, C., Couture, G., Dennis, J.E. Jr., LeDigabel, S.: The Nomad project. http://www.gerad.ca/nomad/
  7. 7.
    Abramson M.A., Audet C., Dennis J.E. Jr: Filter pattern search algorithms for mixed variable constrained optimization problems. Pac. J. Optim. 3, 477–500 (2007)Google Scholar
  8. 8.
    Abramson M.A., Audet C., Dennis J.E. Jr, Le Digabel S.: OrthoMADS: a deterministic MADS instance with orthogonal directions. SIAM J. Optim. 20, 948–966 (2009)CrossRefGoogle Scholar
  9. 9.
    Audet C.: Convergence results for generalized pattern search algorithms are tight. Optim. Eng. 5, 101–122 (2004)CrossRefGoogle Scholar
  10. 10.
    Audet C., Béchard V., Chaouki J.: Spent potliner treatment process optimization using a MADS algorithm. Optim. Eng. 9, 143–160 (2008)CrossRefGoogle Scholar
  11. 11.
    Audet C., Dennis J.E. Jr: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17, 188–217 (2006)CrossRefGoogle Scholar
  12. 12.
    Audet C., Dennis J.E. Jr: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20, 445–472 (2009)CrossRefGoogle Scholar
  13. 13.
    Awasthi, S.: Molecular Docking by Derivative-Free Optimization Solver. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2008)Google Scholar
  14. 14.
    Barros P.A. Jr, Kirby M.R., Mavris D.N.: Impact of sampling techniques selection on the creation of response surface models. SAE Trans. J. Aerosp. 113, 1682–1693 (2004)Google Scholar
  15. 15.
    Bartholomew-Biggs M.C., Parkhurst S.C., Wilson S.P.: Using DIRECT to solve an aircraft routing problem. Comput. Optim. Appl. 21, 311–323 (2002)CrossRefGoogle Scholar
  16. 16.
    Barton, R.R.: Metamodeling: A state of the art review. In: Proceedings of the 1994 Winter Simulation Conference, pp. 237–244 (1994)Google Scholar
  17. 17.
    Bélisle C.J., Romeijn H.E., Smith R.L.: Hit-and-run algorithms for generating multivariate distributions. Math. Oper. Res. 18, 255–266 (1993)CrossRefGoogle Scholar
  18. 18.
    Bethke, J.D.: Genetic Algorithms as Function Optimizers. PhD thesis, Department of Computer and Communication Sciences, University of Michigan, Ann Arbor (1980)Google Scholar
  19. 19.
    Björkman M., Holmström K.: Global optimization of costly nonconvex functions using radial basis functions. Optim. Eng. 1, 373–397 (2000)CrossRefGoogle Scholar
  20. 20.
    Boender C.G.E., Rinnooy Kan A.H.G., Timmer G.T.: A stochastic method for global optimization. Math. Program. 22, 125–140 (1982)CrossRefGoogle Scholar
  21. 21.
    Boneh, A., Golan, A.: Constraints’ redundancy and feasible region boundedness by random feasible point generator (RFPG). In: 3rd European Congress on Operations Research (EURO III), Amsterdam (1979)Google Scholar
  22. 22.
    Booker, A.J., Dennis, J.E., Jr., Frank, P.D., Serafini, D.B., Torczon, V.J., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. In: ICASE Report, pp. 1–24 (1998)Google Scholar
  23. 23.
    Booker A.J., Dennis J.E. Jr, Frank P.D., Serafini D.B., Torczon V.J., Trosset M.W.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Optim. 17, 1–13 (1999)CrossRefGoogle Scholar
  24. 24.
    Booker A.J., Meckesheimer M., Torng T.: Reliability based design optimization using design explorer. Optim. Eng. 5, 179–205 (2004)CrossRefGoogle Scholar
  25. 25.
    Brent R.P.: Algorithms for Minimization without Derivatives. Prentice-Hall, Englewood Cliffs (1973)Google Scholar
  26. 26.
    Chang, K.-F.: Modeling and Optimization of Polymerase Chain Reaction Using Derivative-Free Optimization. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)Google Scholar
  27. 27.
    Chiang T., Chow Y.: A limit theorem for a class of inhomogeneous Markov processes. Ann. Probab. 17, 1483–1502 (1989)CrossRefGoogle Scholar
  28. 28.
    COIN-OR Project. Derivative Free Optimization. http://projects.coin-or.org/Dfo
  29. 29.
    COIN-OR Project. IPOPT 2.3.x A software package for large-scale nonlinear optimization. http://www.coin-or.org/Ipopt/ipopt-fortran.html
  30. 30.
    Conn A.R., Gould N., Lescrenier M., Toint Ph.L.: Performance of a multifrontal scheme for partially separable optimization. In: Gomez, S., Hennart, J.-P. (eds) Advances in Optimization and Numerical Analysis, pp. 79–96. Kluwer, Dordrecht (1994)CrossRefGoogle Scholar
  31. 31.
    Conn A.R., Scheinberg K., Toint P.L.: On the convergence of derivative-free methods for unconstrained optimization. In: Buhmann, M.D., Iserles, A. (eds) Approximation Theory and Optimization, Tribute to M. J. D. Powell, pp. 83–108. Cambridge University Press, Cambridge (1996)Google Scholar
  32. 32.
    Conn A.R., Scheinberg K., Toint P.L.: Recent progress in unconstrained nonlinear optimization without derivatives. Math. Program. 79, 397–414 (1997)Google Scholar
  33. 33.
    Conn, A.R., Scheinberg, K., Toint, P.L.: A derivative free optimization algorithm in practice. In: Proceedings of AIAA St Louis Conference, pp. 1–11 (1998)Google Scholar
  34. 34.
    Conn A.R., Scheinberg K., Vicente L.N.: Global convergence of general derivative-free trust-region algorithms to first and second order critical points. SIAM J. Optim. 20, 387–415 (2009)CrossRefGoogle Scholar
  35. 35.
    Conn A.R., Scheinberg K., Vicente L.N.: Introduction to derivative-free optimization. SIAM, Philadelphia (2009)CrossRefGoogle Scholar
  36. 36.
    Cox, D.D., John, S.: SDO: A statistical method for global optimization. In: Multidisciplinary Design Optimization (Hampton, VA, 1995), pp. 315–329. SIAM, Philadelphia (1997)Google Scholar
  37. 37.
    Csendes T., Pál L., Sendín J.O.H., Banga J.R.: The GLOBAL optimization method revisited. Optim. Lett. 2, 445–454 (2008)CrossRefGoogle Scholar
  38. 38.
    Custódio A.L., Dennis J.E. Jr, Vicente L.N.: Using simplex gradients of nonsmooth functions in direct search methods. IMA J. Numer. Anal. 28, 770–784 (2008)CrossRefGoogle Scholar
  39. 39.
    Custódio, A.L., Rocha, H., Vicente, L.N.: Incorporating minimum Frobenius norm models in direct search. Comput. Optim. Appl. (to appear)Google Scholar
  40. 40.
    Custódio, A.L., Vicente, L.N.: Using sampling and simplex derivatives in pattern search methods. SIAM J. Optim. 18, 537–555 (2007)Google Scholar
  41. 41.
    Custódio, A.L., Vicente, L.N.: SID-PSM: A Pattern Search Method Guided by Simplex Derivatives for Use in Derivative-Free Optimization. Departamento de Matemática, Universidade de Coimbra, Coimbra (2008)Google Scholar
  42. 42.
    Deming S.N., Parker L.R. Jr, Denton M.B.: A review of simplex optimization in analytical chemistry. Crit. Rev. Anal. Chem. 7, 187–202 (1974)CrossRefGoogle Scholar
  43. 43.
    Desai, R.: A Comparison of Algorithms for Optimizing the Omega Function. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2010)Google Scholar
  44. 44.
    Eberhart, R.,Kennedy, J.: A new optimizer using particle swarm theory. In: Proceedings of the 6th International Symposium on Micro Machine and Human Science, Nagoya, pp. 39–43 (1995)Google Scholar
  45. 45.
    Eldred, M.S., Adams, B.M., Gay, D.M., Swiler, L.P., Haskell, K., Bohnhoff, W.J., Eddy, J.P., Hart, W.E., Watson, J-P, Hough, P.D., Kolda, T.G., Williams, P.J., Martinez-Canales, M.L., DAKOTA, A.: Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 4.2 User’s Manual. Sandia National Laboratories, Albuquerque (2008)Google Scholar
  46. 46.
    Fan S.S., Zahara E.: A hybrid simplex search and particle swarm optimization for unconstrained optimization. Eur. J. Oper. Res. 181, 527–548 (2007)CrossRefGoogle Scholar
  47. 47.
    Finkel D.E., Kelley C.T.: Additive scaling and the DIRECT algorithm. J. Glob. Optim. 36, 597–608 (2006)CrossRefGoogle Scholar
  48. 48.
    Fowler K.R., Reese J.P., Kees C.E., Dennis J.E. Jr, Kelley C.T., Miller C.T., Audet C., Booker A.J., Couture G., Darwin R.W., Farthing M.W., Finkel D.E., Gablonsky J.M., Gray G., Kolda T.G.: A comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems. Adv. Water Resour. 31, 743–757 (2008)CrossRefGoogle Scholar
  49. 49.
    Gablonsky, J.M.: Modifications of the DIRECT Algorithm. PhD thesis, Department of Mathematics, North Carolina State University, Raleigh (2001)Google Scholar
  50. 50.
    Gilmore P., Kelley C.T.: An implicit filtering algorithm for optimization of functions with many local minima. SIAM J. Optim. 5, 269–285 (1995)CrossRefGoogle Scholar
  51. 51.
  52. 52.
    Gray G., Kolda T., Sale K., Young M.: Optimizing an empirical scoring function for transmembrane protein structure determination. INFORMS J. Comput. 16, 406–418 (2004)CrossRefGoogle Scholar
  53. 53.
    Gutmann H.-M.: A radial basis function method for global optimization. J. Glob. Optim. 19, 201–227 (2001)CrossRefGoogle Scholar
  54. 54.
    Han J., Kokkolaras M., Papalambros P.Y.: Optimal design of hybrid fuel cell vehicles. J. Fuel Cell Sci. Technol. 5, 041014 (2008)CrossRefGoogle Scholar
  55. 55.
    Hansen, N.: The CMA Evolution Strategy: A tutorial. http://www.lri.fr/hansen/cmaesintro.html
  56. 56.
    Hansen N.: The CMA evolution strategy: a comparing review. In: Lozano, J.A., Larranaga, P., Inza, I., Bengoetxea, E. (eds) Towards a New Evolutionary Computation. Advances on Estimation of Distribution Algorithms, pp. 75–102. Springer, Berlin (2006)CrossRefGoogle Scholar
  57. 57.
    Hayes R.E., Bertrand F.H., Audet C., Kolaczkowski S.T.: Catalytic combustion kinetics: using a direct search algorithm to evaluate kinetic parameters from light-off curves. Can. J. Chem. Eng. 81, 1192–1199 (2003)CrossRefGoogle Scholar
  58. 58.
    Holland J.H.: Adaptation in Natural and Artificial Systems. The University of Michigan Press, Ann Arbor (1975)Google Scholar
  59. 59.
    Holmström, K.: Private Communication (2009)Google Scholar
  60. 60.
    Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB 7. Tomlab Optimization. http://tomopt.com
  61. 61.
    Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/CGO. Tomlab Optimization (2007). http://tomopt.com
  62. 62.
    Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/OQNLP. Tomlab Optimization (2007). http://tomopt.com
  63. 63.
    Holmström K., Quttineh N.-H., Edvall M.M.: An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer constrained global optimization. Optim. Eng. 9, 311–339 (2008)CrossRefGoogle Scholar
  64. 64.
    Hooke R., Jeeves T.A.: Direct search solution of numerical and statistical problems. J. Assoc. Comput. Mach. 8, 212–219 (1961)CrossRefGoogle Scholar
  65. 65.
    Huyer W., Neumaier A.: Global optimization by multilevel coordinate search. J. Glob. Optim. 14, 331–355 (1999)CrossRefGoogle Scholar
  66. 66.
    Huyer W., Neumaier A.: SNOBFIT—Stable noisy optimization by branch and fit. ACM Trans. Math. Softw. 35, 1–25 (2008)CrossRefGoogle Scholar
  67. 67.
    Hvattum L.M., Glover F.: Finding local optima of high-dimensional functions using direct search methods. Eur. J. Oper. Res. 195, 31–45 (2009)CrossRefGoogle Scholar
  68. 68.
    Ingber, L.: Adaptive Simulated Annealing (ASA). http://www.ingber.com/#ASA
  69. 69.
    Järvi, T.: A Random Search Optimizer with an Application to a Max–Min Problem. Technical report, Pulications of the Institute for Applied Mathematics, University of Turku (1973)Google Scholar
  70. 70.
    Jones D.R.: A taxonomy of global optimization methods based on response surfaces. J. Glob. Optim. 21, 345–383 (2001)CrossRefGoogle Scholar
  71. 71.
    Jones, D.R.: The DIRECT global optimization algorithm. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization, vol. 1, pp. 431–440. Kluwer, Boston (2001)Google Scholar
  72. 72.
    Jones D.R., Perttunen C.D., Stuckman B.E.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl. 79, 157–181 (1993)CrossRefGoogle Scholar
  73. 73.
    Jones D.R., Schonlau M., Welch W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13, 455–492 (1998)CrossRefGoogle Scholar
  74. 74.
    Kelley, C.T.: Users Guide for IMFIL version 1.0. http://www4.ncsu.edu/ctk/imfil.html
  75. 75.
    Kelley C.T.: Detection and remediation of stagnation in the Nelder–Mead algorithm using a sufficient decrease condition. SIAM J. Optim. 10, 43–55 (1999)CrossRefGoogle Scholar
  76. 76.
    Kelley C.T.: Iterative Methods for Optimization. SIAM, Philadelphia (1999)CrossRefGoogle Scholar
  77. 77.
    Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks, Piscataway, pp. 1942–1948 (1995)Google Scholar
  78. 78.
    Kirkpatrick S., Gelatt C.D., Vecchi M.P.: Optimization by simulated annealing. Science 220, 671–680 (1983)CrossRefGoogle Scholar
  79. 79.
    Kolda T.G., Lewis R.M., Torczon V.J.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45, 385–482 (2003)CrossRefGoogle Scholar
  80. 80.
    Kolda T.G., Torczon V.J.: On the convergence of asynchronous parallel pattern search. SIAM J. Optim. 14, 939–964 (2004)CrossRefGoogle Scholar
  81. 81.
    Lagarias J.C., Reeds J.A., Wright M.H., Wright P.E.: Convergence properties of the Nelder–Mead simplex method in low dimensions. SIAM J. Optim. 9, 112–147 (1998)CrossRefGoogle Scholar
  82. 82.
    LeDigabel, S.: NOMAD User Guide Version 3.3. Technical report, Les Cahiers du GERAD (2009)Google Scholar
  83. 83.
    Lewis R.M., Torczon V.J.: Pattern search algorithms for bound constrained minimization. SIAM J. Optim. 9, 1082–1099 (1999)CrossRefGoogle Scholar
  84. 84.
    Lewis R.M., Torczon V.J.: Pattern search algorithms for linearly constrained minimization. SIAM J. Optim. 10, 917–941 (2000)CrossRefGoogle Scholar
  85. 85.
    Lewis R.M., Torczon V.J.: A globally convergent augmented lagrangian pattern search algorithm for optimization with general constraints and simple bounds. SIAM J. Optim. 12, 1075–1089 (2002)CrossRefGoogle Scholar
  86. 86.
    Liepins G.E., Hilliard M.R.: Genetic algorithms: foundations and applications. Ann. Oper. Res. 21, 31–58 (1989)CrossRefGoogle Scholar
  87. 87.
    Lin Y., Schrage L.: The global solver in the LINDO API. Optim. Methods Softw. 24, 657–668 (2009)CrossRefGoogle Scholar
  88. 88.
    Lucidi S., Sciandrone M.: On the global convergence of derivative-free methods for unconstrained minimization. SIAM J. Optim. 13, 97–116 (2002)CrossRefGoogle Scholar
  89. 89.
    Lukšan, L., Vlček, J.: Test Problems for Nonsmooth Unconstrained and Linearly Constrained Optimization. Technical report, Institute of Computer Science, Academy of Sciences of the Czech Republic (2000). http://www3.cs.cas.cz/ics/reports/v798-00.ps
  90. 90.
    Marsden A.L., Feinstein J.A., Taylor C.A.: A computational framework for derivative-free optimization of cardiovascular geometries. Comput. Methods Appl. Mech. Eng. 197, 1890–1905 (2008)CrossRefGoogle Scholar
  91. 91.
    Marsden A.L., Wang M., Dennis J.E. Jr, Moin P.: Optimal aeroacustic shape design using the surrogate management framework. Optim. Eng. 5, 235–262 (2004)CrossRefGoogle Scholar
  92. 92.
    Marsden A.L., Wang M., Dennis J.E. Jr, Moin P.: Trailing-edge noise reduction using derivative-free optimization and large-eddy simulation. J. Fluid Mech. 5, 235–262 (2007)Google Scholar
  93. 93.
    Matheron G.: Principles of geostatistics. Econ. Geol. 58, 1246–1266 (1967)CrossRefGoogle Scholar
  94. 94.
    McKinnon K.I.M.: Convergence of the Nelder–Mead simplex method to a nonstationary point. SIAM J. Optim. 9, 148–158 (1998)CrossRefGoogle Scholar
  95. 95.
    Metropolis N., Rosenbluth A.W., Rosenbluth M.N., Teller A.H., Teller E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–1092 (1953)CrossRefGoogle Scholar
  96. 96.
    Mongeau M., Karsenty H., Rouzé V., Hiriart-Urruty J.B.: Comparison of public-domain software for black box global optimization. Optim. Methods Softw. 13, 203–226 (2000)CrossRefGoogle Scholar
  97. 97.
    Moré, J., Wild, S.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20, 172–191 (2009)Google Scholar
  98. 98.
    Mugunthan P., Shoemaker C.A., Regis R.G.: Comparison of function approximation, heuristic, and derivative-based methods for automatic calibration of computationally expensive groundwater bioremediation models. Water Resour. Res. 41, W11427 (2005)CrossRefGoogle Scholar
  99. 99.
    Nelder J.A., Mead R.: A simplex method for function minimization. Comput. J. 7, 308–313 (1965)CrossRefGoogle Scholar
  100. 100.
    Nesterov, Y.: Gradient methods for minimizing composite objective function. CORE Discussion Paper 2007/76 (2007)Google Scholar
  101. 101.
    Neumaier, A.: MCS: Global Optimization by Multilevel Coordinate Search. http://www.mat.univie.ac.at/neum/software/mcs/
  102. 102.
    Neumaier A., Shcherbina O., Huyer W., Vinkó T.: A comparison of complete global optimization solvers. Math. Program. 103, 335–356 (2005)CrossRefGoogle Scholar
  103. 103.
    Oeuvray, R.: Trust-Region Methods Based on Radial Basis Functions with Application to Biomedical Imaging. PhD thesis, Institute of Mathematics, Swiss Federal Institute of Technology, Lausanne (2005, March)Google Scholar
  104. 104.
    Orosz J.E., Jacobson S.H.: Finite-time performance analysis of static simulated annealing algorithms. Comput. Optim. Appl. 21, 21–53 (2002)CrossRefGoogle Scholar
  105. 105.
    Pintér, J.: Homepage of Pintér Consulting Services. http://www.pinterconsulting.com/
  106. 106.
    Pintér J.D.: Global Optimization in Action: Continuous and Lipschitz Optimization. Algorithms, Implementations and Applications. Nonconvex Optimization and its Applications. Kluwer, Dordrecht (1995)Google Scholar
  107. 107.
    Pintér, J.D., Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/LGO. Tomlab Optimization (2006). http://tomopt.com
  108. 108.
    Plantenga, T.D.: HOPSPACK 2.0 User Manual. Technical Report SAND2009-6265, Sandia National Laboratories, Albuquerque (2009)Google Scholar
  109. 109.
    Powell M.J.D.: A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Gomez, S., Hennart, J-P. (eds) (eds.) Advances in Optimization and Numerical Analysis, pp. 51–67. Kluwer, Dordrecht (1994)CrossRefGoogle Scholar
  110. 110.
    Powell, M.J.D.: A direct search optimization method that models the objective by quadratic interpolation. In: Presentation at the 5th Stockholm Optimization Days (1994)Google Scholar
  111. 111.
    Powell, M.J.D.: Recent Research at Cambridge on Radial Basis Functions. Technical report, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1998)Google Scholar
  112. 112.
    Powell M.J.D.: UOBYQA: unconstrained optimization by quadratic approximation. Math. Program. 92, 555–582 (2002)CrossRefGoogle Scholar
  113. 113.
    Powell M.J.D.: The NEWUOA software for unconstrained optimization without derivatives. In: Di Pillo, G., Roma, M. (eds) Large-Scale Nonlinear Optimization, pp. 255–297. Springer, New York (2006)CrossRefGoogle Scholar
  114. 114.
    Powell M.J.D.: Developments of NEWUOA for minimization without derivatives. IMA J. Numer. Anal. 28, 649–664 (2008)CrossRefGoogle Scholar
  115. 115.
    Powell, M.J.D.: The BOBYQA Algorithm for Bound Constrained Optimization Without Derivatives. Technical report, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (2009)Google Scholar
  116. 116.
  117. 117.
    Regis R.G., Shoemaker C.A.: Constrained global optimization of expensive black box functions using radial basis functions. J. Glob. Optim. 31, 153–171 (2005)CrossRefGoogle Scholar
  118. 118.
    Regis R.G., Shoemaker C.A.: Improved strategies for radial basis function methods for global optimization. J. Glob. Optim. 37, 113–135 (2007)CrossRefGoogle Scholar
  119. 119.
    Richtarik, P.: Improved algorithms for convex minimization in relative scale. SIAM J. Optim. (2010, to appear). http://www.optimization-online.org/DB_FILE/2009/02/2226.pdf
  120. 120.
    Rios, L.M.: Algorithms for Derivative-Free Optimization. PhD thesis, Department of Industrial and Enterprise Systems Engineering, University of Illinois, Urbana (2009, May)Google Scholar
  121. 121.
    Romeo F., Sangiovanni-Vincentelli A.: A theoretical framework for simulated annealing. Algorithmica 6, 302–345 (1991)CrossRefGoogle Scholar
  122. 122.
    Sacks J., Welch W.J., Mitchell T.J., Wynn H.P.: Design and analysis of computer experiments. Stat. Sci. 4, 409–423 (1989)CrossRefGoogle Scholar
  123. 123.
    Sahinidis, N.V., Tawarmalani, M.: BARON 7.5: Global Optimization of Mixed-Integer Nonlinear Programs, User’s Manual (2005)Google Scholar
  124. 124.
    Sandia National Laboratories: The Coliny Project. https://software.sandia.gov/trac/acro/wiki/Overview/Projects
  125. 125.
    Scheinberg, K.: Manual for Fortran Software Package DFO v2.0 (2003)Google Scholar
  126. 126.
    Schonlau, M.: Computer Experiments and Global Optimization. PhD thesis, Department of Statistics, University of Waterloo, Waterloo (1997)Google Scholar
  127. 127.
    Serafini, D.B.: A Framework for Managing Models in Nonlinear Optimization of Computationally Expensive Functions. PhD thesis, Department of Computational and Applied Mathematics, Rice University, Houston (1998, Nov)Google Scholar
  128. 128.
    Shah, S.B., Sahinidis, N.V.: SAS-Pro: Simultaneous residue assignment and structure superposition for protein structure alignment. PLoS ONE 7(5), e37493 (2012)Google Scholar
  129. 129.
    Shubert B.O.: A sequential method seeking the global maximum of a function. SIAM J. Numer. Anal. 9, 379–388 (1972)CrossRefGoogle Scholar
  130. 130.
    Smith R.L.: Efficient Monte Carlo procedures for generating points uniformly distributed over bounded regions. Oper. Res. 32, 1296–1308 (1984)CrossRefGoogle Scholar
  131. 131.
    Søndergaard, J.: Optimization Using Surrogate Models—by the Space Mapping Technique. PhD thesis, Technical University of Denmark, Department of Mathematical Modelling, Lingby (2003)Google Scholar
  132. 132.
    Spendley W., Hext G.R., Himsworth F.R.: Sequential application for simplex designs in optimisation and evolutionary operation. Technometrics 4, 441–461 (1962)CrossRefGoogle Scholar
  133. 133.
    Tawarmalani M., Sahinidis N.V.: A polyhedral branch-and-cut approach to global optimization. Math. Program. 103, 225–249 (2005)CrossRefGoogle Scholar
  134. 134.
    Torczon V.J.: On the convergence of multidirectional search algorithms. SIAM J. Optim. 1, 123–145 (1991)CrossRefGoogle Scholar
  135. 135.
    Torczon V.J.: On the convergence of pattern search algorithms. SIAM J. Optim. 7, 1–25 (1997)CrossRefGoogle Scholar
  136. 136.
    Tseng P.: Fortified-descent simplicial search method: a general approach. SIAM J. Optim. 10, 269–288 (1999)CrossRefGoogle Scholar
  137. 137.
    Vaz, A.I.F.: PSwarm Home Page. http://www.norg.uminho.pt/aivaz/pswarm/
  138. 138.
    Vaz A.I.F., Vicente L.N.: A particle swarm pattern search method for bound constrained global optimization. J. Glob. Optim. 39, 197–219 (2007)CrossRefGoogle Scholar
  139. 139.
    Wang, H.: Application of Derivative-Free Algorithms in Powder Diffraction. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)Google Scholar
  140. 140.
    Wild S.M., Regis R.G., Shoemaker C.A.: ORBIT: Optimization by radial basis function interpolation in trust-regions. SIAM J. Sci. Comput. 30, 3197–3219 (2008)CrossRefGoogle Scholar
  141. 141.
    Winfield, D.: Function and Functional Optimization by Interpolation in Data Tables. PhD thesis, Harvard University, Cambridge (1969)Google Scholar
  142. 142.
    Winslow, T.A., Trew, R.J., Gilmore, P., Kelley, C.T.: Simulated performance optimization of gaas mesfet amplifiers. In: IEEE/Cornell Conference on Advanced Concepts in High Speed Semiconductor Devices and Circuits, Piscataway, pp. 393–402 (1991)Google Scholar
  143. 143.
    Zhao Z., Meza J.C., Van Hove M.: Using pattern search methods for surface structure determination of nanomaterials. J. Phys. Condens. Matter 18, 8693–8706 (2006)CrossRefGoogle Scholar
  144. 144.
    Zheng, Y.: Pairs Trading and Portfolio Optimization. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)Google Scholar

Copyright information

© Springer Science+Business Media, LLC. 2012

Authors and Affiliations

  1. 1.Department of Chemical EngineeringCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations