Journal of Global Optimization

, Volume 59, Issue 4, pp 865–889 | Cite as

SO-I: a surrogate model algorithm for expensive nonlinear integer programming problems including global optimization applications

  • Juliane Müller
  • Christine A. Shoemaker
  • Robert Piché


This paper presents the surrogate model based algorithm SO-I for solving purely integer optimization problems that have computationally expensive black-box objective functions and that may have computationally expensive constraints. The algorithm was developed for solving global optimization problems, meaning that the relaxed optimization problems have many local optima. However, the method is also shown to perform well on many local optimization problems, and problems with linear objective functions. The performance of SO-I, a genetic algorithm, Nonsmooth Optimization by Mesh Adaptive Direct Search (NOMAD), SO-MI (Müller et al. in Comput Oper Res 40(5):1383–1400, 2013), variable neighborhood search, and a version of SO-I that only uses a local search has been compared on 17 test problems from the literature, and on eight realizations of two application problems. One application problem relates to hydropower generation, and the other one to throughput maximization. The numerical results show that SO-I finds good solutions most efficiently. Moreover, as opposed to SO-MI, SO-I is able to find feasible points by employing a first optimization phase that aims at minimizing a constraint violation function. A feasible user-supplied point is not necessary.


Integer optimization Derivative-free Computationally expensive  Surrogate model Response surface  Linear and nonlinear Nonconvex  Radial basis functions  Multimodal Global optimization 

List of symbols


Genetic algorithm


Nonsmooth Optimization by Mesh Adaptive Direct Search


Radial basis function


Standard error of means


Surrogate Optimization-Integer


Surrogate Optimization-Mixed Integer


local-Surrogate Optimization-Integer


Variable neighborhood search

\(\mathbb R \)

Real numbers

\(\mathbb Z \)

Integer numbers

\(\mathbf u \)

Discrete decision variable vector, see Eq. (1d)

\(f(\cdot )\)

Objective function, see Eq. (1a)

\(c_j(\cdot )\)

\(j\)th constraint function, \(j=1,\ldots , m\), see Eq. (1b)


Number of constraints


Problem dimension


Index for the constraints


Index for the variables


Lower and upper bounds for the \(i\)th variable, see Eq. (1c)

\(\varOmega _b\)

Box-constrained variable domain

\(\varOmega \)

Feasible variable domain

\(\mathcal S \)

Set of already evaluated points


Number of points in initial experimental design

\(q(\cdot )\)

Auxiliary function for minimizing constraint violation in phase 1, see Eq. (5)

\(f_p(\cdot )\)

Objective function value augmented with penalty term, see Eq. (6)

\(f_\text {max}\)

Objective function value of the worst feasible point found so far, see Eq. (6)


Penalty factor, see Eq. (6)

\(v(\cdot )\)

Squared constraint violation function, see Eq. (7)

\({\varvec{\chi }}_\jmath \)

\(\jmath \)th candidate point for next sample site, \(\jmath =1,\ldots , t\)


Number of already sampled points

\(s(\cdot )\)

Radial basis function interpolant

\(V(\cdot )\)

Weighted score, see Eq. (8)

\(\omega _R, \omega _D\)

Weights for response surface and distance criteria, respectively



The project has been supported by NSF CISE: AF 1116298 and CPA-CSA-T 0811729. The first author thanks the Finnish Academy of Science and Letters, Vilho, Yrjö and Kalle Väisälä Foundation for the financial support. The authors thank the anonymous reviewers for their helpful comments and suggestions.

Supplementary material

10898_2013_101_MOESM1_ESM.pdf (99 kb)
Supplementary material 1 (pdf 99 KB)


  1. 1.
    Abramson, M.A., Audet, C.: Convergence of mesh adaptive direct search to second-order stationary points. SIAM J. Optim. 17, 606–619 (2006)CrossRefGoogle Scholar
  2. 2.
    Abramson, M.A., Audet, C., Chrissis, J., Walston, J.: Mesh adaptive direct search algorithms for mixed variable optimization. Optim. Lett. 3, 35–47 (2009)CrossRefGoogle Scholar
  3. 3.
    Abramson, M.A., Audet, C., Couture, G., Dennis, J.E. Jr., Le Digabel, S.: The NOMAD project. Software available at (2011)
  4. 4.
    Abramson, M.A., Audet, C., Dennis Jr, J.E.: Filter pattern search algorithms for mixed variable constrained optimization problems. SIAM J. Optim. 11, 573–594 (2004)Google Scholar
  5. 5.
    Audet, C., Béchard, V., Le Digabel, S.: Nonsmooth optimization through mesh adaptive direct search and variable neighborhood search. J. Global Optim. 41, 299–318 (2008)CrossRefGoogle Scholar
  6. 6.
    Bäck, T.: Evolutionary Algorithms in Theory and Practice. Oxford University Press, Oxford (1996)Google Scholar
  7. 7.
    Bäck, T., Fogel, D., Michalewicz, Z.: Handbook of Evolutionary Computation. Oxford University Press, Oxford (1997)CrossRefGoogle Scholar
  8. 8.
    Booker, A.J., Dennis Jr, J.E., Frank, P.D., Serafini, D.B., Torczon, V., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Multidiscip. Optim. 17, 1–13 (1999)CrossRefGoogle Scholar
  9. 9.
    Bussieck, M.R., Drud, A.S., Meeraus, A.: MINLPLib—a collection of test models for mixed-integer nonlinear programming. INFORMS J. Comput. 15, 114–119 (2003)CrossRefGoogle Scholar
  10. 10.
    Coit, D.W., Smith, A.E.: Reliability optimization of series-parallel systems using a genetic algorithm. IEEE Trans. Reliab. 45, 254–266 (1996)CrossRefGoogle Scholar
  11. 11.
    Currin, C., Mitchell, T., Morris, M., Ylvisaker, D.: A Bayesian approach to the design and analysis of computer experiments. Technical report, Oak Ridge National Laboratory, Oak Ridge, TN (1988)Google Scholar
  12. 12.
    Davis, E., Ierapetritou, M.: Kriging based method for the solution of mixed-integer nonlinear programs containing black-box functions. J. Global Optim. 43, 191–205 (2009)CrossRefGoogle Scholar
  13. 13.
    Dorigo, M.: Optimization, Learning and Natural Algorithms. Ph.D. thesis, Politecnico di Milano, Italie (1992)Google Scholar
  14. 14.
    Duchon, J.: Constructive Theory of Functions of Several Variables. Springer, Berlin (1977)Google Scholar
  15. 15.
    Fiechter, C.-N.: A parallel tabu search algorithm for large traveling salesman problems. Discret. Appl. Math. 51, 243–267 (1994)CrossRefGoogle Scholar
  16. 16.
    Friedman, J.H.: Multivariate adaptive regression splines. Ann. Stat. 19, 1–141 (1991)CrossRefGoogle Scholar
  17. 17.
    Gershwin, S.B., Schor, J.E.: Efficient algorithms for buffer space allocation. Ann. Oper. Res. 93, 117–144 (2000)CrossRefGoogle Scholar
  18. 18.
    Glaz, B., Friedmann, P.P., Liu, L.: Surrogate based optimization of helicopter rotor blades for vibration reduction in forward flight. Struct. Multidiscip. Optim. 35, 341–363 (2008)CrossRefGoogle Scholar
  19. 19.
    Glover, F.W.: Heuristics for integer programming using surrogate constraints. Decis. Sci. 8, 156–166 (1977)CrossRefGoogle Scholar
  20. 20.
    Glover, F.W.: Tabu search—part 1. ORSA J. Comput. 1, 190–206 (1989)CrossRefGoogle Scholar
  21. 21.
    Glover, F.W.: Tabu search—part 2. ORSA J. Comput. 2, 4–32 (1990)CrossRefGoogle Scholar
  22. 22.
    Glover, F.W.: A Template for Scatter Search and Path Relinking. Springer, Heidelberg (1998)Google Scholar
  23. 23.
    Gutmann, H.-M.: A radial basis function method for global optimization. J. Global Optim. 19, 201–227 (2001)CrossRefGoogle Scholar
  24. 24.
    Hansen, P., Mladenović, N.: Variable neighborhood search: principles and applications. Eur. J. Oper. Res. 130, 449–467 (2001)CrossRefGoogle Scholar
  25. 25.
    Haupt, R.L.: Antenna design with a mixed integer genetic algorithm. IEEE Trans. Antennas Propag. 55, 577–582 (2007)CrossRefGoogle Scholar
  26. 26.
    Hesser, J., Männer, R.: Towards an optimal mutation probability for genetic algorithms. Lect. Notes Comput. Sci. 496, 23–32 (1991)CrossRefGoogle Scholar
  27. 27.
    Holland, J.H.: Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor (1975)Google Scholar
  28. 28.
    Holmström, K.: An adaptive radial basis algorithm (ARBF) for expensive black-box global optimization. J. Global Optim. 41, 447–464 (2008)CrossRefGoogle Scholar
  29. 29.
    Hu, J., Ogras, U.Y., Marculescu, R.: System-level buffer allocation for application-specific networks-on-chip router design. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 25, 2919–2933 (2006)CrossRefGoogle Scholar
  30. 30.
    Jones, D.R.: A taxonomy of global optimization methods based on response surfaces. J. Global Optim. 21, 345–383 (2001)CrossRefGoogle Scholar
  31. 31.
    Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Global Optim. 13, 455–492 (1998)CrossRefGoogle Scholar
  32. 32.
    Jouhaud, J.-C., Sagaut, P., Montagnac, M., Laurenceau, J.: A surrogate-model based multidisciplinary shape optimization method with application to a 2D subsonic airfoil. Comput. Fluids 36, 520–529 (2007)CrossRefGoogle Scholar
  33. 33.
    Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of IEEE International Conference on, Neural Networks, vol. 4, pp. 1942–1948 (1995)Google Scholar
  34. 34.
    Kennedy, J., Eberhart, R.: Swarm Intelligence. Morgan Kaufmann, San Francisco (2001)Google Scholar
  35. 35.
    Koziel, S., Michalewicz, Z.: Evolutionary algorithms, homomorphous mappings, and constrained parameter optimization. Evolut. Comput. 7, 19–44 (1999)CrossRefGoogle Scholar
  36. 36.
    Laguna, M., Martí, R.: Experimental testing of advanced scatter search designs for global optimization of multimodal functions. Technical report, University of Colorado at Boulder (2000)Google Scholar
  37. 37.
    Lam, X.B., Kim, Y.S., Hoang, A.D., Park, C.W.: Coupled aerostructural design optimization using the kriging model and integrated multiobjective optimization algorithm. J. Optim. Theory Appl. 142, 533–556 (2009)CrossRefGoogle Scholar
  38. 38.
    Land, A.H., Doig, A.G.: An automatic method of solving discrete programming problems. Econometrica 28, 497–520 (1960)CrossRefGoogle Scholar
  39. 39.
    Laskari, E.C., Parsopoulos, K.E., Vrahatis, M.N.: Particle swarm optimization for integer programming. In: Proceedings of the 2002 Congress on Evolutionary Computation vol. 2, pp. 1582–1587 (2002)Google Scholar
  40. 40.
    Li, C., Wang, F.-L., Chang, Y.-Q., Liu, Y.: A modified global optimization method based on surrogate model and its application in packing profile optimization of injection molding process. Int. J. Adv. Manuf. Technol. 48, 505–511 (2010)CrossRefGoogle Scholar
  41. 41.
    Li, F., Shoemaker, C.A., Wei, J., Fu, X.: Estimating maximal annual energy given heterogeneous hydropower generating units with application to the Three Gorges System. J. Water Resour. Plan. Manag. doi: 10.1061/(ASCE)WR.1943-5452.0000250 (2012)
  42. 42.
    Liang, Y.-C.: An ant colony optimization algorithm for the redundancy allocation problem (RAP). IEEE Trans. Reliab. 53, 417–423 (2004)CrossRefGoogle Scholar
  43. 43.
    Malek, M., Guruswamy, M., Pandya, M., Owens, H.: Serial and parallel simulated annealing and tabu search algorithms for the traveling salesman problem. Ann. Oper. Res. 21, 59–84 (1989)CrossRefGoogle Scholar
  44. 44.
    Michalewicz, Z., Fogel, D.: How to Solve it: Modern Heuristics. Springer, New York (2004)CrossRefGoogle Scholar
  45. 45.
    Mladenović, N., Hansen, P.: Variable neighborhood search. Comput. Oper. Res. 24, 1097–1100 (1997)CrossRefGoogle Scholar
  46. 46.
    Moré, J.J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20, 172–191 (2009)CrossRefGoogle Scholar
  47. 47.
    Müller, J., Piché, R.: Mixture surrogate models based on Dempster-Shafer theory for global optimization problems. J. Global Optim. 51, 79–104 (2011)CrossRefGoogle Scholar
  48. 48.
    Müller, J., Shoemaker, C.A., Piché, R.: SO-MI: a surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems. Comput. Oper. Res. 40, 1383–1400 (2013)CrossRefGoogle Scholar
  49. 49.
    Myers, R.H., Montgomery, D.C.: Response Surface Methodology, Process and Product Optimization Using Designed Experiments. Wiley-Interscience Publication, New York (1995)Google Scholar
  50. 50.
    Pichitlamken, J., Nelson, B.L., Hong, L.J.: A sequential procedure for neighborhood selection-of-the-best in optimization via simulation. Eur. J. Oper. Res. 173, 283–298 (2006)CrossRefGoogle Scholar
  51. 51.
    Poli, R.: Analysis of the publications on the applications of particle swarm optimisation. J. Artif. Evolut. Appl. 1–10, 2008 (2008)Google Scholar
  52. 52.
    Powell, M.J.D.: The theory of radial basis function approximation in 1990. In: Advances in Numerical Analysis, Wavelets, Subdivision Algorithms and Radial basis Functions, vol. 2, pp. 105–210. Oxford University Press, Oxford (1992)Google Scholar
  53. 53.
    Rashid, K., Ambani, S., Cetinkaya, E.: An adaptive multiquadric radial basis function method for expensive black-box mixed-integer nonlinear constrained optimization. Eng. Optim. doi: 10.1080/0305215X.2012.665450 (2012)
  54. 54.
    Regis, R.G., Shoemaker, C.A.: Constrained global optimization of expensive black-box functions using radial basis functions. J. Global Optim. 31, 153–171 (2005)CrossRefGoogle Scholar
  55. 55.
    Regis, R.G., Shoemaker, C.A.: A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J. Comput. 19, 497–509 (2007)CrossRefGoogle Scholar
  56. 56.
    Regis, R.G., Shoemaker, C.A.: Improved strategies for radial basis function methods for global optimization. J. Global Optim. 37, 113–135 (2007)CrossRefGoogle Scholar
  57. 57.
    Schoen, F.: A wide class of test functions for global optimization. J. Global Optim. 3, 133–137 (1993)CrossRefGoogle Scholar
  58. 58.
    Shan, S., Wang, G.G.: Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions. Struct. Multidiscip. Optim. 41, 219–241 (2010)CrossRefGoogle Scholar
  59. 59.
    Shi, Y., Eberhart, R.: A modified particle swarm optimizer. In: Proceedings of IEEE International Conference on Evolutionary Computation, pp. 69–73 (1998)Google Scholar
  60. 60.
    Spinellis, D.D., Papadopoulos, C.T.: A simulated annealing approach for buffer allocation in reliable production lines. Ann. Oper. Res. 93, 373–384 (2000)CrossRefGoogle Scholar
  61. 61.
    Srinivas, M., Patnaik, L.M.: Adaptive probabilities of crossover and mutation in genetic algorithms. IEEE Trans. Syst. Man Cybern. 24, 565–667 (1994)CrossRefGoogle Scholar
  62. 62.
    Törn, A., Zilinskas, A.: Global Optimization. In: Lecture Notes in Computer Science, 350. Springer, Berlin (1989)Google Scholar
  63. 63.
    Wild, S.M., Regis, R.G., Shoemaker, C.A.: ORBIT: optimization by radial basis function interpolation in trust-regions. SIAM J. Sci. Comput. 30, 3197–3219 (2007)CrossRefGoogle Scholar
  64. 64.
    Wu, Q.H., Cao, Y.J., Wen, J.Y.: Optimal reactive power dispatch using an adaptive genetic algorithm. Int. J. Electr. Power Energy Syst. 20, 563–569 (1998)CrossRefGoogle Scholar
  65. 65.
    Ye, K.Q., Li, W., Sudjianto, A.: Algorithmic construction of optimal symmetric Latin hypercube designs. J. Stat. Plan. Inference 90, 145–159 (2000)CrossRefGoogle Scholar
  66. 66.
    Zhang, J., Chung, H.S.-H., Lo, W.-L.: Clustering-based adaptive crossover and mutation probabilities for genetic algorithms. IEEE Trans. Evol. Comput. 11, 326–335 (2007)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Juliane Müller
    • 1
  • Christine A. Shoemaker
    • 2
    • 3
  • Robert Piché
    • 4
  1. 1.School of Civil and Environmental EngineeringCornell UniversityIthacaUSA
  2. 2.School of Civil and Environmental Engineering, Center of Applied MathematicsCornell UniversityIthacaUSA
  3. 3.School of Operations Research and Information Engineering, Center of Applied MathematicsCornell UniversityIthacaUSA
  4. 4.Department of MathematicsTampere University of TechnologyTampereFinland

Personalised recommendations