Skip to main content
SpringerLink
Log in

Asynchronously parallel optimization solver for finding multiple minima

  • Full Length Paper
  • Published:
Mathematical Programming Computation Aims and scope Submit manuscript

Abstract

We propose and analyze an asynchronously parallel optimization algorithm for finding multiple, high-quality minima of nonlinear optimization problems. Our multistart algorithm considers all previously evaluated points when determining where to start or continue a local optimization run. Theoretical results show that when there are finitely many minima, the algorithm almost surely starts a finite number of local optimization runs and identifies every minimum. The algorithm is applicable to general optimization settings, but our numerical results focus on the case when derivatives are unavailable. In numerical tests, a Python implementation of the algorithm is shown to yield good approximations of many minima (including a global minimum), and this ability is shown to scale well with additional resources. Our implementation’s time to solution is shown also to scale well even when the time to perform the function evaluation is highly variable. An implementation of the algorithm is available in the libEnsemble library at https://github.com/Libensemble/libensemble.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Audet, C., Dennis Jr., J.E., Le Digabel, S.: Parallel space decomposition of the mesh adaptive direct search algorithm. SIAM J. Optim. 19(3), 1150–1170 (2008). https://doi.org/10.1137/070707518

    Article  MathSciNet  MATH  Google Scholar 

  2. Balay, S., Abhyankar, S., Adams, M.F., Brown, J., Brune, P., Buschelman, K., Dalcin, L., Eijkhout, V., Gropp, W.D., Kaushik, D., Knepley, M.G., McInnes, L.C., Rupp, K., Smith, B.F., Zampini, S., Zhang, H.: PETSc Web page (2017). http://www.mcs.anl.gov/petsc

  3. Besserud, K., Cotten, J.: Architectural genomics, silicon + skin: biological processes and computation. In: Proceedings of the 28th Annual Conference of the Association for Computer Aided Design in Architecture, pp. 978–989 (2008)

  4. Cave, R.J., Burke, K., Castner Jr., E.W.: Theoretical investigation of the ground and excited states of Coumarin 151 and Coumarin 120. J. Phys. Chem. A 106(40), 9294–9305 (2002). https://doi.org/10.1021/jp026071x

    Article  Google Scholar 

  5. Custódio, A.L., Madeira, J.F.A.: GLODS: global and local optimization using direct search. J. Glob. Optim. 62(1), 1–28 (2015). https://doi.org/10.1007/s10898-014-0224-9

    Article  MathSciNet  MATH  Google Scholar 

  6. Dalcín, L., Paz, R., Storti, M., D’Elía, J.: MPI for Python: performance improvements and MPI-2 extensions. J. Parallel Distrib. Comput. 68(5), 655–662 (2008). https://doi.org/10.1016/j.jpdc.2007.09.005

    Article  Google Scholar 

  7. Easterling, D.R., Watson, L.T., Madigan, M.L., Castle, B.S., Trosset, M.W.: Parallel deterministic and stochastic global minimization of functions with very many minima. Comput. Optim. Appl. 57(2), 469–492 (2014). https://doi.org/10.1007/s10589-013-9592-1

    Article  MathSciNet  MATH  Google Scholar 

  8. García-Palomares, U.M., Rodríguez, J.F.: New sequential and parallel derivative-free algorithms for unconstrained minimization. SIAM J. Optim. 13(1), 79–96 (2002). https://doi.org/10.1137/S1052623400370606

    Article  MathSciNet  MATH  Google Scholar 

  9. Gaviano, M., Kvasov, D.E., Lera, D., Sergeyev, Y.D.: Algorithm 829: software for generation of classes of test functions with known local and global minima for global optimization. ACM Trans. Math. Softw. 29(4), 469–480 (2003). https://doi.org/10.1145/962437.962444

    Article  MathSciNet  MATH  Google Scholar 

  10. Gheribi, A.E., Robelin, C., Le Digabel, S., Audet, C., Pelton, A.D.: Calculating all local minima on liquidus surfaces using the FactSage software and databases and the mesh adaptive direct search algorithm. J. Chem. Thermodyn. 43(9), 1323–1330 (2011). https://doi.org/10.1016/j.jct.2011.03.021

    Article  Google Scholar 

  11. Gray, G.A., Kolda, T.G.: Algorithm 856: APPSPACK 4.0: asynchronous parallel pattern search for derivative-free optimization. ACM Trans. Math. Softw. 32(3), 485–507 (2006). https://doi.org/10.1145/1163641.1163647

    Article  MathSciNet  MATH  Google Scholar 

  12. Hansen, N.: CMA-ES. https://www.lri.fr/~hansen/cmaes_inmatlab.html#matlab. Accessed Nov 2016

  13. Hansen, N., Müller, S.D., Koumoutsakos, P.: Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11(1), 1–18 (2003). https://doi.org/10.1162/106365603321828970

    Article  Google Scholar 

  14. He, J., Verstak, A., Sosonkina, M., Watson, L.: Performance modeling and analysis of a massively parallel DIRECT-Part 2. Int. J. High Perform. Comput. Appl. 23(1), 29–41 (2009). https://doi.org/10.1177/1094342008098463

    Article  Google Scholar 

  15. He, J., Verstak, A., Watson, L., Sosonkina, M.: Performance modeling and analysis of a massively parallel DIRECT-Part 1. Int. J. High Perform. Comput. Appl. 23(1), 14–28 (2009). https://doi.org/10.1177/1094342008098462

    Article  Google Scholar 

  16. He, J., Verstak, A., Watson, L.T., Sosonkina, M.: Design and implementation of a massively parallel version of DIRECT. Comput. Optim. Appl. 40(2), 217–245 (2007). https://doi.org/10.1007/s10589-007-9092-2

    Article  MathSciNet  MATH  Google Scholar 

  17. Hough, P.D., Kolda, T.G., Torczon, V.J.: Asynchronous parallel pattern search for nonlinear optimization. SIAM J. Sci. Comput. 23(1), 134–156 (2001). https://doi.org/10.1137/S1064827599365823

    Article  MathSciNet  MATH  Google Scholar 

  18. Johnson, S.G.: The NLopt Nonlinear-Optimization Package. http://ab-initio.mit.edu/nlopt (2017)

  19. Jones, D.R., Perttunen, C.D., Stuckman, B.E.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl. 79(1), 157–181 (1993). https://doi.org/10.1007/BF00941892

    Article  MathSciNet  MATH  Google Scholar 

  20. Larson, J.: libEnsemble. https://github.com/Libensemble/libensemble (2017)

  21. Larson, J., Wild, S.M.: A batch, derivative-free algorithm for finding multiple local minima. Optim. Eng. 17(1), 205–228 (2016). https://doi.org/10.1007/s11081-015-9289-7

    Article  MathSciNet  MATH  Google Scholar 

  22. Liuzzi, G., Truemper, K.: Parallelized hybrid optimization methods for nonsmooth problems using NOMAD and linesearch. Comput. Appl. Math. (2017). https://doi.org/10.1007/s40314-017-0505-2

    Google Scholar 

  23. Moré, J.J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009). https://doi.org/10.1137/080724083

    Article  MathSciNet  MATH  Google Scholar 

  24. Olsson, P.M.: Methods for Network Optimization and Parallel Derivative-Free Optimization, Ph.D. Thesis. Linköping University. http://liu.diva-portal.org/smash/get/diva2:695431/FULLTEXT02.pdf (2014)

  25. Plantenga, T.D.: HOPSPACK 3.0 User Manual, Technical Report October. Sandia National Laboratories, Albuquerque (2009)

  26. Powell, M.J.D.: The BOBYQA Algorithm for Bound Constrained Optimization Without Derivatives, Technical Report. DAMTP 2009/NA06, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (2009)

  27. Rinnooy Kan, A.H.G., Timmer, G.T.: Stochastic global optimization methods, part I: clustering methods. Math. Program. 39(1), 27–56 (1987). https://doi.org/10.1007/BF02592070

    Article  MATH  Google Scholar 

  28. Rinnooy Kan, A.H.G., Timmer, G.T.: Stochastic global optimization methods, part II: multi level methods. Math. Program. 39(1), 57–78 (1987). https://doi.org/10.1007/BF02592071

    Article  MATH  Google Scholar 

  29. Ross, S.M.: A First Course in Probability, 8th edn. Prentice Hall, Upper Saddle River (2009)

    Google Scholar 

  30. Törn, A., Zilinskas, A.: Global Optimization. Springer, New York (1989). https://doi.org/10.1007/3-540-50871-6

    Book  MATH  Google Scholar 

  31. Vanden Berghen, F.: CONDOR: A Constrained, Non-linear, Derivative-Free Parallel Optimizer for Continuous, High Computing Load, Noisy Objective Functions, Ph.D. Thesis. Université Libre de Bruxelles. http://www.applied-mathematics.net/optimization/thesis_optimization.pdf (2004)

  32. Vaz, A.I.F., Vicente, L.N.: A particle swarm pattern search method for bound constrained global optimization. J. Glob. Optim. 39(2), 197–219 (2007). https://doi.org/10.1007/s10898-007-9133-5

    Article  MathSciNet  MATH  Google Scholar 

  33. Wild, S.M.: Derivative-Free Optimization Algorithms for Computationally Expensive Functions, Ph.D. Thesis. Cornell University. http://ecommons.cornell.edu/handle/1813/11248 (2009)

Download references

Acknowledgements

This material was based upon work supported by the U.S. Department of Energy, Office of Science through the Office of Advanced Scientific Computing Research (Contract No. DE-AC02-06CH11357) and the Exascale Computing Project (Contract No. 17-SC-20-SC). We gratefully acknowledge the computing resources provided by the Laboratory Computing Resource Center at Argonne National Laboratory. We thank Gail Pieper for her editing.

Author information

Authors and Affiliations

  1. Argonne National Laboratory, 9700 S. Cass Ave., Bldg. 240, Lemont, IL, 60439, USA

    Jeffrey Larson & Stefan M. Wild

Authors
  1. Jeffrey Larson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefan M. Wild
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan M. Wild.

A The APOSMM implementation in libEnsemble

A The APOSMM implementation in libEnsemble

The libEnsemble library [20] is designed to manage collections of computations such as those related to concurrent objective function evaluations; libEnsemble includes an implementation of APOSMM. The libEnsemble library coordinates the custodian, manager, and worker functions presented in Fig. 1. By letting libEnsemble handle the coordination of (asynchronous or batch) function evaluations and other calculations, APOSMM can focus on efficiently requesting particular points to be evaluated. This allows flexibility for users wish to customize APOSMM or write their own point-generating and/or point-prioritizing routines.

The sequence of points generated by APOSMM within libEnsemble correspond to the sequence generated by Algorithm 1. When APOSMM is queried by libEnsemble (after prior/initial evaluations have been performed), APOSMM generates points in existing local optimization runs or starts new runs as necessary. If no local optimization runs exist or if insufficient sampling has occurred, then APOSMM produces points drawn uniformly from the specified domain.

The libEnsemble library contains a growing set of example scripts to call APOSMM (and other generating functions) to optimize specified objective functions under different settings. Provided objective examples include simple Python functions as well as functions based on output from a simulation involving an external MPI-based parallel evaluation. The libEnsemble library also includes example scripts capable of terminating unresponsive simulation-based objectives or observing intermediate simulation output and preempting evaluations. Algorithmic parameters for APOSMM are readily available as part of the examples provided in libEnsemble.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Larson, J., Wild, S.M. Asynchronously parallel optimization solver for finding multiple minima. Math. Prog. Comp. 10, 303–332 (2018). https://doi.org/10.1007/s12532-017-0131-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12532-017-0131-4

Keywords

Mathematics Subject Classification

Search

Navigation