Deterministic Global Optimization with Artificial Neural Networks Embedded

  • Artur M. Schweidtmann
  • Alexander MitsosEmail author


Artificial neural networks are used in various applications for data-driven black-box modeling and subsequent optimization. Herein, we present an efficient method for deterministic global optimization of optimization problems with artificial neural networks embedded. The proposed method is based on relaxations of algorithms using McCormick relaxations in a reduced space (Mitsos et al. in SIAM J Optim 20(2):573–601, 2009) employing the convex and concave envelopes of the nonlinear activation function. The optimization problem is solved using our in-house deterministic global solver. The performance of the proposed method is shown in four optimization examples: an illustrative function, a fermentation process, a compressor plant and a chemical process. The results show that computational solution time is favorable compared to a state-of-the-art global general-purpose optimization solver.


Surrogate-based optimization Multilayer perceptron McCormick relaxations Machine learning MAiNGO 

Mathematics Subject Classification

90C26 90C30 90C90 68T01 



The authors gratefully acknowledge the financial support of the Kopernikus project SynErgie by the Federal Ministry of Education and Research (BMBF) and the project supervision by the project management organization Projektträger Jülich (PtJ). We are grateful to Jaromił Najam, Dominik Bongartz and Susanne Sass for their work on MAiNGO and Benoît Chachuat for providing MC++. We thank Eduardo Schultz for providing the model of the Cumene process, Adrian Caspari and Pascal Schäfer for helpful discussions and Linus Netze and Nils Graß  for implementing case studies. Finally, we thank the associate editor and the anonymous reviewers for their valuable comments and suggestions.


  1. 1.
    Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989). zbMATHGoogle Scholar
  2. 2.
    Gasteiger, J., Zupan, J.: Neural networks in chemistry. Angew. Chem. Int. Edn. Engl. 32(4), 503–527 (1993). Google Scholar
  3. 3.
    Azlan Hussain, M.: Review of the applications of neural networks in chemical process control—simulation and online implementation. Artif. Intell. Eng. 13(1), 55–68 (1999). Google Scholar
  4. 4.
    Agatonovic-Kustrin, S., Beresford, R.: Basic concepts of artificial neural network modeling and its application in pharmaceutical research. J. Pharm. Biomed. Anal. 22(5), 717–727 (2000). Google Scholar
  5. 5.
    Witek-Krowiak, A., Chojnacka, K., Podstawczyk, D., Dawiec, A., Pokomeda, K.: Application of response surface methodology and artificial neural network methods in modelling and optimization of biosorption process. Bioresour. Technol. 160, 150–160 (2014). Google Scholar
  6. 6.
    Meireles, M., Almeida, P., Simoes, M.G.: A comprehensive review for industrial applicability of artificial neural networks. IEEE Trans. Ind. Electron. 50(3), 585–601 (2003). Google Scholar
  7. 7.
    Del Rio-Chanona, E.A., Fiorelli, F., Zhang, D., Ahmed, N.R., Jing, K., Shah, N.: An efficient model construction strategy to simulate microalgal lutein photo-production dynamic process. Biotechnol. Bioeng. 114(11), 2518–2527 (2017). Google Scholar
  8. 8.
    Cheema, J.J.S., Sankpal, N.V., Tambe, S.S., Kulkarni, B.D.: Genetic programming assisted stochastic optimization strategies for optimization of glucose to gluconic acid fermentation. Biotechnol. Progr. 18(6), 1356–1365 (2002). Google Scholar
  9. 9.
    Desai, K.M., Survase, S.A., Saudagar, P.S., Lele, S.S., Singhal, R.S.: Comparison of artificial neural network and response surface methodology in fermentation media optimization: case study of fermentative production of scleroglucan. Biochem. Eng. J. 41(3), 266–273 (2008). Google Scholar
  10. 10.
    Nagata, Y., Chu, K.H.: Optimization of a fermentation medium using neural networks and genetic algorithms. Biotechnol. Lett. 25(21), 1837–1842 (2003). Google Scholar
  11. 11.
    Fahmi, I., Cremaschi, S.: Process synthesis of biodiesel production plant using artificial neural networks as the surrogate models. Comput. Chem. Eng. 46, 105–123 (2012). Google Scholar
  12. 12.
    Nascimento, C.A.O., Giudici, R., Guardani, R.: Neural network based approach for optimization of industrial chemical processes. Comput. Chem. Eng. 24(9–10), 2303–2314 (2000). Google Scholar
  13. 13.
    Nascimento, C.A.O., Giudici, R.: Neural network based approach for optimisation applied to an industrial nylon-6,6 polymerisation process. Comput. Chem. Eng. 22, 595–S600 (1998). Google Scholar
  14. 14.
    Chambers, M., Mount-Campbell, C.A.: Process optimization via neural network metamodeling. Int. J. Prod. Econ. 79(2), 93–100 (2002). Google Scholar
  15. 15.
    Henao, C.A., Maravelias, C.T.: Surrogate-based process synthesis. In: Pierucci, S., Ferraris, G.B. (eds.) 20th European Symposium on Computer Aided Process Engineering, Computer Aided Chemical Engineering, vol. 28, pp. 1129–1134. Elsevier, Milan, Italy (2010).
  16. 16.
    Henao, C.A., Maravelias, C.T.: Surrogate-based superstructure optimization framework. AIChE J. 57(5), 1216–1232 (2011). Google Scholar
  17. 17.
    Sant Anna, H.R., Barreto, A.G., Tavares, F.W., de Souza, M.B.: Machine learning model and optimization of a PSA unit for methane–nitrogen separation. Comput. Chem. Eng. 104, 377–391 (2017). Google Scholar
  18. 18.
    Smith, J.D., Neto, A.A., Cremaschi, S., Crunkleton, D.W.: CFD-based optimization of a flooded bed algae bioreactor. Ind. Eng. Chem. Res. 52(22), 7181–7188 (2013). Google Scholar
  19. 19.
    Henao, C.A.: A superstructure modeling framework for process synthesis using surrogate models. Dissertation, University of Wisconsin, Madison (2012)Google Scholar
  20. 20.
    Kajero, O.T., Chen, T., Yao, Y., Chuang, Y.C., Wong, D.S.H.: Meta-modelling in chemical process system engineering. J. Taiwan Inst. Chem. Eng. 73, 135–145 (2017). Google Scholar
  21. 21.
    Lewandowski, J., Lemcoff, N.O., Palosaari, S.: Use of neural networks in the simulation and optimization of pressure swing adsorption processes. Chem. Eng. Technol. 21(7), 593–597 (1998).\(<\)593::AID-CEAT593\(>\)3.0.CO;2-UGoogle Scholar
  22. 22.
    Gutiérrez-Antonio, C.: Multiobjective stochastic optimization of dividing-wall distillation columns using a surrogate model based on neural networks. Chem. Biochem. Eng. Q. 29(4), 491–504 (2016).
  23. 23.
    Chen, C.R., Ramaswamy, H.S.: Modeling and optimization of variable retort temperature thermal processing using coupled neural networks and genetic algorithms. J. Food Eng. 53(3), 209–220 (2002). Google Scholar
  24. 24.
    Dornier, M., Decloux, M., Trystram, G., Lebert, A.: Interest of neural networks for the optimization of the crossflow filtration process. LWT-Food Sci. Technol. 28(3), 300–309 (1995). Google Scholar
  25. 25.
    Fernandes, F.A.N.: Optimization of Fischer–Tropsch synthesis using neural networks. Chem. Eng. Technol. 29(4), 449–453 (2006). Google Scholar
  26. 26.
    Grossmann, I.E., Viswanathan, J., Vecchietti, A., Raman, R., Kalvelagen, E.: GAMS/DICOPT: A discrete continuous optimization package. GAMS Corporation Inc, Cary (2002)Google Scholar
  27. 27.
    Drud, A.S.: Conopt—a large-scale GRG code. ORSA J. Comput. 6(2), 207–216 (1994). zbMATHGoogle Scholar
  28. 28.
    Nandi, S., Ghosh, S., Tambe, S.S., Kulkarni, B.D.: Artificial neural-network-assisted stochastic process optimization strategies. AIChE J. 47(1), 126–141 (2001). Google Scholar
  29. 29.
    Tawarmalani, M., Sahinidis, N.V.: A polyhedral branch-and-cut approach to global optimization. Math. Program. 103(2), 225–249 (2005). MathSciNetzbMATHGoogle Scholar
  30. 30.
    de Weerdt, E., Chu, Q.P., Mulder, J.A.: Neural network output optimization using interval analysis. IEEE Trans. Neural Netw. 20(4), 638–653 (2009). Google Scholar
  31. 31.
    Moore, R.E., Bierbaum, F.: Methods and Applications of Interval Analysis, 2 edn. SIAM Studies in Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia (1979).
  32. 32.
    Misener, R., Floudas, C.A.: ANTIGONE: Algorithms for continuous/integer global optimization of nonlinear equations. J. Glob. Optim. 59(2), 503–526 (2014). MathSciNetzbMATHGoogle Scholar
  33. 33.
    Maher, S.J., Fischer, T., Gally, T., Gamrath, G., Gleixner, A., Gottwald, R.L., Hendel, G., Koch, T., Lübbecke, M.E., Miltenberger, M., Müller, B., Pfetsch, M.E., Puchert, C., Rehfeldt, D., Schenker, S., Schwarz, R., Serrano, F., Shinano, Y., Weninger, D., Witt, J.T., Witzig, J.: The SCIP optimization suite (version 4.0)Google Scholar
  34. 34.
    Epperly, T.G.W., Pistikopoulos, E.N.: A reduced space branch and bound algorithm for global optimization. J. Glob. Optim. 11(3), 287–311 (1997). MathSciNetzbMATHGoogle Scholar
  35. 35.
    Mitsos, A., Chachuat, B., Barton, P.I.: McCormick-based relaxations of algorithms. SIAM J. Optim. 20(2), 573–601 (2009). MathSciNetzbMATHGoogle Scholar
  36. 36.
    Scott, J.K., Stuber, M.D., Barton, P.I.: Generalized McCormick relaxations. J. Glob. Optim. 51(4), 569–606 (2011). MathSciNetzbMATHGoogle Scholar
  37. 37.
    Bongartz, D., Mitsos, A.: Deterministic global optimization of process flowsheets in a reduced space using McCormick relaxations. J. Glob. Optim. 20(9), 419 (2017). MathSciNetzbMATHGoogle Scholar
  38. 38.
    Huster, W.R., Bongartz, D., Mitsos, A.: Deterministic global optimization of the design of a geothermal organic rankine cycle. Energy Proc. 129, 50–57 (2017). Google Scholar
  39. 39.
    McCormick, G.P.: Computability of global solutions to factorable nonconvex programs: part I—convex underestimating problems. Math. Program. 10(1), 147–175 (1976). zbMATHGoogle Scholar
  40. 40.
    Bompadre, A., Mitsos, A.: Convergence rate of McCormick relaxations. J. Glob. Optim. 52(1), 1–28 (2012). MathSciNetzbMATHGoogle Scholar
  41. 41.
    Najman, J., Mitsos, A.: Convergence analysis of multivariate McCormick relaxations. J. Glob. Optim. 66(4), 597–628 (2016). MathSciNetzbMATHGoogle Scholar
  42. 42.
    Tsoukalas, A., Mitsos, A.: Multivariate McCormick relaxations. J. Glob. Optim. 59(2–3), 633–662 (2014). MathSciNetzbMATHGoogle Scholar
  43. 43.
    Najman, J., Bongartz, D., Tsoukalas, A., Mitsos, A.: Erratum to multivariate McCormick relaxations. J. Glob. Optim. 68(1), 219–225 (2017). zbMATHGoogle Scholar
  44. 44.
    Khan, K.A., Watson, H.A.J., Barton, P.I.: Differentiable McCormick relaxations. J. Glob. Optim. 67(4), 687–729 (2017). MathSciNetzbMATHGoogle Scholar
  45. 45.
    Khan, K.A., Wilhelm, M., Stuber, M.D., Cao, H., Watson, H.A.J., Barton, P.I.: Corrections to differentiable McCormick relaxations. J. Glob. Optim. 70(3), 705–706 (2018). zbMATHGoogle Scholar
  46. 46.
    Bongartz, D., Mitsos, A.: Infeasible path global flowsheet optimization using McCormick relaxations. In: Espuna, A. (ed.) 27th European Symposium on Computer Aided Process Engineering, Computer Aided Chemical Engineering, vol. 40. Elsevier, San Diego (2017).
  47. 47.
    Wechsung, A., Scott, J.K., Watson, H.A.J., Barton, P.I.: Reverse propagation of McCormick relaxations. J. Glob. Optim. 63(1), 1–36 (2015). MathSciNetzbMATHGoogle Scholar
  48. 48.
    Stuber, M.D., Scott, J.K., Barton, P.I.: Convex and concave relaxations of implicit functions. Optim. Methods Softw. 30(3), 424–460 (2015). MathSciNetzbMATHGoogle Scholar
  49. 49.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics, 8th edn. Springer, New York (2009)Google Scholar
  50. 50.
    Bertsekas, D.P., Nedic, A., Ozdaglar, A.E.: Convex Analysis and Optimization, Athena Scientific Optimization and Computation Series, vol. 1. Athena Scientific, Belmont (2003)zbMATHGoogle Scholar
  51. 51.
    Bongartz, D., Najman, J., Sass, S., Mitsos, A.: MAiNGO: McCormick based Algorithm for mixed integer Nonlinear Global Optimization. Technical report (2018)Google Scholar
  52. 52.
    Chachuat, B.: MC++ (version 2.0): A toolkit for bounding factorable functions (2014)Google Scholar
  53. 53.
    Chachuat, B., Houska, B., Paulen, R., Peri’c, N., Rajyaguru, J., Villanueva, M.E.: Set-theoretic approaches in analysis, estimation and control of nonlinear systems. IFAC-PapersOnLine 48(8), 981–995 (2015). Google Scholar
  54. 54.
    Hofschuster, W., Krämer, W.: FILIB++ Interval Library (version 3.0.2) (1998)Google Scholar
  55. 55.
    International Business Machies: IBM ilog CPLEX (version 12.1) (2009)Google Scholar
  56. 56.
    Gleixner, A.M., Berthold, T., Müller, B., Weltge, S.: Three enhancements for optimization-based bound tightening. J. Glob. Optim. 67(4), 731–757 (2017). MathSciNetzbMATHGoogle Scholar
  57. 57.
    Ryoo, H.S., Sahinidis, N.V.: Global optimization of nonconvex NLPs and MINLPs with applications in process design. Comput. Chem. Eng. 19(5), 551–566 (1995). Google Scholar
  58. 58.
    Locatelli, M., Schoen, F. (eds.): Global optimization: theory, algorithms, and applications. MOS-SIAM series on optimization. Mathematical Programming Society, Philadelphia, PA (2013).
  59. 59.
    Kraft, D.: A software package for sequential quadratic programming. Deutsche Forschungs- und Versuchsanstalt für Luft- und Raumfahrt Köln: Forschungsbericht. Wiss. Berichtswesen d. DFVLR, Köln (1988)Google Scholar
  60. 60.
    Johnson, S.G.: The NLopt nonlinear-optimization package (version 2.4.2) (2016)Google Scholar
  61. 61.
    Bendtsen, C., Stauning, O.: Fadbad++ (version 2.1): a flexible C++ package for automatic differentiation (2012)Google Scholar
  62. 62.
    Najman, J., Mitsos, A.: Tighter McCormick relaxations through subgradient propagation. Optimization online. (2017)
  63. 63.
    Ghorbanian, K., Gholamrezaei, M.: An artificial neural network approach to compressor performance prediction. Appl. Energy 86(7–8), 1210–1221 (2009). Google Scholar
  64. 64.
    Luyben, W.L.: Design and control of the cumene process. Ind. Eng. Chem. Res. 49(2), 719–734 (2010). Google Scholar
  65. 65.
    Schultz, E.S., Trierweiler, J.O., Farenzena, M.: The importance of nominal operating point selection in self-optimizing control. Ind. Eng. Chem. Res. 55(27), 7381–7393 (2016). Google Scholar
  66. 66.
    Lee, U., Burre, J., Caspari, A., Kleinekorte, J., Schweidtmann, A.M., Mitsos, A.: Techno-economic optimization of a green-field post-combustion CO\(_2\) capture process using superstructure and rate-based models. Ind. Eng. Chem. Res. 55(46), 12014–12026 (2016). Google Scholar
  67. 67.
    Helmdach, D., Yaseneva, P., Heer, P.K., Schweidtmann, A.M., Lapkin, A.A.: A multiobjective optimization including results of life cycle assessment in developing biorenewables-based processes. ChemSusChem 10(18), 3632–3643 (2017). Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Aachener Verfahrenstechnik, Process Systems EngineeringRWTH Aachen UniversityAachenGermany

Personalised recommendations