Advertisement

Deterministic Global Optimization with Artificial Neural Networks Embedded

  • Artur M. Schweidtmann
  • Alexander Mitsos
Article
  • 277 Downloads

Abstract

Artificial neural networks are used in various applications for data-driven black-box modeling and subsequent optimization. Herein, we present an efficient method for deterministic global optimization of optimization problems with artificial neural networks embedded. The proposed method is based on relaxations of algorithms using McCormick relaxations in a reduced space (Mitsos et al. in SIAM J Optim 20(2):573–601, 2009) employing the convex and concave envelopes of the nonlinear activation function. The optimization problem is solved using our in-house deterministic global solver. The performance of the proposed method is shown in four optimization examples: an illustrative function, a fermentation process, a compressor plant and a chemical process. The results show that computational solution time is favorable compared to a state-of-the-art global general-purpose optimization solver.

Keywords

Surrogate-based optimization Multilayer perceptron McCormick relaxations Machine learning MAiNGO 

Mathematics Subject Classification

90C26 90C30 90C90 68T01 

Notes

Acknowledgements

The authors gratefully acknowledge the financial support of the Kopernikus project SynErgie by the Federal Ministry of Education and Research (BMBF) and the project supervision by the project management organization Projektträger Jülich (PtJ). We are grateful to Jaromił Najam, Dominik Bongartz and Susanne Sass for their work on MAiNGO and Benoît Chachuat for providing MC++. We thank Eduardo Schultz for providing the model of the Cumene process, Adrian Caspari and Pascal Schäfer for helpful discussions and Linus Netze and Nils Graß  for implementing case studies. Finally, we thank the associate editor and the anonymous reviewers for their valuable comments and suggestions.

References

  1. 1.
    Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989).  https://doi.org/10.1016/0893-6080(89)90020-8 CrossRefzbMATHGoogle Scholar
  2. 2.
    Gasteiger, J., Zupan, J.: Neural networks in chemistry. Angew. Chem. Int. Edn. Engl. 32(4), 503–527 (1993).  https://doi.org/10.1002/anie.199305031 CrossRefGoogle Scholar
  3. 3.
    Azlan Hussain, M.: Review of the applications of neural networks in chemical process control—simulation and online implementation. Artif. Intell. Eng. 13(1), 55–68 (1999).  https://doi.org/10.1016/S0954-1810(98)00011-9 CrossRefGoogle Scholar
  4. 4.
    Agatonovic-Kustrin, S., Beresford, R.: Basic concepts of artificial neural network modeling and its application in pharmaceutical research. J. Pharm. Biomed. Anal. 22(5), 717–727 (2000).  https://doi.org/10.1016/S0731-7085(99)00272-1 CrossRefGoogle Scholar
  5. 5.
    Witek-Krowiak, A., Chojnacka, K., Podstawczyk, D., Dawiec, A., Pokomeda, K.: Application of response surface methodology and artificial neural network methods in modelling and optimization of biosorption process. Bioresour. Technol. 160, 150–160 (2014).  https://doi.org/10.1016/j.biortech.2014.01.021 CrossRefGoogle Scholar
  6. 6.
    Meireles, M., Almeida, P., Simoes, M.G.: A comprehensive review for industrial applicability of artificial neural networks. IEEE Trans. Ind. Electron. 50(3), 585–601 (2003).  https://doi.org/10.1109/TIE.2003.812470 CrossRefGoogle Scholar
  7. 7.
    Del Rio-Chanona, E.A., Fiorelli, F., Zhang, D., Ahmed, N.R., Jing, K., Shah, N.: An efficient model construction strategy to simulate microalgal lutein photo-production dynamic process. Biotechnol. Bioeng. 114(11), 2518–2527 (2017).  https://doi.org/10.1002/bit.26373 CrossRefGoogle Scholar
  8. 8.
    Cheema, J.J.S., Sankpal, N.V., Tambe, S.S., Kulkarni, B.D.: Genetic programming assisted stochastic optimization strategies for optimization of glucose to gluconic acid fermentation. Biotechnol. Progr. 18(6), 1356–1365 (2002).  https://doi.org/10.1021/bp015509s CrossRefGoogle Scholar
  9. 9.
    Desai, K.M., Survase, S.A., Saudagar, P.S., Lele, S.S., Singhal, R.S.: Comparison of artificial neural network and response surface methodology in fermentation media optimization: case study of fermentative production of scleroglucan. Biochem. Eng. J. 41(3), 266–273 (2008).  https://doi.org/10.1016/j.bej.2008.05.009 CrossRefGoogle Scholar
  10. 10.
    Nagata, Y., Chu, K.H.: Optimization of a fermentation medium using neural networks and genetic algorithms. Biotechnol. Lett. 25(21), 1837–1842 (2003).  https://doi.org/10.1023/A:1026225526558 CrossRefGoogle Scholar
  11. 11.
    Fahmi, I., Cremaschi, S.: Process synthesis of biodiesel production plant using artificial neural networks as the surrogate models. Comput. Chem. Eng. 46, 105–123 (2012).  https://doi.org/10.1016/j.compchemeng.2012.06.006 CrossRefGoogle Scholar
  12. 12.
    Nascimento, C.A.O., Giudici, R., Guardani, R.: Neural network based approach for optimization of industrial chemical processes. Comput. Chem. Eng. 24(9–10), 2303–2314 (2000).  https://doi.org/10.1016/S0098-1354(00)00587-1 CrossRefGoogle Scholar
  13. 13.
    Nascimento, C.A.O., Giudici, R.: Neural network based approach for optimisation applied to an industrial nylon-6,6 polymerisation process. Comput. Chem. Eng. 22, 595–S600 (1998).  https://doi.org/10.1016/S0098-1354(98)00105-7 CrossRefGoogle Scholar
  14. 14.
    Chambers, M., Mount-Campbell, C.A.: Process optimization via neural network metamodeling. Int. J. Prod. Econ. 79(2), 93–100 (2002).  https://doi.org/10.1016/S0925-5273(00)00188-2 CrossRefGoogle Scholar
  15. 15.
    Henao, C.A., Maravelias, C.T.: Surrogate-based process synthesis. In: Pierucci, S., Ferraris, G.B. (eds.) 20th European Symposium on Computer Aided Process Engineering, Computer Aided Chemical Engineering, vol. 28, pp. 1129–1134. Elsevier, Milan, Italy (2010).  https://doi.org/10.1016/S1570-7946(10)28189-0 Google Scholar
  16. 16.
    Henao, C.A., Maravelias, C.T.: Surrogate-based superstructure optimization framework. AIChE J. 57(5), 1216–1232 (2011).  https://doi.org/10.1002/aic.12341 CrossRefGoogle Scholar
  17. 17.
    Sant Anna, H.R., Barreto, A.G., Tavares, F.W., de Souza, M.B.: Machine learning model and optimization of a PSA unit for methane–nitrogen separation. Comput. Chem. Eng. 104, 377–391 (2017).  https://doi.org/10.1016/j.compchemeng.2017.05.006 CrossRefGoogle Scholar
  18. 18.
    Smith, J.D., Neto, A.A., Cremaschi, S., Crunkleton, D.W.: CFD-based optimization of a flooded bed algae bioreactor. Ind. Eng. Chem. Res. 52(22), 7181–7188 (2013).  https://doi.org/10.1021/ie302478d CrossRefGoogle Scholar
  19. 19.
    Henao, C.A.: A superstructure modeling framework for process synthesis using surrogate models. Dissertation, University of Wisconsin, Madison (2012)Google Scholar
  20. 20.
    Kajero, O.T., Chen, T., Yao, Y., Chuang, Y.C., Wong, D.S.H.: Meta-modelling in chemical process system engineering. J. Taiwan Inst. Chem. Eng. 73, 135–145 (2017).  https://doi.org/10.1016/j.jtice.2016.10.042 CrossRefGoogle Scholar
  21. 21.
    Lewandowski, J., Lemcoff, N.O., Palosaari, S.: Use of neural networks in the simulation and optimization of pressure swing adsorption processes. Chem. Eng. Technol. 21(7), 593–597 (1998). https://doi.org/10.1002/(SICI)1521-4125(199807)21:7\(<\)593::AID-CEAT593\(>\)3.0.CO;2-UCrossRefGoogle Scholar
  22. 22.
    Gutiérrez-Antonio, C.: Multiobjective stochastic optimization of dividing-wall distillation columns using a surrogate model based on neural networks. Chem. Biochem. Eng. Q. 29(4), 491–504 (2016).  https://doi.org/10.15255/CABEQ.2014.2132 CrossRefGoogle Scholar
  23. 23.
    Chen, C.R., Ramaswamy, H.S.: Modeling and optimization of variable retort temperature thermal processing using coupled neural networks and genetic algorithms. J. Food Eng. 53(3), 209–220 (2002).  https://doi.org/10.1016/S0260-8774(01)00159-5 CrossRefGoogle Scholar
  24. 24.
    Dornier, M., Decloux, M., Trystram, G., Lebert, A.: Interest of neural networks for the optimization of the crossflow filtration process. LWT-Food Sci. Technol. 28(3), 300–309 (1995).  https://doi.org/10.1016/S0023-6438(95)94364-1 CrossRefGoogle Scholar
  25. 25.
    Fernandes, F.A.N.: Optimization of Fischer–Tropsch synthesis using neural networks. Chem. Eng. Technol. 29(4), 449–453 (2006).  https://doi.org/10.1002/ceat.200500310 CrossRefGoogle Scholar
  26. 26.
    Grossmann, I.E., Viswanathan, J., Vecchietti, A., Raman, R., Kalvelagen, E.: GAMS/DICOPT: A discrete continuous optimization package. GAMS Corporation Inc, Cary (2002)Google Scholar
  27. 27.
    Drud, A.S.: Conopt—a large-scale GRG code. ORSA J. Comput. 6(2), 207–216 (1994).  https://doi.org/10.1287/ijoc.6.2.207 CrossRefzbMATHGoogle Scholar
  28. 28.
    Nandi, S., Ghosh, S., Tambe, S.S., Kulkarni, B.D.: Artificial neural-network-assisted stochastic process optimization strategies. AIChE J. 47(1), 126–141 (2001).  https://doi.org/10.1002/aic.690470113 CrossRefGoogle Scholar
  29. 29.
    Tawarmalani, M., Sahinidis, N.V.: A polyhedral branch-and-cut approach to global optimization. Math. Program. 103(2), 225–249 (2005).  https://doi.org/10.1007/s10107-005-0581-8 MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    de Weerdt, E., Chu, Q.P., Mulder, J.A.: Neural network output optimization using interval analysis. IEEE Trans. Neural Netw. 20(4), 638–653 (2009).  https://doi.org/10.1109/TNN.2008.2011267 CrossRefGoogle Scholar
  31. 31.
    Moore, R.E., Bierbaum, F.: Methods and Applications of Interval Analysis, 2 edn. SIAM Studies in Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia (1979).  https://doi.org/10.1137/1.9781611970906
  32. 32.
    Misener, R., Floudas, C.A.: ANTIGONE: Algorithms for continuous/integer global optimization of nonlinear equations. J. Glob. Optim. 59(2), 503–526 (2014).  https://doi.org/10.1007/s10898-014-0166-2 MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Maher, S.J., Fischer, T., Gally, T., Gamrath, G., Gleixner, A., Gottwald, R.L., Hendel, G., Koch, T., Lübbecke, M.E., Miltenberger, M., Müller, B., Pfetsch, M.E., Puchert, C., Rehfeldt, D., Schenker, S., Schwarz, R., Serrano, F., Shinano, Y., Weninger, D., Witt, J.T., Witzig, J.: The SCIP optimization suite (version 4.0)Google Scholar
  34. 34.
    Epperly, T.G.W., Pistikopoulos, E.N.: A reduced space branch and bound algorithm for global optimization. J. Glob. Optim. 11(3), 287–311 (1997).  https://doi.org/10.1023/A:1008212418949 MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Mitsos, A., Chachuat, B., Barton, P.I.: McCormick-based relaxations of algorithms. SIAM J. Optim. 20(2), 573–601 (2009).  https://doi.org/10.1137/080717341 MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Scott, J.K., Stuber, M.D., Barton, P.I.: Generalized McCormick relaxations. J. Glob. Optim. 51(4), 569–606 (2011).  https://doi.org/10.1007/s10898-011-9664-7 MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Bongartz, D., Mitsos, A.: Deterministic global optimization of process flowsheets in a reduced space using McCormick relaxations. J. Glob. Optim. 20(9), 419 (2017).  https://doi.org/10.1007/s10898-017-0547-4 MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Huster, W.R., Bongartz, D., Mitsos, A.: Deterministic global optimization of the design of a geothermal organic rankine cycle. Energy Proc. 129, 50–57 (2017).  https://doi.org/10.1016/j.egypro.2017.09.181 CrossRefGoogle Scholar
  39. 39.
    McCormick, G.P.: Computability of global solutions to factorable nonconvex programs: part I—convex underestimating problems. Math. Program. 10(1), 147–175 (1976).  https://doi.org/10.1007/BF01580665 CrossRefzbMATHGoogle Scholar
  40. 40.
    Bompadre, A., Mitsos, A.: Convergence rate of McCormick relaxations. J. Glob. Optim. 52(1), 1–28 (2012).  https://doi.org/10.1007/s10898-011-9685-2 MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Najman, J., Mitsos, A.: Convergence analysis of multivariate McCormick relaxations. J. Glob. Optim. 66(4), 597–628 (2016).  https://doi.org/10.1007/s10898-016-0408-6 MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Tsoukalas, A., Mitsos, A.: Multivariate McCormick relaxations. J. Glob. Optim. 59(2–3), 633–662 (2014).  https://doi.org/10.1007/s10898-014-0176-0 MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Najman, J., Bongartz, D., Tsoukalas, A., Mitsos, A.: Erratum to multivariate McCormick relaxations. J. Glob. Optim. 68(1), 219–225 (2017).  https://doi.org/10.1007/s10898-016-0470-0 CrossRefzbMATHGoogle Scholar
  44. 44.
    Khan, K.A., Watson, H.A.J., Barton, P.I.: Differentiable McCormick relaxations. J. Glob. Optim. 67(4), 687–729 (2017).  https://doi.org/10.1007/s10898-016-0440-6 MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Khan, K.A., Wilhelm, M., Stuber, M.D., Cao, H., Watson, H.A.J., Barton, P.I.: Corrections to differentiable McCormick relaxations. J. Glob. Optim. 70(3), 705–706 (2018).  https://doi.org/10.1007/s10898-017-0601-2 CrossRefzbMATHGoogle Scholar
  46. 46.
    Bongartz, D., Mitsos, A.: Infeasible path global flowsheet optimization using McCormick relaxations. In: Espuna, A. (ed.) 27th European Symposium on Computer Aided Process Engineering, Computer Aided Chemical Engineering, vol. 40. Elsevier, San Diego (2017).  https://doi.org/10.1016/B978-0-444-63965-3.50107-0 Google Scholar
  47. 47.
    Wechsung, A., Scott, J.K., Watson, H.A.J., Barton, P.I.: Reverse propagation of McCormick relaxations. J. Glob. Optim. 63(1), 1–36 (2015).  https://doi.org/10.1007/s10898-015-0303-6 MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Stuber, M.D., Scott, J.K., Barton, P.I.: Convex and concave relaxations of implicit functions. Optim. Methods Softw. 30(3), 424–460 (2015).  https://doi.org/10.1080/10556788.2014.924514 MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics, 8th edn. Springer, New York (2009)Google Scholar
  50. 50.
    Bertsekas, D.P., Nedic, A., Ozdaglar, A.E.: Convex Analysis and Optimization, Athena Scientific Optimization and Computation Series, vol. 1. Athena Scientific, Belmont (2003)zbMATHGoogle Scholar
  51. 51.
    Bongartz, D., Najman, J., Sass, S., Mitsos, A.: MAiNGO: McCormick based Algorithm for mixed integer Nonlinear Global Optimization. Technical report (2018)Google Scholar
  52. 52.
    Chachuat, B.: MC++ (version 2.0): A toolkit for bounding factorable functions (2014)Google Scholar
  53. 53.
    Chachuat, B., Houska, B., Paulen, R., Peri’c, N., Rajyaguru, J., Villanueva, M.E.: Set-theoretic approaches in analysis, estimation and control of nonlinear systems. IFAC-PapersOnLine 48(8), 981–995 (2015).  https://doi.org/10.1016/j.ifacol.2015.09.097 CrossRefGoogle Scholar
  54. 54.
    Hofschuster, W., Krämer, W.: FILIB++ Interval Library (version 3.0.2) (1998)Google Scholar
  55. 55.
    International Business Machies: IBM ilog CPLEX (version 12.1) (2009)Google Scholar
  56. 56.
    Gleixner, A.M., Berthold, T., Müller, B., Weltge, S.: Three enhancements for optimization-based bound tightening. J. Glob. Optim. 67(4), 731–757 (2017).  https://doi.org/10.1007/s10898-016-0450-4 MathSciNetCrossRefzbMATHGoogle Scholar
  57. 57.
    Ryoo, H.S., Sahinidis, N.V.: Global optimization of nonconvex NLPs and MINLPs with applications in process design. Comput. Chem. Eng. 19(5), 551–566 (1995).  https://doi.org/10.1016/0098-1354(94)00097-2 CrossRefGoogle Scholar
  58. 58.
    Locatelli, M., Schoen, F. (eds.): Global optimization: theory, algorithms, and applications. MOS-SIAM series on optimization. Mathematical Programming Society, Philadelphia, PA (2013).  https://doi.org/10.1137/1.9781611972672
  59. 59.
    Kraft, D.: A software package for sequential quadratic programming. Deutsche Forschungs- und Versuchsanstalt für Luft- und Raumfahrt Köln: Forschungsbericht. Wiss. Berichtswesen d. DFVLR, Köln (1988)Google Scholar
  60. 60.
    Johnson, S.G.: The NLopt nonlinear-optimization package (version 2.4.2) (2016)Google Scholar
  61. 61.
    Bendtsen, C., Stauning, O.: Fadbad++ (version 2.1): a flexible C++ package for automatic differentiation (2012)Google Scholar
  62. 62.
    Najman, J., Mitsos, A.: Tighter McCormick relaxations through subgradient propagation. Optimization online. http://www.optimization-online.org/DB_FILE/2017/10/6296.pdf (2017)
  63. 63.
    Ghorbanian, K., Gholamrezaei, M.: An artificial neural network approach to compressor performance prediction. Appl. Energy 86(7–8), 1210–1221 (2009).  https://doi.org/10.1016/j.apenergy.2008.06.006 CrossRefGoogle Scholar
  64. 64.
    Luyben, W.L.: Design and control of the cumene process. Ind. Eng. Chem. Res. 49(2), 719–734 (2010).  https://doi.org/10.1021/ie9011535 CrossRefGoogle Scholar
  65. 65.
    Schultz, E.S., Trierweiler, J.O., Farenzena, M.: The importance of nominal operating point selection in self-optimizing control. Ind. Eng. Chem. Res. 55(27), 7381–7393 (2016).  https://doi.org/10.1021/acs.iecr.5b02044 CrossRefGoogle Scholar
  66. 66.
    Lee, U., Burre, J., Caspari, A., Kleinekorte, J., Schweidtmann, A.M., Mitsos, A.: Techno-economic optimization of a green-field post-combustion CO\(_2\) capture process using superstructure and rate-based models. Ind. Eng. Chem. Res. 55(46), 12014–12026 (2016).  https://doi.org/10.1021/acs.iecr.6b01668 CrossRefGoogle Scholar
  67. 67.
    Helmdach, D., Yaseneva, P., Heer, P.K., Schweidtmann, A.M., Lapkin, A.A.: A multiobjective optimization including results of life cycle assessment in developing biorenewables-based processes. ChemSusChem 10(18), 3632–3643 (2017).  https://doi.org/10.1002/cssc.201700927 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Aachener Verfahrenstechnik, Process Systems EngineeringRWTH Aachen UniversityAachenGermany

Personalised recommendations