Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

An advanced active set L-BFGS algorithm for training weight-constrained neural networks

  • 16 Accesses

Abstract

In this work, a new advanced active set limited memory BFGS (Broyden–Fletcher–Goldfarb–Shanno) algorithm is proposed for efficiently training weight-constrained neural networks, called AA-L-BFGS. The proposed algorithm possesses the significant property of approximating the curvature of the error function with high-order accuracy by utilizing the theoretically advanced secant condition proposed by Livieris and Pintelas (Appl Math Comput 221:491–502, 2013). Moreover, the global convergence of the proposed algorithm is established provided that the line search satisfies the modified Armijo condition. The presented numerical experiments illustrate the efficiency of the proposed AA-L-BFGS, providing empirical evidence that it significantly accelerates the convergence of the training process.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. 1.

    Anastasiadis AD, Magoulas GD, Vrahatis MN (2005) New globally convergent training scheme based on the resilient propagation algorithm. Neurocomputing 64:253–270

  2. 2.

    Awan SM, Aslam M, Khan ZA, Saeed H (2014) An efficient model based on artificial bee colony optimization algorithm with neural networks for electric load forecasting. Neural Comput Appl 25(7–8):1967–1978

  3. 3.

    Azar AT (2013) Fast neural network learning algorithms for medical applications. Neural Comput Appl 23(3–4):1019–1034

  4. 4.

    Badem H, Basturk A, Caliskan A, Yuksel ME (2017) A new efficient training strategy for deep neural networks by hybridization of artificial bee colony and limited-memory BFGS optimization algorithms. Neurocomputing 266:506–526

  5. 5.

    Bilski K, Smolag J, Galushkin AI (2014) The parallel approach to the conjugate gradient learning algorithm for the feedforward neural networks. In: International conference on artificial intelligence and soft computing. Springer, pp 12–21

  6. 6.

    Demertzis K, Iliadis L (2015) Intelligent bio-inspired detection of food borne pathogen by DNA barcodes: the case of invasive fish species Lagocephalus Sceleratus. In: International conference on engineering applications of neural networks. Springer, pp 89–99

  7. 7.

    Dolan E, Moré JJ (2002) Benchmarking optimization software with performance profiles. Math Program 91:201–213

  8. 8.

    Dua D, Taniskidou EK (2017) UCI machine learning repository

  9. 9.

    Erzin Y, Gul TO (2014) The use of neural networks for the prediction of the settlement of one-way footings on cohesionless soils based on standard penetration test. Neural Comput Appl 24(3–4):891–900

  10. 10.

    Facchinei F, Júdice J, Soares J (1998) An active set Newton algorithm for large-scale nonlinear programs with box constraints. SIAM J Optim 8(1):158–186

  11. 11.

    Gatys LA, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. In: Proceedings of the 28th international conference on neural information processing systems, vol 1, pp 262–270

  12. 12.

    Hertz J, Krogh A, Palmer R (1991) Introduction to the theory of neural computation. Addison-Wesley, Reading

  13. 13.

    Horton P, Nakai K (1996) A probabilistic classification system for predicting the cellular localization sites of proteins. Ismb 4:109–115

  14. 14.

    Iliadis L, Mansfield SD, Avramidis S, El-Kassaby YA (2013) Predicting Douglas-fir wood density by artificial neural networks (ANN) based on progeny testing information. Holzforschung 67(7):771–777

  15. 15.

    Karras DA, Perantonis SJ (1995) An efficient constrained training algorithm for feedforward networks. IEEE Trans Neural Netw 6(6):1420–1434

  16. 16.

    Kayaer K, Yıldırım T (2003) Medical diagnosis on Pima Indian diabetes using general regression neural networks. In: Proceedings of the international conference on artificial neural networks and neural information processing (ICANN/ICONIP), pp 181–184

  17. 17.

    Khadse CB, Chaudhari MA, Borghate VB (2016) Conjugate gradient back-propagation based artificial neural network for real time power quality assessment. Int J Electr Power Energy Syst 82:197–206

  18. 18.

    Kostić S, Vasović D (2015) Prediction model for compressive strength of basic concrete mixture using artificial neural networks. Neural Comput Appl 26(5):1005–1024

  19. 19.

    Li F, Zhang X, Zhang X, Du C, Xu Y, Tian YC (2018) Cost-sensitive and hybrid-attribute measure multi-decision tree over imbalanced data sets. Inf Sci 422:242–256

  20. 20.

    Liu DC, Nocedal J (1989) On the limited memory BFGS method for large scale optimization. Math Program 45(1–3):503–528

  21. 21.

    Livieris IE (2018) Improving the classification efficiency of an ANN utilizing a new training methodology. Informatics 6(1):1

  22. 22.

    Livieris IE (2019) Forecasting economy-related data utilizing constrained recurrent neural networks. Algorithms 12(85):1

  23. 23.

    Livieris IE, Pintelas P (2013) A new conjugate gradient algorithm for training neural networks based on a modified secant equation. Appl Math Comput 221:491–502

  24. 24.

    Livieris IE, Pintelas P (2015) A new class of nonmonotone conjugate gradient training algorithms. Appl Math Comput 266:404–413

  25. 25.

    Livieris IE, Pintelas P (2019) An adaptive nonmonotone active set -weight constrained- neural network training algorithm. Neurocomputing. https://doi.org/10.1016/j.neucom.2019.06.033

  26. 26.

    Livieris IE, Pintelas P (2019) An improved weight-constrained neural network training algorithm. Neural Comput Appl. https://doi.org/10.1007/s00521-019-04342-2

  27. 27.

    Maren AJ, Harston CT, Pap RM (2014) Handbook of neural computing applications. Academic Press, Cambridge

  28. 28.

    Nguyen D, Widrow B (1990) Improving the learning speed of 2-layer neural network by choosing initial values of adaptive weights. Biol Cybern 59:71–113

  29. 29.

    Noordewier MO, Towell GG, Shavlik JW (1990) Training knowledge-based neural networks to recognize genes in DNA sequences. In: Proceedings of the 3rd international conference on neural information processing systems, pp 530–536

  30. 30.

    Perantonis SJ, Karras DA (1995) An efficient constrained learning algorithm with momentum acceleration. Neural Netw 8(2):237–249

  31. 31.

    Prechelt L (1994) PROBEN1-A set of benchmarks and benchmarking rules for neural network training algorithms. Technical report 21/94, Fakultt fr Informatik, University of Karlsruhe

  32. 32.

    Shi ZJ, Wang S (2011) Modified nonmonotone Armijo line search for descent method. Numer Algorithms 57(1):1–25

  33. 33.

    Wan Zhong, Huang Shuai, Zheng Xiao Dong (2012) New cautious BFGS algorithm based on modified Armijo-type line search. J Inequal Appl 2012(1):241

  34. 34.

    Yuan G, Lu X (2011) An active set limited memory BFGS algorithm for bound constrained optimization. Appl Math Model 35:3561–3573

Download references

Author information

Correspondence to Ioannis E. Livieris.

Ethics declarations

Conflict of interest

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Livieris, I.E. An advanced active set L-BFGS algorithm for training weight-constrained neural networks. Neural Comput & Applic (2020). https://doi.org/10.1007/s00521-019-04689-6

Download citation

Keywords

  • Artificial neural networks
  • Constrained optimization
  • L-BFGS
  • Modified secant equation