In this work, a new advanced active set limited memory BFGS (Broyden–Fletcher–Goldfarb–Shanno) algorithm is proposed for efficiently training weight-constrained neural networks, called AA-L-BFGS. The proposed algorithm possesses the significant property of approximating the curvature of the error function with high-order accuracy by utilizing the theoretically advanced secant condition proposed by Livieris and Pintelas (Appl Math Comput 221:491–502, 2013). Moreover, the global convergence of the proposed algorithm is established provided that the line search satisfies the modified Armijo condition. The presented numerical experiments illustrate the efficiency of the proposed AA-L-BFGS, providing empirical evidence that it significantly accelerates the convergence of the training process.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Anastasiadis AD, Magoulas GD, Vrahatis MN (2005) New globally convergent training scheme based on the resilient propagation algorithm. Neurocomputing 64:253–270
Awan SM, Aslam M, Khan ZA, Saeed H (2014) An efficient model based on artificial bee colony optimization algorithm with neural networks for electric load forecasting. Neural Comput Appl 25(7–8):1967–1978
Azar AT (2013) Fast neural network learning algorithms for medical applications. Neural Comput Appl 23(3–4):1019–1034
Badem H, Basturk A, Caliskan A, Yuksel ME (2017) A new efficient training strategy for deep neural networks by hybridization of artificial bee colony and limited-memory BFGS optimization algorithms. Neurocomputing 266:506–526
Bilski K, Smolag J, Galushkin AI (2014) The parallel approach to the conjugate gradient learning algorithm for the feedforward neural networks. In: International conference on artificial intelligence and soft computing. Springer, pp 12–21
Demertzis K, Iliadis L (2015) Intelligent bio-inspired detection of food borne pathogen by DNA barcodes: the case of invasive fish species Lagocephalus Sceleratus. In: International conference on engineering applications of neural networks. Springer, pp 89–99
Dolan E, Moré JJ (2002) Benchmarking optimization software with performance profiles. Math Program 91:201–213
Dua D, Taniskidou EK (2017) UCI machine learning repository
Erzin Y, Gul TO (2014) The use of neural networks for the prediction of the settlement of one-way footings on cohesionless soils based on standard penetration test. Neural Comput Appl 24(3–4):891–900
Facchinei F, Júdice J, Soares J (1998) An active set Newton algorithm for large-scale nonlinear programs with box constraints. SIAM J Optim 8(1):158–186
Gatys LA, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. In: Proceedings of the 28th international conference on neural information processing systems, vol 1, pp 262–270
Hertz J, Krogh A, Palmer R (1991) Introduction to the theory of neural computation. Addison-Wesley, Reading
Horton P, Nakai K (1996) A probabilistic classification system for predicting the cellular localization sites of proteins. Ismb 4:109–115
Iliadis L, Mansfield SD, Avramidis S, El-Kassaby YA (2013) Predicting Douglas-fir wood density by artificial neural networks (ANN) based on progeny testing information. Holzforschung 67(7):771–777
Karras DA, Perantonis SJ (1995) An efficient constrained training algorithm for feedforward networks. IEEE Trans Neural Netw 6(6):1420–1434
Kayaer K, Yıldırım T (2003) Medical diagnosis on Pima Indian diabetes using general regression neural networks. In: Proceedings of the international conference on artificial neural networks and neural information processing (ICANN/ICONIP), pp 181–184
Khadse CB, Chaudhari MA, Borghate VB (2016) Conjugate gradient back-propagation based artificial neural network for real time power quality assessment. Int J Electr Power Energy Syst 82:197–206
Kostić S, Vasović D (2015) Prediction model for compressive strength of basic concrete mixture using artificial neural networks. Neural Comput Appl 26(5):1005–1024
Li F, Zhang X, Zhang X, Du C, Xu Y, Tian YC (2018) Cost-sensitive and hybrid-attribute measure multi-decision tree over imbalanced data sets. Inf Sci 422:242–256
Liu DC, Nocedal J (1989) On the limited memory BFGS method for large scale optimization. Math Program 45(1–3):503–528
Livieris IE (2018) Improving the classification efficiency of an ANN utilizing a new training methodology. Informatics 6(1):1
Livieris IE (2019) Forecasting economy-related data utilizing constrained recurrent neural networks. Algorithms 12(85):1
Livieris IE, Pintelas P (2013) A new conjugate gradient algorithm for training neural networks based on a modified secant equation. Appl Math Comput 221:491–502
Livieris IE, Pintelas P (2015) A new class of nonmonotone conjugate gradient training algorithms. Appl Math Comput 266:404–413
Livieris IE, Pintelas P (2019) An adaptive nonmonotone active set -weight constrained- neural network training algorithm. Neurocomputing. https://doi.org/10.1016/j.neucom.2019.06.033
Livieris IE, Pintelas P (2019) An improved weight-constrained neural network training algorithm. Neural Comput Appl. https://doi.org/10.1007/s00521-019-04342-2
Maren AJ, Harston CT, Pap RM (2014) Handbook of neural computing applications. Academic Press, Cambridge
Nguyen D, Widrow B (1990) Improving the learning speed of 2-layer neural network by choosing initial values of adaptive weights. Biol Cybern 59:71–113
Noordewier MO, Towell GG, Shavlik JW (1990) Training knowledge-based neural networks to recognize genes in DNA sequences. In: Proceedings of the 3rd international conference on neural information processing systems, pp 530–536
Perantonis SJ, Karras DA (1995) An efficient constrained learning algorithm with momentum acceleration. Neural Netw 8(2):237–249
Prechelt L (1994) PROBEN1-A set of benchmarks and benchmarking rules for neural network training algorithms. Technical report 21/94, Fakultt fr Informatik, University of Karlsruhe
Shi ZJ, Wang S (2011) Modified nonmonotone Armijo line search for descent method. Numer Algorithms 57(1):1–25
Wan Zhong, Huang Shuai, Zheng Xiao Dong (2012) New cautious BFGS algorithm based on modified Armijo-type line search. J Inequal Appl 2012(1):241
Yuan G, Lu X (2011) An active set limited memory BFGS algorithm for bound constrained optimization. Appl Math Model 35:3561–3573
Conflict of interest
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Livieris, I.E. An advanced active set L-BFGS algorithm for training weight-constrained neural networks. Neural Comput & Applic (2020). https://doi.org/10.1007/s00521-019-04689-6
- Artificial neural networks
- Constrained optimization
- Modified secant equation