Advertisement

RMIL/AG: A New Class of Nonlinear Conjugate Gradient for Training Back Propagation Algorithm

  • Sri Mazura Muhammad Basri
  • Nazri Mohd Nawi
  • Mustafa Mamat
  • Norhamreeza Abdul Hamid
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 700)

Abstract

The conventional back propagation (BP) algorithm is generally known for some disadvantages, such as slow training, easy to getting trapped into local minima and being sensitive to the initial weights and bias. This paper introduced a new class of efficient second order conjugate gradient (CG) for training BP called Rivaie, Mustafa, Ismail and Leong (RMIL)/AG. The RMIL uses the value of adaptive gain parameter in the activation function to modify the gradient based search direction. The efficiency of the proposed method is verified by means of simulation on four classification problems. The results show that the computational efficiency of the proposed method was better than the conventional BP algorithm.

Keywords

Second order method Back-propagation Conjugate gradient Search direction Classification 

Notes

Acknowledgements

The authors would like to thank Universiti Tun Hussein Onn Malaysia (UTHM) Ministry of Higher Education (MOHE) Malaysia for financially supporting this Research under Trans-disciplinary Research Grant Scheme (TRGS) vote no. T003. This research also supported by GATES IT Solution Sdn. Bhd under its publication scheme.

References

  1. 1.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: David, E.R., James, L.M., C.P.R. Group (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, pp. 318–362. MIT Press (1986)Google Scholar
  2. 2.
    Zhang, S.L., Chang, T.C.: A study of image classification of remote sensing based on back-propagation neural network with extended delta bar delta. Math. Problems Eng. 2015, 10 (2016)Google Scholar
  3. 3.
    Nawi, N.M., Rehman, M., Khan, A.: WS-BP: an efficient wolf search based back-propagation algorithm. In: International Conference on Mathematics, Engineering and Industrial Applications 2014 (ICoMEIA 2014). AIP Publishing (2015)Google Scholar
  4. 4.
    Rehman, M.Z., Nawi, N.M.: The effect of adaptive momentum in improving the accuracy of gradient descent back propagation algorithm on classification problems. In: Mohamad Zain, J., Wan Mohd, W.M.B., El-Qawasmeh, E. (eds.) Second International Conference on Software Engineering and Computer Systems, ICSECS 2011, Kuantan, Pahang, Malaysia, 27–29 June 2011, Proceedings, Part I, pp. 380–390. Springer, Berlin, Heidelberg (2011)Google Scholar
  5. 5.
    Nawi, N.M., Khan, A., Rehman, M.Z.: A new back-propagation neural network optimized with cuckoo search algorithm. In: International Conference on Computational Science and Its Applications. Springer, Berlin, Heidelberg (2013)Google Scholar
  6. 6.
    Liu, Y., Jing, W., Xu, L.: Parallelizing backpropagation neural network using MapReduce and cascading model. Comput. Intell. Neurosci. 2016, 11 (2016)Google Scholar
  7. 7.
    Chen, Y., et al.: Three-dimensional short-term prediction model of dissolved oxygen content based on PSO-BPANN algorithm coupled with Kriging interpolation. Math. Problems Eng. 2016, 10 (2016)Google Scholar
  8. 8.
    Cao, J., Chen, J., Li, H.: An adaboost-backpropagation neural network for automated image sentiment classification. Sci. World J. 2014, 9 (2014)Google Scholar
  9. 9.
    Abdul Hamid, N., et al.: A review on improvement of back propagation algorithm. Glob. J. Technol. 1 (2012)Google Scholar
  10. 10.
    Van Ooyen, A., Nienhuis, B.: Improving the convergence of the back-propagation algorithm. Neural Netw. 5(3), 465–471 (1992)CrossRefGoogle Scholar
  11. 11.
    Haykin, S.: Neural Networks: A Comprehensive Foundation, 842 pp. Prentice Hall PTR (1998)Google Scholar
  12. 12.
    Kumar, P., Merchant, S.N., Desai, U.B.: Improving performance in pulse radar detection using Bayesian regularization for neural network training. Digit. Signal Proc. 14(5), 438–448 (2004)CrossRefGoogle Scholar
  13. 13.
    Bishop, C.M.: Neural Networks for Pattern Recognition, 482 pp. Oxford University Press, Inc. (1995)Google Scholar
  14. 14.
    Fletcher, R., Powell, M.J.D.: A rapidly convergent descent method for minimization. Comput. J. 6(2), 163–168 (1963)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Fletcher, R., Reeves, C.M.: Function minimization by conjugate gradients. Comput. J. 7(2), 149–154 (1964)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Hestenes, M., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Res. Natl. Bureau Stand. 49(6), 409–436 (1952)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Maier, H.R., Dandy, G.C.: The effect of internal parameters and geometry on the performance of back-propagation neural networks: an empirical study. Environ. Model Softw. 13(2), 193–209 (1998)CrossRefGoogle Scholar
  18. 18.
    Thimm, G., Moerland, P., Fiesler, E.: The interchangeability of learning rate and gain in backpropagation neural networks. Neural Comput. 8(2), 451–460 (1996)CrossRefGoogle Scholar
  19. 19.
    Nawi, N.M., Ransing, R., Hamid, N.A.: BPGD-AG: a new improvement of back-propagation neural network learning algorithms with adaptive gain. J. Sci. Technol. 2(2) (2010)Google Scholar
  20. 20.
    Nawi, N.M., et al.: An improved back propagation neural network algorithm on classification problems. In: Zhang, Y., et al. (eds.) Database Theory and Application, Bio-Science and Bio-Technology: International Conferences, DTA and BSBT 2010, Held as Part of the Future Generation Information Technology Conference, FGIT 2010, Jeju Island, Korea, 13–15 December 2010, Proceedings, pp. 177–188. Springer, Berlin, Heidelberg (2010)Google Scholar
  21. 21.
    Nawi, N.M., et al.: Enhancing back propagation neural network algorithm with adaptive gain on classification problems. Networks 4(2) (2011)Google Scholar
  22. 22.
    Nawi, N.M., et al.: Predicting patients with heart disease by using an improved back-propagation algorithm. J. Comput. 3(2) (2011)Google Scholar
  23. 23.
    Nawi, N.M., Wahid, N., Idris, M.M.: A new gradient based search direction for conjugate gradient algorithm. Int. J. Adv. Data Inf. Eng. 1(1), 1–7 (2016)Google Scholar
  24. 24.
    Abdul Hamid, N., et al.: Learning efficiency improvement of back propagation algorithm by adaptively changing gain parameter together with momentum and learning rate. In: Zain, J.M., Wan Mohd, W.M.B., El-Qawasmeh, E. (eds.) Second International Conference on Software Engineering and Computer Systems, ICSECS 2011, Kuantan, Pahang, Malaysia, 27–29 June 2011, Proceedings, Part III, pp. 812–824. Springer, Berlin, Heidelberg (2011)Google Scholar
  25. 25.
    Abdul Hamid, N., Nawi, N.M., Ghazali, R.: The effect of adaptive gain and adaptive momentum in improving training time of gradient descent back propagation algorithm on classification problems. Int. J. Adv. Sci. Eng. Inf. Technol. 1(2), 178–184 (2011)Google Scholar
  26. 26.
    Abdul Hamid, N., et al.: Improvements of back propagation algorithm performance by adaptively changing gain, momentum and learning rate. Int. J. New Comput. Archit. Appl. (IJNCAA) 1(4), 866–878 (2011)Google Scholar
  27. 27.
    Abdul Hamid, N., et al.: Solving local minima problem in back propagation algorithm using adaptive gain, adaptive momentum and adaptive learning rate on classification problems, In: International Journal of Modern Physics: Conference Series, pp. 448–455. World Scientific Publishing Company (2012)Google Scholar
  28. 28.
    Mohd Nawi, N., et al.: Predicting patients with heart disease by using an improved back-propagation algorithm. J. Comput. 3(2) (2011)Google Scholar
  29. 29.
    Nawi, N.M., Wahid, N., Idris, M.M.: A new gradient based search direction for conjugate gradient algorithm. Int. J. Adv. Data Inf. Eng. 1(1), 1–7 (2013)Google Scholar
  30. 30.
    Rivaie, M., et al.: A new class of nonlinear conjugate gradient coefficients with global convergence properties. Appl. Math. Comput. 218(22), 11323–11332 (2012)MathSciNetMATHGoogle Scholar
  31. 31.
    Angiulli, G., et al.: Accurate modelling of lossy SIW resonators using a neural network residual Kriging approach. IEICE Electron. Express 14(6), 20170073 (2017). 20170073CrossRefGoogle Scholar
  32. 32.
    RodrÃguez-QuiÃonez, J.C., et al.: Improve a 3D distance measurement accuracy in stereo vision systems using optimization methods’ approach. Opto-Electron. Rev. 25(1), 24–32 (2017)Google Scholar
  33. 33.
    Prechelt, L.: PROBEN1-A Set of Benchmarks and Benchmarking Rules for Neural Network Training Algorithms. Technical Report 21/94, Fakultat fur Informatik, Universitat Karlsruhe, /Anonymous FTP: /pub/papers/techreports/1994/1994-21 (1994)Google Scholar
  34. 34.
    Fisher, R.A.: The use of multiple measurements in taxonomic problems. Ann. Eugen. 7(2), 179–188 (1936)CrossRefGoogle Scholar
  35. 35.
    Evett, I.W., Spiehler, E.J.: Rule induction in forensic science. In: Knowledge Based Systems, pp. 152–160. Halsted Press (1988)Google Scholar
  36. 36.
    Smith, J.W., et al.: Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In: Proceedings of the Annual Symposium on Computer Application in Medical Care, pp. 261–265 (1988)Google Scholar
  37. 37.
    Mangasarian, O.L., Street, W.N., Wolberg, W.H.: Breast cancer diagnosis and prognosis via linear programming. Oper. Res. 43(4), 570–577 (1995)MathSciNetCrossRefMATHGoogle Scholar
  38. 38.
    Nawi, N.M., et al.: Second order back propagation neural network (SOBPNN) algorithm for medical data classification. In: Phon-Amnuaisuk, S., Au, T.W. (eds.) Computational Intelligence in Information Systems: Proceedings of the Fourth INNS Symposia Series on Computational Intelligence in Information Systems (INNS-CIIS 2014), pp. 73–83. Springer International Publishing, Cham (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Sri Mazura Muhammad Basri
    • 1
  • Nazri Mohd Nawi
    • 2
  • Mustafa Mamat
    • 3
  • Norhamreeza Abdul Hamid
    • 2
  1. 1.Faculty of Science, Technology and Human DevelopmentUniversiti Tun Hussein Onn MalaysiaParit Raja, Batu PahatMalaysia
  2. 2.Faculty of Computer Science and Information TechnologyUniversiti Tun Hussein Onn MalaysiaParit Raja, Batu PahatMalaysia
  3. 3.Faculty of Informatics and ComputingUniversiti Sultan Zainal Abidin (UniSZA)BesutMalaysia

Personalised recommendations