Abstract
This paper introduces conjugate gradient algorithms for training quaternion-valued feedforward neural networks. Because these algorithms had better performance than the gradient descent algorithm in the real- and complex-valued cases, the extension to the quaternion-valued case was a natural idea. The classical variants of the conjugate gradient algorithm are deduced starting from their real-valued variants, and using the framework of the HR calculus. The resulting quaternion-valued training methods are exemplified on time series prediction applications, showing a significant improvement over the quaternion gradient descent algorithm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arena, P., Fortuna, L., Muscato, G., Xibilia, M.: Multilayer perceptrons to approximate quaternion valued functions. Neural Netw. 10(2), 335–342 (1997)
Arena, P., Fortuna, L., Muscato, G., Xibilia, M.: Neural Networks in Multidimensional Domains Fundamentals and New Trends in Modelling and Control. Lecture Notes in Control and Information Sciences, vol. 234. Springer, London (1998)
Beale, E.: A derivation of conjugate gradients. In: Lootsma, F.A. (ed.) Numerical Methods for Nonlinear Optimization, pp. 39–43. Academic Press, London (1972)
Bishop, C.: Neural Networks for Pattern Recognition. Oxford University Press Inc., New York (1995)
Buchholz, S., Le Bihan, N.: Polarized signal classification by complex and quaternionic multi-layer perceptrons. Int. J. Neural Syst. 18(2), 75–85 (2008)
Charalambous, C.: Conjugate gradient algorithm for efficient training of artificial neural networks. IEE Proc. G Circuits Devices Syst. 139(3), 301–310 (1992)
Ujang, C.B., Took, C., Mandic, D.: Split quaternion nonlinear adaptive filtering. Neural Netw. 23(3), 426–434 (2010)
Ujang, C.B., Took, C., Mandic, D.: Quaternion-valued nonlinear adaptive filtering. IEEE Trans. Neural Netw. 22(8), 1193–1206 (2011)
Hestenes, M., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Res. Natl. Bur. Stand. 49(6), 409–436 (1952)
Isokawa, T., Kusakabe, T., Matsui, N., Peper, F.: Quaternion neural network and its application. In: Palade, V., Howlett, R.J., Jain, L. (eds.) KES 2003. LNCS (LNAI), vol. 2774, pp. 318–324. Springer, Heidelberg (2003). doi:10.1007/978-3-540-45226-3_44
Jahanchahi, C., Took, C., Mandic, D.: On HR calculus, quaternion valued stochastic gradient, and adaptive three dimensional wind forecasting. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–5. IEEE, July 2010
Johansson, E., Dowla, F., Goodman, D.: Backpropagation learning for multilayer feed-forward neural networks using the conjugate gradient method. Int. J. Neural Syst. 2(4), 291–301 (1991)
Kusamichi, H., Isokawa, T., Matsui, N., Ogawa, Y., Maeda, K.: A new scheme for color night vision by quaternion neural network. In: International Conference on Autonomous Robots and Agents, pp. 101–106, December 2004
Luenberger, D., Ye, Y.: Linear and Nonlinear Programming. International Series in Operations Research & Management Science, vol. 116. Springer, Heidelberg (2008)
Mandic, D., Chambers, J.: Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability. Wiley, New York (2001)
Polak, E., Ribiere, G.: Note sur la convergence de méthodes de directions conjuguées. Revue Française d’Informatique et de Recherche Opérationnelle 3(16), 35–43 (1969)
Popa, C.-A.: Conjugate gradient algorithms for complex-valued neural networks. In: Arik, S., Huang, T., Lai, W.K., Liu, Q. (eds.) ICONIP 2015. LNCS, vol. 9490, pp. 412–422. Springer, Cham (2015). doi:10.1007/978-3-319-26535-3_47
Powell, M.: Restart procedures for the conjugate gradient method. Math. Program. 12(1), 241–254 (1977)
Reeves, C., Fletcher, R.: Function minimization by conjugate gradients. Comput. J. 7(2), 149–154 (1964)
Took, C., Mandic, D.: The quaternion lms algorithm for adaptive filtering of hypercomplex processes. IEEE Trans. Sig. Process. 57(4), 1316–1327 (2009)
Took, C., Mandic, D.: Quaternion-valued stochastic gradient-based adaptive IIR filtering. IEEE Trans. Sig. Process. 58(7), 3895–3901 (2010)
Took, C., Mandic, D.: A quaternion widely linear adaptive filter. IEEE Trans. Sig. Process. 58(8), 4427–4431 (2010)
Took, C., Mandic, D., Aihara, K.: Quaternion-valued short term forecasting of wind profile. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–6. IEEE, July 2010
Took, C., Strbac, G., Aihara, K., Mandic, D.: Quaternion-valued short-term joint forecasting of three-dimensional wind and atmospheric parameters. Renewable Energy 36(6), 1754–1760 (2011)
Xia, Y., Jahanchahi, C., Mandic, D.: Quaternion-valued echo state networks. IEEE Trans. Neural Netw. Learn. Syst. 26(4), 663–673 (2015)
Xu, D., Xia, Y., Mandic, D.: Optimization in quaternion dynamic systems: gradient, Hessian, and learning algorithms. IEEE Trans. Neural Netw. Learn. Syst. 27(2), 249–261 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Popa, CA. (2017). Conjugate Gradient Algorithms for Quaternion-Valued Neural Networks. In: Matoušek, R. (eds) Recent Advances in Soft Computing. ICSC-MENDEL 2016. Advances in Intelligent Systems and Computing, vol 576. Springer, Cham. https://doi.org/10.1007/978-3-319-58088-3_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-58088-3_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-58087-6
Online ISBN: 978-3-319-58088-3
eBook Packages: EngineeringEngineering (R0)