Advertisement

A survey of quaternion neural networks

  • Titouan ParcolletEmail author
  • Mohamed Morchid
  • Georges Linarès
Article

Abstract

Quaternion neural networks have recently received an increasing interest due to noticeable improvements over real-valued neural networks on real world tasks such as image, speech and signal processing. The extension of quaternion numbers to neural architectures reached state-of-the-art performances with a reduction of the number of neural parameters. This survey provides a review of past and recent research on quaternion neural networks and their applications in different domains. The paper details methods, algorithms and applications for each quaternion-valued neural networks proposed.

Keywords

Hypercomplex numbers Quaternion neural networks Deep Learning 

Abbreviations

ML

Machine learning

AI

Artificial intelligence

(R, G, B)

Red, green, blue

Q{Model}

Quaternion{Model}

CVNN

Complex-valued neural network

NN

Neural network

MLP

Multilayer perceptron

DNN

Deep neural network

RNN

Recurrent neural network

CNN

Convolutional neural network

DAE

Denoising autoencoder

CAE

Convolutional autoencoder

HNN

Hopfield neural network

SVM

Support vector machine

PCA

Principal component analysis

LDA

Latent Dirichlet allocation

ReLU

Rectified linear unit

tanh

Hyperbolic tangent

eLU

Exponential linear unit

CRF

Cauthy–Riemann–Fueter

MSE

Mean squared error

GAN

Gaussian angular noise

PSNR

Peak signal to noise ratio

ABr

Average brightness

HOG

Histograms oriented gradient

PolSAR

Polarimetric synthetic aperture radar

CCS

Customer care service

Notes

References

  1. Adavanne S, Politis A, Nikunen J, Virtanen T (2018) Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE J Sel Top Signal Process 13:34–48 Google Scholar
  2. Aizenberg IN, Gonzalez A (2018) Image recognition using MLMVN and frequency domain features. In: 2018 International joint conference on neural networks (IJCNN), pp 1–8Google Scholar
  3. Aizenberg I, Alexander S, Jackson J (2011) Recognition of blurred images using multilayer neural network based on multi-valued neurons. In: 2011 41st IEEE International symposium on multiple-valued logic. IEEE, pp 282–287Google Scholar
  4. Arena P, Fortuna L, Re R, Xibilia MG (1993) On the capability of neural networks with complex neurons in complex valued functions approximation. In: 1993 IEEE International symposium on circuits and systems, ISCAS’93. IEEE, pp 2168–2171Google Scholar
  5. Arena P, Fortuna L, Occhipinti L, Xibilia MG (1994) Neural networks for quaternion-valued function approximation. In: 1994 IEEE International symposium on circuits and systems, ISCAS’94, vol 6. IEEE, pp 307–310Google Scholar
  6. Arena P, Fortuna L, Muscato G, Xibilia MG (1997) Multilayer perceptrons to approximate quaternion valued functions. Neural Netw 10(2):335–342Google Scholar
  7. Bayro-Corrochano E, Lechuga-Gutiérrez L, Garza-Burgos M (2018) Geometric techniques for robotics and hmi: Interpolation and haptics in conformal geometric algebra and control using quaternion spike neural networks. Robot Auton Syst 104:72–84Google Scholar
  8. Bechet F, Maza B, Bigouroux N, Bazillon T, El-Beze M, De Mori R, Arbillot E (2012) Decoda: a call-centre human–human spoken conversation corpus. In: LREC, pp 1343–1347Google Scholar
  9. Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3(Jan):993–1022zbMATHGoogle Scholar
  10. Buchholz S, Sommer G (2000) Quaternionic spinor MLP. CiteSeer, PrincetonGoogle Scholar
  11. Buchholz S, Le Bihan N (2006) Optimal separation of polarized signals by quaternionic neural networks. In: 2006 14th European signal processing conference. IEEE, pp 1–5Google Scholar
  12. Chou JC (1992) Quaternion kinematic and dynamic differential equations. IEEE Trans Robot Autom 8(1):53–64Google Scholar
  13. Comminiello D, Lella M, Scardapane S, Uncini A (2018) Quaternion convolutional neural networks for detection and localization of 3D sound events. arXiv:181206811
  14. Cui Y, Takahashi K, Hashimoto M (2013) Design of control systems using quaternion neural network and its application to inverse kinematics of robot manipulator. In: 2013 IEEE/SICE International symposium on system integration (SII). IEEE, pp 527–532Google Scholar
  15. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Computer society conference on computer vision and pattern recognition, CVPR 2005, vol 1. IEEE, pp 886–893Google Scholar
  16. De Boer PT, Kroese DP, Mannor S, Rubinstein RY (2005) A tutorial on the cross-entropy method. Ann Oper Res 134(1):19–67MathSciNetzbMATHGoogle Scholar
  17. De Leo S, Rotelli P (1997) Local hypercomplex analyticity. arXiv preprint arXiv:9703002 [funct-an]
  18. Diebel J (2006) Representing attitude: Euler angles, unit quaternions, and rotation vectors. Matrix 58(15–16):1–35Google Scholar
  19. Dornaika F, Horaud R (1998) Simultaneous robot-world and hand-eye calibration. IEEE Trans Robot Autom 14(4):617–622Google Scholar
  20. Fortuna L, Muscato G, Xibilia M (1996) An hypercomplex neural network platform for robot positioning. In: 1996 IEEE International symposium on circuits and systems, ISCAS’96. Connecting the World, vol 3. IEEE, pp 609–612Google Scholar
  21. Fortuna L, Muscato G, Xibilia MG (2001) A comparison between hmlp and hrbf for attitude control. IEEE Trans Neural Netw 12(2):318–328Google Scholar
  22. Garofolo JS, Lamel LF, Fisher WM, Fiscus JG, Pallett DS (1993) Darpa timit acoustic-phonetic continous speech corpus CD-ROM. NIST speech disc 1-1.1. NASA STI/Recon technical report no. 93Google Scholar
  23. Gaudet CJ, Maida AS (2018) Deep quaternion networks. In: 2018 International joint conference on neural networks (IJCNN). IEEE, pp 1–8Google Scholar
  24. Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: the KITTI dataset. Int J Robot Res (IJRR) 32:1231–1237Google Scholar
  25. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 249–256Google Scholar
  26. Graves A, Mohamed Ar, Hinton G (2013) Speech recognition with deep recurrent neural networks. In: 2013 IEEE International conference on acoustics, speech and signal processing. IEEE, pp 6645–6649Google Scholar
  27. Greenblatt A, Mosquera-Lopez C, Agaian S (2013) Quaternion neural networks applied to prostate cancer gleason grading. In: 2013 IEEE International conference on systems, man, and cybernetics (SMC). IEEE, pp 1144–1149Google Scholar
  28. Hamilton WR (1844) Ii. on quaternions; or on a new system of imaginaries in algebra. Lond Edinb Dublin Philos Mag J Sci 25(163):10–13Google Scholar
  29. Hearst MA, Dumais ST, Osuna E, Platt J, Scholkopf B (1998) Support vector machines. IEEE Intell Syst Appl 13(4):18–28Google Scholar
  30. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp 1026–1034Google Scholar
  31. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778Google Scholar
  32. Higham NJ (1990) Analysis of the Cholesky decomposition of a semi-definite matrix. Oxford University Press, OxfordzbMATHGoogle Scholar
  33. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507MathSciNetzbMATHGoogle Scholar
  34. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554MathSciNetzbMATHGoogle Scholar
  35. Hinton G, Deng L, Yu D, Dahl GE, Mohamed A, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97Google Scholar
  36. Hirose A (2012) Complex-valued neural networks, vol 400. Springer, BerlinzbMATHGoogle Scholar
  37. Hitzer EM (2007) Quaternion fourier transform on quaternion fields and generalizations. Adv Appl Clifford Algebras 17(3):497–517MathSciNetzbMATHGoogle Scholar
  38. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780Google Scholar
  39. Hopfield JJ, Tank DW (1985) “Neural” computation of decisions in optimization problems. Biol Cybern 52(3):141–152zbMATHGoogle Scholar
  40. Huang FJ, LeCun Y (2006) Large-scale learning with SVM and convolutional for generic object categorization. In: null. IEEE, pp 284–291Google Scholar
  41. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:150203167
  42. Isokawa T, Kusakabe T, Matsui N, Peper F (2003) Quaternion neural network and its application. In: International conference on knowledge-based and intelligent information and engineering systems. Springer, pp 318–324Google Scholar
  43. Isokawa T, Nishimura H, Kamiura N, Matsui N (2006) Fundamental properties of quaternionic hopfield neural network. In: 2006 International joint conference on neural networks, IJCNN’06. IEEE, pp 218–223Google Scholar
  44. Isokawa T, Nishimura H, Kamiura N, Matsui N (2008) Associative memory in quaternionic hopfield neural network. Int J Neural Syst 18(02):135–145Google Scholar
  45. Isokawa T, Matsui N, Nishimura H (2009) Quaternionic neural networks: fundamental properties and applications. In: Complex-valued neural networks: utilizing high-dimensional parameters. IGI global. pp 411–439 Google Scholar
  46. Isokawa T, Nishimura H, Matsui N (2012) Quaternionic multilayer perceptron with local analyticity. Information 3(4):756–770Google Scholar
  47. Jolliffe I (2011) Principal component analysis. In: Lovric M (ed) International encyclopedia of statistical science. Springer, Berlin, pp 1094–1096 Google Scholar
  48. Karney CF (2007) Quaternions in molecular modeling. J Mol Graph Model 25(5):595–604Google Scholar
  49. Kinugawa K, Shang F, Usami N, Hirose A (2018) Isotropization of quaternion-neural-network-based PolSAR adaptive land classification in Poincare-sphere parameter space. IEEE Geosci Remote Sens Lett 15:1234–1238Google Scholar
  50. Kobayashi M (2015) Hybrid quaternionic hopfield neural network. IEICE Trans Fundam Electron Commun Comput Sci 98(7):1512–1518Google Scholar
  51. Kobayashi M, Nakajima A (2012) Twisted quaternary neural networks. IEEJ Trans Electr Electron Eng 7(4):397–401Google Scholar
  52. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  53. Krizhevsky A, Nair V, Hinton G (2014) The cifar-10 dataset. http://www.cs.toronto.edu/kriz/cifar html
  54. Kusamichi H, Isokawa T, Matsui N, Ogawa Y, Maeda K (2004) A new scheme for color night vision by quaternion neural network. In: Proceedings of the 2nd international conference on autonomous robots and agents, vol 1315. CiteseerGoogle Scholar
  55. Lin JS, Cheng KS, Mao CW (1996) A fuzzy hopfield neural network for medical image segmentation. IEEE Trans Nucl Sci 43(4):2389–2398Google Scholar
  56. Mandic DP, Goh VSL (2009) Complex valued nonlinear adaptive filters: noncircularity, widely linear and neural models, vol 59. Wiley, New YorkGoogle Scholar
  57. Mandic DP, Jahanchahi C, Took CC (2011) A quaternion gradient operator and its applications. IEEE Signal Process Lett 18(1):47–50Google Scholar
  58. Matsui N, Isokawa T, Kusamichi H, Peper F, Nishimura H (2004) Quaternion neural network with geometrical operators. J Intell Fuzzy Syst 15(3, 4):149–164zbMATHGoogle Scholar
  59. Mikolov T, Karafiát M, Burget L, Černockỳ J, Khudanpur S (2010) Recurrent neural network based language model. In: Eleventh annual conference of the international speech communication associationGoogle Scholar
  60. Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp 807–814Google Scholar
  61. Nitta T (1995) A quaternary version of the back-propagation algorithm. In: IEEE International conference on neural networks, 1995. Proceedings, vol 5. IEEE, pp 2753–2756Google Scholar
  62. Nitta T (2004) A solution to the 4-bit parity problem with a single quaternary neuron. Neural Inf Process Lett Rev 5(2):33–39Google Scholar
  63. Nitta T, de Garis H (1992) A 3D vector version of the back-propagation algorithm. In: Proceedings of international joint conference on neural networks, pp 511–516Google Scholar
  64. Ogawa T (2016) Neural network inversion for multilayer quaternion neural networks. Comput Technol Appl 7:73–82Google Scholar
  65. Parcollet T, Morchid M, Bousquet PM, Dufour R, Linarès G, De Mori R (2016) Quaternion neural networks for spoken language understanding. In: 2016 IEEE Spoken language technology workshop (SLT). IEEE, pp 362–368Google Scholar
  66. Parcollet T, Morchid M, Linares G (2017a) Deep quaternion neural networks for spoken language understanding. In: 2017 IEEE Automatic speech recognition and understanding workshop (ASRU). IEEE, pp 504–511Google Scholar
  67. Parcollet T, Morchid M, Linares G (2017b) Quaternion denoising encoder–decoder for theme identification of telephone conversations. Proceedings of Interspeech 2017, pp 3325–3328Google Scholar
  68. Parcollet T, Morchid M, Linarès G (2018a) Quaternion convolutional neural networks for heterogeneous image processing. arXiv preprint arXiv:181102656
  69. Parcollet T, Ravanelli M, Morchid M, Linarès G, Trabelsi C, Mori RD, Bengio Y (2018b) Quaternion recurrent neural networks. arXiv preprint arXiv:1806.04418
  70. Parcollet T, Zhang Y, Morchid M, Trabelsi C, Linarès G, de Mori R, Bengio Y (2018c) Quaternion convolutional neural networks for end-to-end automatic speech recognition. In: Interspeech 2018, 19th Annual conference of the international speech communication association, Hyderabad, India, 2–6 September 2018, pp 22–26. https://doi.org/10.21437/Interspeech.2018-1898
  71. Pascanu R, Mikolov T, Bengio Y (2013) On the difficulty of training recurrent neural networks. In: International conference on machine learning, pp 1310–1318Google Scholar
  72. Platt J, et al. (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in large margin classifiers. MIT Press, pp 61–74 Google Scholar
  73. Pletinckx D (1989) Quaternion calculus as a basic tool in computer graphics. Vis Comput 5(1–2):2–13zbMATHGoogle Scholar
  74. Popa CA (2018) Learning algorithms for quaternion-valued neural networks. Neural Process Lett 47(3):949–973Google Scholar
  75. Sangwine SJ (1996) Fourier transforms of colour images using quaternion or hypercomplex, numbers. Electron Lett 32(21):1979–1980Google Scholar
  76. Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681Google Scholar
  77. Shang F, Hirose A (2014) Quaternion neural-network-based PolSAR land classification in poincare-sphere-parameter space. IEEE Trans Geosci Remote Sensing 52(9):5693–5703Google Scholar
  78. Shoemake K (1985) Animating rotation with quaternion curves. In: ACM SIGGRAPH computer graphics, vol 19. ACM, pp 245–254Google Scholar
  79. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556
  80. Soulard R, Carré P (2011) Quaternionic wavelets for texture classification. Pattern Recognit Lett 32(13):1669–1678Google Scholar
  81. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958MathSciNetzbMATHGoogle Scholar
  82. Takahashi K, Takahashi S, Cui Y, Hashimoto M (2014) Remarks on computational facial expression recognition from HOG features using quaternion multi-layer neural network. In: International conference on engineering applications of neural networks. Springer, pp 15–24Google Scholar
  83. Takahashi K, Isaka A, Fudaba T, Hashimoto M (2017) Remarks on quaternion neural network-based controller trained by feedback error learning. In: 2017 IEEE/SICE International symposium on system integration (SII), pp 875–880Google Scholar
  84. Tokuda, K., Zen, H., Kitamura, T. (2003) Trajectory modeling based on HMMs with the explicit relationship between static and dynamic features. In Eighth European conference on speech communication and technologyGoogle Scholar
  85. Trabelsi C, Bilaniuk O, Zhang Y, Serdyuk D, Subramanian S, Santos JF, Mehri S, Rostamzadeh N, Bengio Y, Pal CJ (2017) Deep complex networks. arXiv preprint arXiv:170509792
  86. Ujang BC, Jahanchahi C, Took CC, Mandic D (2010) Quaternion valued neural networks and nonlinear adaptive filters.Google Scholar
  87. Ujang BC, Took CC, Mandic DP (2011) Quaternion-valued nonlinear adaptive filtering. IEEE Trans Neural Netw 22(8):1193–1206Google Scholar
  88. Valle ME, de Castro FZ (2018) On the dynamics of hopfield neural networks on unit quaternions. IEEE Trans Neural Netw Learn Syst 29(6):2464–2471Google Scholar
  89. Vincent P, Larochelle H, Bengio Y, Manzagol PA (2008) Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th international conference on machine learning. ACM, pp 1096–1103Google Scholar
  90. Willmott CJ, Matsuura K (2005) Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Clim Res 30(1):79–82Google Scholar
  91. Xu D, Jahanchahi C, Took CC, Mandic DP (2015) Enabling quaternion derivatives: the generalized HR calculus. R Soc Open Sci 2(8):150255MathSciNetGoogle Scholar
  92. Xu D, Zhang L, Zhang H (2017) Learning algorithms in quaternion neural networks using GHR calculus. Neural Netw World 27(3):271Google Scholar
  93. Yoshida M, Kuroe Y, Mori T (2005) Models of hopfield-type quaternion neural networks and their energy functions. Int J Neural Syst 15:129–135Google Scholar
  94. Yun X, Bachmann ER (2006) Design, implementation, and experimental results of a quaternion-based kalman filter for human body motion tracking. IEEE Trans Robot 22(6):1216–1227Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Laboratoire Informatique d’Avignon (LIA)Université d’AvignonAvignonFrance
  2. 2.ORKISAix-en-ProvenceFrance

Personalised recommendations