Progress in Computing, Analytics and Networking pp 695-709 | Cite as

# Higher Order Neural Network and Its Applications: A Comprehensive Survey

## Abstract

Over the years, neural networks have shown its strength in various fields of research. There is a vast improvement in the efficiency and effectiveness of various classification techniques mainly with the introduction of higher order neural networks. Due to great learning and storage capacity with grater computational ability than the existing traditional neural networks, nowadays, researchers are very much attracted toward the higher order neural network due to their nonlinear mapping ability with less number of input units. In this paper, a comprehensive survey on Pi-Sigma higher order neural network and its different applications to various domains over more than a decade has been reviewed. These techniques are vastly used in classification and regression in several domains including medical, time series forecasting, image processing, and engineering. The extensive survey provides a recent development in higher order neural network and its applications in several application domains.

## Keywords

Pi-Sigma neural network (PSNN) Jordan Pi-Sigma neural network (JPSNN) Ridge polynomial neural network (RPNN) Dynamic ridge polynomial neural network (DRPNN) Recurrent Pi-Sigma neural network (RPSNN)## References

- 1.Lippmann R.P. (1989) Pattern Classification using Neural Networks, IEEE Communications Magazine, 47–50.Google Scholar
- 2.Reid M.B. (1989) Rapid Training of Higher-order Neural Networks for Invariant Pattern Recognition, Processing of IJCNN Washington D.C, 1:689–692.Google Scholar
- 3.Ivakhnenko A.G. (1971) Polynomial theory of complex systems, IEEE transactions on Systems, Man, and Cybernetics, 1(4):364–378.Google Scholar
- 4.Giles C.L., Maxwell T. (1987) Learning, invariance, and generalization in high-order neural networks, 26:4972–4978.Google Scholar
- 5.Mats B. (1990) Higher order artificial neural networks. DIANE Publishing Company, Darby PA, USA, ISBN 0941375927.Google Scholar
- 6.Lippmann. R. P (1987) An introduction to computing with neural nets. IEEE ASSP Magazine, 4–22.Google Scholar
- 7.Shin Y., Ghosh, J. (1991) Realization of boolean functions using binary pi-sigma networks. In: Dagli, C.H., Kumara, S.R.T., Shin, Y.C. (eds.) Intelligent engineering systems through artificial neural networks, ASME Press, New York, 205–210.Google Scholar
- 8.Shin Y., Ghosh J. (1991) The Pi-Sigma Networks: An Efficient Higher-Order Neural Network for Pattern Classification and Function Approximation, In: Proceedings of International Joint Conference on Neural Networks, Seattle, WA, USA, 13–18.Google Scholar
- 9.Shin Y., Ghosh J. (1992) Approximation of multivariate functions using ridge polynomial networks, In Neural Networks, IJCNN., International Joint Conference on, 380–385.Google Scholar
- 10.Shin Y. (1992) Efficient higher-order feed forward networks for function approximation and classification (Doctoral dissertation, University of Texas at Austin).Google Scholar
- 11.Ghosh J., Shin Y. (1992) Efficient higher-order neural networks for classification and function approximation, Int J Neural Syst, 4(3):323–350.Google Scholar
- 12.Shin Y., Ghosh J., Samani D. (1992) Computationally efficient invariant pattern recognition with higher order Pi-Sigma Networks, The University of Texas at Austin.Google Scholar
- 13.Shin Y., Ghosh J., Samani D. (1992) computationally efficient invariant pattern classification with higher order pi-sigma networks, in Intelligent Engineering Systems through Artificial Neural Networks, C. H. Dagli, L. I. Burke, and Y. C. Shin, Eds. New York: ASME Press, 2: 379–384.Google Scholar
- 14.Shin Y., Ghosh I. (1995) Ridge Polynomial Networks, IEEE Transactions on Neural Networks, 6(3): 610–622.Google Scholar
- 15.Shin Y., Jin K-S. Yoon Byung-Moon. (1997) A complex pi-sigma network and its application to equalization of nonlinear satellite channels Neural Networks, International Conference on, 1:148–152.Google Scholar
- 16.Medsker L., Jain L.C. (1999) Recurrent neural networks: design and applications. CRC press.Google Scholar
- 17.Hussain A.J., Liatsis P. (2002) Recurrent pi-sigma networks for DPCM image coding, Neurocomputing, 55:363–382.Google Scholar
- 18.Voutriaridis C., Boutalis Y.S., Mertzios G. (2003) Ridge Polynomial Networks in pattern recognition, EC-VIP-MC 2003, 4th EURASIP Conference focused on Video/Image Processing and Multimedia Communications, Croatia, 519–524.Google Scholar
- 19.Ghazali R., Hussain A., El-Dereby W. (2006) Application of Ridge Polynomial Neural Networks to Financial Time Series Prediction, International Joint Conference on Neural Networks (IJCNN), 913–920.Google Scholar
- 20.Xiu J., Chang-Liang Xia. (2007) Modeling of Switched Reluctance Motor Based on Pi-sigma Neural Network, IEEE International Symposium on Industrial Electronics, 1258–1263.Google Scholar
- 21.Ge S., Peng C., Miao X. (2008) Visual Cryptography Scheme Using Pi-sigma Neural Networks, International Symposium on Information Science and Engineering, 2: 679–682.Google Scholar
- 22.Nie Y., Deng W. (2008) A Hybrid Genetic Learning Algorithm for Pi-Sigma Neural Network and the Analysis of Its Convergence, Fourth International Conference on Natural Computation, 3:19–23.Google Scholar
- 23.Husssain A.J., Jameel A.J., Al-Jumeily D., Ghazali R. (2009) Speech prediction using higher order neural networks, International Conference on Innovations in Information Technology (IIT), 294–298.Google Scholar
- 24.Ghazali R., Hussain A. J., Al-Jumeily D., & Merabti M. (2007, April) Dynamic ridge polynomial neural networks in exchange rates time series forecasting. In International Conference on Adaptive and Natural Computing Algorithms, Springer, Berlin, Heidelberg, 123–132.Google Scholar
- 25.Ghazali R., Hussain A.J., Al-Jumeily D., Lisboa P. (2009)Time series prediction using dynamic ridge polynomial neural networks, In Developments in eSystems Engineering (DESE), Second International Conference on, 354–363.Google Scholar
- 26.Ghazali R., Hussain A.J., Nawi N.M., Mohamad B. (2009) Non-stationary and stationary prediction of financial time series using dynamic ridge polynomial neural network, Neurocomputing, 72(10): 2359–2367.Google Scholar
- 27.Ghazali R., Jumeily D. (2009) Application of Pi-Sigma Neural Networks and Ridge Polynomial Neural networks to Financial Time Series Prediction, In: Zhang, M. (ed.) Artificial Higher order Neural Networks for Economics and Business, Information Science Reference, 271–293.Google Scholar
- 28.Ghazali R., Hussain A., Nawi N.M. (2010) Dynamic ridge polynomial higher order neural network, Artificial Higher Order Neural Networks for Computer Science and Engineering, 255–268.Google Scholar
- 29.Husaini N.A., Ghazali R., Nawi N.M. and Ismail L.H. (2011) Jordan Pi-sigma neural network for temperature prediction, in Ubiquitous Computing and Multimedia Applications. Springer, Berlin, Heidelberg, 547–558.Google Scholar
- 30.Husaini N.A., Ghazali R., Nawi N.M., Ismail L.H. (2011) Pi-Sigma Neural Network for Temperature Forecasting in Batu Pahat, In: Zain, J.M., Wan Mohd, W.M.b., El-Qawasmeh, E. (eds.) ICSECS, Part II. CCIS Springer, Heidelberg, 180: 530–541.Google Scholar
- 31.Ghazali R., Husaini N.A., Ismail L.H., Samsuddin N.A. (2012) An application of Jordan Pi-sigma neural network for the prediction of temperature time series signal, Recurrent Neural Networks and Soft Computing, 13(4):275–290.Google Scholar
- 32.Yu X., FengChen Q. (2012) Convergence of gradient method with penalty for Ridge Polynomial neural network, Neurocomputing, (97): 405–409.Google Scholar
- 33.Cao Q., Ewing B.T., Thompson M.A. (2012) Forecasting wind speed with recurrent neural networks. Eur. J. Oper. Res, 221(1): 148–154.Google Scholar
- 34.Panigrahi S., Pandey S., Singh R. (2013) A Novel Evolutionary Higher Order Neural Network for Pattern Classification, International Journal of Engineering Research and Technology, 2(9).Google Scholar
- 35.Karali Y., Panigrahi S., Behera H.S. (2013) A novel Differential evolution based algorithm for higher order neural network training, Journal of Theoretical & Applied Information Technology, 56(3).Google Scholar
- 36.Sahu K.K., Panigrahi S., Behera H.S. (2013) A Novel Chemical Reaction Optimization Algorithm For Higher Order Neural Network Training, Journal of Theoretical & Applied Information Technology, 53(3).Google Scholar
- 37.Panigrahi S., Behera H.S. (2013) Effect of Normalization Techniques on Univariate Time Series Forecasting using Evolutionary Higher Order Neural Network, Int. J. Eng. Adv. Technol 3(2): 280–285.Google Scholar
- 38.Nayak J., Naik B., Behera H.S. (2014) A hybrid PSO-GA based Pi sigma neural network (PSNN) with standard back propagation gradient descent learning for classification. Control, Instrumentation, Communication and Computational Technologies (ICCICCT), International Conference on. IEEE (2014).Google Scholar
- 39.Behera N.K.S., Behera H.S. (2014) Firefly based ridge polynomial neural network for classification, IEEE International Conference on Advanced Communications, Control and Computing Technologies, 1110–1113.Google Scholar
- 40.Nayak J., Kanungo D.P., Naik B., Behera H.S. (2014) A higher order evolutionary Jordan Pi-Sigma Neural Network with gradient descent learning for classification, International Conference on High Performance Computing and Applications (ICHPCA), 1–6.Google Scholar
- 41.Nayak S. C., Misra B.B., Behera H.S. (2015) A Pi-Sigma Higher Order Neural Network for Stock Index Forecasting. Computational Intelligence in Data Mining, Springer India, 2: 311–319.Google Scholar
- 42.Nayak J., Naik B., Behera H.S. (2016) A novel nature inspired firefly algorithm with higher order neural network: Performance analysis, Engineering Science and Technology, an International Journal, 19(1):197–211.Google Scholar
- 43.Nayak J., Naik B., Behera H.S. (2016) Optimizing a higher order neural network through teaching learning based optimization algorithm, Computational Intelligence in Data Mining, Springer India, (1):57–71.Google Scholar
- 44.Nayak J., Naik B., Behera H.S. (2016) Solving nonlinear Classification problems with black hole optimization and higher order Jordan Pi-Sigma neural network: a novel approach, 2(4): 236–251.Google Scholar
- 45.Panigrahi S. (2017) A Novel Hybrid Chemical Reaction Optimization Algorithm with Adaptive Differential Evolution Mutation Strategies for Higher Order Neural Network Training, International Arab Journal of Information Technology (IAJIT), 14(1).Google Scholar
- 46.Waheeb W., Ghazali R., Herawan T. (2017) Time Series Forecasting Using Ridge Polynomial Neural Network with Error Feedback, In: Herawan T., Ghazali R., Nawi N., Deris M. (eds) Recent Advances on Soft Computing and Data Mining. SCDM 2016. Advances in Intelligent Systems and Computing Springer, Cham, 549.Google Scholar
- 47.Schmitt M. (2002) On the complexity of computing and learning with multiplicative neurons, Neural Computation, 14(2):241–301.Google Scholar
- 48.Jordan M.I. (1986) Attractor Dynamics and Parallelism in a Connectionist Sequential Machine, Proceedings of the Eighth Conference of the Cognitive Science Society, New Jersey, USA.Google Scholar
- 49.Husaini N.A., Ghazali R., Ismail L.H., Herawan T. (2014) A Jordan Pi-Sigma Neural Network for Temperature Forecasting in Batu Pahat Region. In: Herawan T., Ghazali R., Deris M. (eds) Recent Advances on Soft Computing and Data Mining, Advances in Intelligent Systems and Computing Springer, Cham, 287.Google Scholar
- 50.Ghazali R. (2007) Higher order neural networks for financial time series prediction, PhD diss., Liverpool John Moores University.Google Scholar
- 51.Liu Y., Yang J., Yang D., Wu W. (2014) A modified gradient-based neuro-fuzzy learning algorithm for pi-sigma network based on first-order takagi-sugeno system. J Math Res Appl, 34(1): 114–126.Google Scholar
- 52.Ghazali R., Nazri M.N., Mohd N.M.S. (2011) Dynamic Ridge Polynomial Neural Network with a Real Time Recurrent Learning Algorithm: Forecasting the S&P 500 (<Special Issue> SOFT COMPUTING METHODOLOGIES AND ITS APPLICATIONS), Biomedical fuzzy and human sciences: the official journal of the Biomedical Fuzzy Systems Association, 16(2): 97–103.Google Scholar
- 53.Ghazali R., Hussain A.J., Liatsis P. (2011) Dynamic Ridge Polynomial Neural Network: Forecasting the univariate non-stationary and stationary trading signals, Expert Systems with Applications, 38(4): 3765–3776.Google Scholar
- 54.Barbounis T., Theocharis J. (2007) A locally recurrent fuzzy neural network with application to the wind speed prediction using spatial correlation. Neurocomputing 70: 1525–1452.Google Scholar
- 55.Rao R.V., Savsani V.J., Vakharia D.P. (2012) Teaching–learning-based optimization: an optimization method for continuous non-linear large scale problems, Information sciences 183(1):1–15.Google Scholar
- 56.L. Spirkovska and M.B. Reid, (1990) Connectivity Strategies for Higher-order Neural Networks applied to Pattern Recognition, Proceedings of IJCNN, San Diego, 1:21–26.Google Scholar
- 57.Hara Y. (1994) Application of neural networks to radar image classification, IEEE Transactions on Geoscience and Remote Sensing, 32(1): 100–109.Google Scholar
- 58.Hsieh T.J., Hsiao H.F., Yeh W.C. (2011) Forecasting stock markets using wavelet transforms and recurrent neural networks: an integrated system based on artificial bee colony algorithm. Appl. Soft. Comput, 11(2): 2510–2525.Google Scholar
- 59.D.S. Huang, H.H.S. lp, K.C.K. Law and Z. Chi (2005) Zeroing polynomials using modified constrained neural network approach. IEEE Transactions on neural networks, 16(3):721–732.Google Scholar
- 60.S. Perantonis, N. Ampazis, S. Varoufakis and G. Antoniou (1998) Constrained learning in neural networks, Application to factorization of 2-d polynomials, Neural Processing Letter, 7(1):5–14.Google Scholar
- 61.Tawfik H., Liatsis P. (1997) Prediction of Non-linear Time series using Higher Order Neural Network, Proceeding IWSSIP’97 Conference, Poznan, Poland.Google Scholar
- 62.Voutriaridis C., Boutalis Y.S, Mertzios G. (2003) Ridge Polynomial Networks in Pattern Recognition.EC-VIP-MC 2003, 4th EURASIP Conference focused on Video/Image Processing and Multimedia Communications, Croatia, 519–524.Google Scholar