Abstract
In this chapter we review two additional types of Recurrent Neural Network, which present important differences with respect to the architectures described so far. More specifically, we introduce the nonlinear auto-regressive with eXogenous inputs (NARX) neural network and the Echo State Network. Both these networks have been largely employed in Short Term Load Forecast applications and they have been shown to be more effective than other methods based on statistical models. The main differences of NARX networks and Echo State Networks with respect to the other previously described models, are both in terms of their architecture and, in particular, in their training procedure. Indeed, both these architectures are designed in such a way that Back Propagation Through Time is not necessary. Specifically, in NARX the network output is replaced by the expected ground truth and this allows to train the network like a feedforward architecture. On the other hand, in a Echo State Network only the outermost linear layer is trained, usually by means of ridge regression. Due to these fundamental differences, some of the properties and training approaches discussed in the previous sections do not hold for the NARX and Echo State Network models and we reserved a separate chapter to review these models.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Alexandre LA, Embrechts MJ, Linton J (2009) Benchmarking reservoir computing on time-independent classification tasks. In: 2009 International Joint Conference on Neural Networks, IJCNN 2009. IEEE, pp 89–93
Appeltant L, Soriano MC, Van der Sande G, Danckaert J, Massar S, Dambre J, Schrauwen B, Mirasso CR, Fischer I (2011) Information processing using a single dynamical node as complex system. Nature Commun 2:468. https://doi.org/10.1038/ncomms1476
Bianchi FM, De Santis E, Rizzi A, Sadeghian A (2015) Short-term electric load forecasting using echo state networks and PCA decomposition. IEEE Access 3:1931–1943. https://doi.org/10.1109/ACCESS.2015.2485943
Bianchi FM, Scardapane S, Uncini A, Rizzi A, Sadeghian A (2015) Prediction of telephone calls load using echo state network with exogenous variables. Neural Netw 71:204–213. https://doi.org/10.1016/j.neunet.2015.08.010
Bianchi FM, Livi L, Alippi C (2016a) Investigating echo-state networks dynamics by means of recurrence analysis. IEEE Trans Neural Netw Learn Syst 99:1–13. https://doi.org/10.1109/TNNLS.2016.2630802
Billings SA (2013) Nonlinear system identification: NARMAX methods in the time, frequency, and spatio-temporal domains. Wiley
Boedecker J, Obst O, Lizier JT, Mayer NM, Asada M (2012) Information processing in echo state networks at the edge of chaos. Theory Biosci 131(3):205–213
Cambria E, Huang GB, Kasun LLC, Zhou H, Vong CM, Lin J, Yin J, Cai Z, Liu Q, Li K et al (2013) Extreme learning machines [trends & controversies]. IEEE Intell Syst 28(6):30–59
Deihimi A, Showkati H (2012) Application of echo state networks in short-term electric load forecasting. Energy 39(1):327–340
Deihimi A, Orang O, Showkati H (2013) Short-term electric load and temperature forecasting using wavelet echo state networks with neural reconstruction. Energy 57:382–401
Diaconescu E (2008) The use of narx neural networks to predict chaotic time series. Wseas Trans Comput Res 3(3):182–191
Filippo MB, Livi L, Alippi C, Jenssen R (2017) Multiplex visibility graphs to investigate recurrent neural network dynamics. Sci Rep 7(44):037. https://doi.org/10.1038/srep4403710.1038/srep44037
Gallicchio C, Micheli A, Pedrelli L (2017) Deep reservoir computing: a critical experimental analysis. Neurocomputing. https://doi.org/10.1016/j.neucom.2016.12.089
Hai-yan D, Wen-jiang P, Zhen-ya H (2005) A multiple objective optimization based echo state network tree and application to intrusion detection. In: 2005 Proceedings of 2005 IEEE international workshop on VLSI design and video technology, pp 443–446. https://doi.org/10.1109/IWVDVT.2005.1504645
Han S, Lee J (2014a) Fuzzy echo state neural networks and funnel dynamic surface control for prescribed performance of a nonlinear dynamic system. IEEE Trans Ind Electron 61(2):1099–1112. https://doi.org/10.1109/TIE.2013.2253072
Han SI, Lee JM (2014b) Fuzzy echo state neural networks and funnel dynamic surface control for prescribed performance of a nonlinear dynamic system. IEEE Trans Ind Electron 61(2):1099–1112
Huang CM, Huang CJ, Wang ML (2005) A particle swarm optimization to identifying the armax model for short-term load forecasting. IEEE Trans Power Syst 20(2):1126–1133
Jaeger H (2001) The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. Technical Report 148:34, German National Research Center for Information Technology GMD, Bonn, Germany
Jaeger H, Haas H (2004) Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 304(5667):78–80. https://doi.org/10.1126/science.1091277
Leontaritis I, Billings SA (1985) Input-output parametric models for non-linear systems part i: deterministic non-linear systems. Int J Control 41(2):303–328
Li D, Han M, Wang J (2012a) Chaotic time series prediction based on a novel robust echo state network. IEEE Trans Neural Netw Learn Syst 23(5):787–799
Lin T, Horne BG, Tiňo P, Giles CL (1996) Learning long-term dependencies in narx recurrent neural networks. IEEE Trans Neural Netw 7(6):1329–1338
Lin TN, Giles CL, Horne BG, Kung SY (1997) A delay damage model selection algorithm for narx neural networks. IEEE Trans Signal Process 45(11):2719–2730
Løkse S, Bianchi FM, Jenssen R (2017) Training echo state networks with regularization through dimensionality reduction. Cogn Comput 1–15. https://doi.org/10.1007/s12559-017-9450-z
Lukoševičius M, Jaeger H (2009) Reservoir computing approaches to recurrent neural network training. Comput Sci Rev 3(3):127–149. https://doi.org/10.1016/j.cosrev.2009.03.005
Maass W, Natschläger T, Markram H (2002) Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput 14(11):2531–2560. https://doi.org/10.1162/089976602760407955
Maiorino E, Bianchi F, Livi L, Rizzi A, Sadeghian A (2017) Data-driven detrending of nonstationary fractal time series with echo state networks. Inf Sci 382–383:359–373. https://doi.org/10.1016/j.ins.2016.12.015
Mazumdar J, Harley R (2008) Utilization of echo state networks for differentiating source and nonlinear load harmonics in the utility network. IEEE Trans Power Electron 23(6):2738–2745. https://doi.org/10.1109/TPEL.2008.2005097
Menezes JMP, Barreto GA (2008) Long-term time series prediction with the narx network: an empirical evaluation. Neurocomputing 71(16):3335–3343
Napoli R, Piroddi L (2010) Nonlinear active noise control with narx models. IEEE Trans Audio, Speech, and Lang Process 18(2):286–295
Niu D, Ji L, Xing M, Wang J (2012) Multi-variable echo state network optimized by Bayesian regulation for daily peak load forecasting. J Netw 7(11):1790–1795
Peng Y, Lei M, Li JB, Peng XY (2014) A novel hybridization of echo state networks and multiplicative seasonal ARIMA model for mobile communication traffic series forecasting. Neural Comput Appl 24(3–4):883–890
Plett GL (2003) Adaptive inverse control of linear and nonlinear systems using dynamic neural networks. IEEE Trans Neural Netw 14(2):360–376
Rodan A, Tiňo P (2011) Minimum complexity echo state network. IEEE Trans Neural Netw 22(1):131–144. https://doi.org/10.1109/TNN.2010.2089641
Rodan A, Tiňo P (2012) Simple deterministically constructed cycle reservoirs with regular jumps. Neural Comput 24(7):1822–1852. https://doi.org/10.1162/NECO_a_00297
Scardapane S, Comminiello D, Scarpiniti M, Uncini A (2015) Online sequential extreme learning machine with kernels. IEEE Trans Neural Netw Learn Syst 26(9):2214–2220. https://doi.org/10.1109/TNNLS.2014.2382094
Siegelmann HT, Horne BG, Giles CL (1997) Computational capabilities of recurrent narx neural networks. IEEE Trans Syst Man Cybern Part B: Cybern 27(2):208–215
Skowronski MD, Harris JG (2007) Automatic speech recognition using a predictive echo state network classifier. Neural Netw 20(3):414–423
Varshney S, Verma T (2014) Half hourly electricity load prediction using echo state network. Int J Sci Res 3(6):885–888
Verstraeten D, Schrauwen B (2009) On the quantification of dynamics in reservoir computing. In: Alippi C, Polycarpou M, Panayiotou C, Ellinas G (eds) Artificial neural networks—ICANN 2009, vol 5768. Springer, Berlin, pp 985–994. https://doi.org/10.1007/978-3-642-04274-4_101
Xie H, Tang H, Liao YH (2009) Time series prediction based on narx neural networks: An advanced approach. In: 2009 International conference on machine learning and cybernetics, vol 3, pp 1275–1279. https://doi.org/10.1109/ICMLC.2009.5212326
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2017 The Author(s)
About this chapter
Cite this chapter
Bianchi, F.M., Maiorino, E., Kampffmeyer, M.C., Rizzi, A., Jenssen, R. (2017). Other Recurrent Neural Networks Models. In: Recurrent Neural Networks for Short-Term Load Forecasting. SpringerBriefs in Computer Science. Springer, Cham. https://doi.org/10.1007/978-3-319-70338-1_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-70338-1_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70337-4
Online ISBN: 978-3-319-70338-1
eBook Packages: Computer ScienceComputer Science (R0)