Abstract
This is short overview of the authors’ research in the area of the sequential or recursive Bayesian estimation of recurrent neural networks. Our approach is founded on the joint estimation of synaptic weights, neuron outputs and structure of the recurrent neural networks. Joint estimation enables generalization of the training heuristic known as teacher forcing, which improves the training speed, to the sequential training on noisy data. By applying Gaussian mixture approximation of relevant probability density functions, we have derived training algorithms capable to deal with non-Gaussian (multi modal or heavy tailed) noise on training samples. Finally, we have used statistics, recursively updated during sequential Bayesian estimation, to derive criteria for growing and pruning of synaptic connections and hidden neurons in recurrent neural networks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Williams, R. J. & Zipser, D.: Gradient-based learning algorithms for recurrent connectionist networks, TR NU_CCS_90-9, Boston: Northeastern University, CCS, 1990.
Williams, R. J.: Some observations on the use of the extended Kalman filter as a recurrent network learning algorithm, TR NU_CCS_92-1. Boston: Northeastern University, CCS, 1992.
Todorović B., Stanković M., Moraga C.: Derivative Free Training of Recurrent Neural Networks – A Comparison of Algorithms and Architectures, NCTA 2014 – Proceedings of the International Conference on Neural Computation Theory and Applications, part of IJCCI 2014, Rome, Italy, October, 2014, pp. 76–84.
Todorović B., Stanković M., Moraga C.: Recurrent Neural Networks Training Using Derivative Free Nonlinear Bayesian Filters, Computational Intelligence, in: Merelo, J. J.; Rosa, A.; Cadenas, J. M.; Dourado, A.; Madani, K.; Filipe, J. (eds.): Computational Intelligence, Proceedinga of the International Joint Conference, IJCCI 2014 Rome, Italy, October 22-24, 2014 Revised Selected Papers Berlin: Springer (Studies in Computational Intelligence, vol. 620), 2015, pp. pp 383–410.
Horne, B. G., Giles, C. L.: An experimental comparison of recurrent neural networks, Advances in Neural Information Processing Systems, vol. 7, 1995, pp. 697–704.
Nørgaard, M., Poulsen, N. K., and Ravn, O.: Advances in derivative free state estimation for nonlinear systems, Technical Report, IMM-REP-1998-15, Department of Mathematical Modelling, DTU, revised April 2000.
Julier, S. J., Uhlmann, J. K.: A new extension of the Kalman filter to nonlinear systems, Proceedings of AeroSense, The 11th international symposium on aero-space/defence sensing, simulation and controls, Orlando, FL, 1997.
Todorović, B., Stanković, M., Moraga C.: Nonlinear Bayesian Estimation of Recurrent Neural Networks, Proceedings of IEEE 4th International Conference on Intelligent Systems Design and Applications ISDA 2004, Budapest, Hungary, August 26–28, 2004, pp. 855–860.
Alspach, D. L. and Sorenson, H. W.: Nonlinear Bayesian Estimation using Gaussian Sum Approximation, IEEE Transactions on Automatic Control, vol. 17 (4), 1972, pp. 439–448.
Todorović,B., Stanković, M.: Sequential Growing and Pruning of Radial Basis Function network, Proceedings of IJCNN 2001, Washington DC, vol.3, 2001, pp., 1954–1959
Todorović B., Stanković M., Moraga C.: Gaussian sum filters for recurrent neural net-works training, NEUREL 2006: Eight Seminar on Neural Network Applications in Electrical Engineering, Proceedings, 2006, pp. 53–58.
Todorović, B., Stanković, M., Moraga C.: Modeling non-stationary dynamic systems using recurrent radial basis function networks, in Proc. of the 6th Seminar on Neural Network Applications in Electrical Engineering, NEUREL September 2002, Belgrade.
Todorović, B., Stanković, M., Moraga, C.: Extended Kalman Filter trained Recurrent Radial Basis Function Network in Nonlinear System Identification, in: Dorronsoro, José R. (ed.): Artificial Neural Networks – ICANN 2002, Proceedings of the International Conference, Madrid, Spain, August 2830, 2002, Berlin, Heidelberg: Springer (Lecture Notes in Computer Science Vol. 2415), pp 819–824.
Todorović, B., Moraga C., Stanković, M., Kovačević, B.: Neural Network training using Derivative Free Kalman filters, Proceedings of a Workshop on Computational Intelligence and Information Technologies, Nis, Serbia, October 13, 2003, pp. 39–46,
Todorović, B., Stanković, M., Moraga, C.: On-line Learning in Recurrent Neural Networks using Nonlinear Kalman Filters, in: Signal Processing and Information Technology, 2003. ISSPIT 2003. Proceedings of the 3rd IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), Darmstadt, Germany, December 2003.
Todorović, B., Stanković, M., Moraga C.: On-line Adaptation of Radial Basis Function Networks using the Extended Kalman Filter, in: Sin k, P., Vaščák, J. and Hirota, K. (eds.): Machine Intelligence: Quo Vadis? Advances in Fuzzy Systems-Applications and Theory, vol. 21, World Scientific, pp 73–92, 2004.
Todorović, B., Stanković, M., Moraga C.: Extended Kalman Filter Based Adaptation of Time-varying Recurrent Radial Basis Function Networks Structure, in Machine In-telligence: Quo Vadis?, in: Sinčák, P., Vaščák, J. and Hirota, K. (eds.): Machine Intelligence: Quo Vadis? Advances in Fuzzy Systems-Applications and Theory, vol. 21, World Scientific, pp 115–124, 2004.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Todorović, B., Moraga, C., Stanković, M. (2017). Sequential Bayesian Estimation of Recurrent Neural Networks. In: Seising, R., Allende-Cid, H. (eds) Claudio Moraga: A Passion for Multi-Valued Logic and Soft Computing. Studies in Fuzziness and Soft Computing, vol 349. Springer, Cham. https://doi.org/10.1007/978-3-319-48317-7_11
Download citation
DOI: https://doi.org/10.1007/978-3-319-48317-7_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-48316-0
Online ISBN: 978-3-319-48317-7
eBook Packages: EngineeringEngineering (R0)