Abstract
This chapter continues from the general introduction to neural networks, to a focus on recurrent networks. The recurrent neural network is the most popular neural network approach for working with sequences of dynamic size. As with the prior chapter, readers familiar with RNNs can reasonably skip this. Note that this chapter does not pertain specifically to NLP. However, as NLP tasks are almost always sequential in nature, RNNs are fundamental to many NLP systems
I’ve the RNN with and works, but the computedwith program of the RNN with and the computed of the RNN with with and the code — The output of an RNN created by Andrej Karpathy (2015), trained on an article on the use of RNNs for generating text (it works poorly due to the very low amount of training data)
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bengio, Yoshua, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5 (2): 157–166.
Cho, Kyunghyun, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: Encoder-decoder approaches. In Eighth workshop on syntax, semantics and structure in statistical translation (SSST-8).
Cho, Kyunghyun, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). Doha, Qatar: Association for Computational Linguistics, 1724–1734.
Chung, Junyoung, Caglar Gulcehre, Kyung Hyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555.
Dyer, Chris, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In CoRR. arXiv:1505.08075.
Elman, Jeffrey L. 1990. Finding structure in time. Cognitive Science 14 (2): 179–211.
Gers, Felix A, Jürgen Schmidhuber, and Fred Cummins. 1999. Learning to forget: Continual prediction with LSTM. Neural Computation 12 (10): 2451–2471
Graves, Alex, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. In CoRR. arXiv:1410.5401.
Hochreiter, Sepp, and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9 (8): 1735–1780.
Jordan, M.I. 1986. Serial order: A parallel distributed processing approach (Technical Report No. 8604). San Diego, La Jolla, CA: Institute for Cognitive Science, University of California.
Jozefowicz, Rafal, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd international conference on machine learning (ICML-15), 2342–2350.
Mikolov, Tomas, Armand Joulin, Sumit Chopra, Michaël Mathieu, and Marc’Aurelio Ranzato. 2014. Learning longer memory in recurrent neural networks. In CoRR. arXiv:1412.7753.
Monroe, W., N. D. Goodman, and C. Potts. 2016. Learning to generate compositional color descriptions. arXiv:1606.03821 [cs.CL].
Schuster, Mike, and Kuldip K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45 (11): 2673–2681.
Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In The IEEE conference on computer vision and pattern recognition (CVPR).
Weston, Jason, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. In CoRR. arXiv:1410.3916.
White, L., R. Togneri, W. Liu, and M. Bennamoun. 2017. Learning distributions of meant color. arXiv:1709.09360 [cs.CL].
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
White, L., Togneri, R., Liu, W., Bennamoun, M. (2019). Recurrent Neural Networks for Sequential Processing. In: Neural Representations of Natural Language. Studies in Computational Intelligence, vol 783. Springer, Singapore. https://doi.org/10.1007/978-981-13-0062-2_2
Download citation
DOI: https://doi.org/10.1007/978-981-13-0062-2_2
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-0061-5
Online ISBN: 978-981-13-0062-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)