Abstract
We present our first results in applications of recurrent neural networks to Russian. The problem of re-scoring of equiprobable hypotheses has been solved. We train several recurrent neural networks on a lemmatized news corpus to mitigate the problem of data sparseness. We also make use of morphological information to make the predictions more accurate. Finally we train the Ranking SVM model and show that combination of recurrent neural networks and morphological information gives better results than 5-gram model with Knesser-Ney discounting.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bellegarda, J.R.: Exploiting latent semantic information in statistical language modeling. Proc. IEEE 88(8), 1279–1296 (2000)
Bengio, Y., Ducharme, R., Vincent, P., Janvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Transact. Neural Netw. 5(2), 157–166 (1994)
Deoras, A., Mikolov, T., Kombrink, S., Church, K.: Approximate inference: a sampling based modeling technique to capture complex dependencies in a language model. Speech Commun. 55(1), 162–177 (2013)
Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990)
Gildea, D., Hofmann, T.: Topic-based language models using em. History (1999)
Hochreiter, S., Schmidhuber, J.: Bridging long time lags by weight guessing and long short-term memory. In: Silva, F.L., PrincÃpe, J.C., Almeida, L.B. (eds.) Spatiotemporal Models in Biological and Artificial Systems, vol. 37, pp. 65–72. IOS Press, Amsterdam (1996)
Joachims, T.: Optimizing search engines using clickthrough data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 133–142. ACM (2002)
Mikolov, T.: Statistical language models based on neural networks. Ph.D. thesis, Brno University of Technology (2012)
Mikolov, T., Karafiát, M., Burget, L., Cernockỳ, J., Khudanpur, S.: Recurrent neural network based language model. In: INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, pp. 1045–1048, 26–30 September 2010 (2010)
Muzychka S.A., Romanenko A.A., Piontkovskaja I.I.: Conditional random field for morphological disambiguation in Russian. In: Conference Dialog-2014 (2014)
Oparin, I.: Language models for automatic speech recognition of inflectional languages. Ph.D. thesis, University of West Bohemia (2008)
Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks (2012). arXiv preprint arXiv:1211.5063
Vazhenina, D., Markov, K.: Evaluation of advanced language modeling techniques for Russian LVCSR. In: Železný, M., Habernal, I., Ronzhin, A. (eds.) SPECOM 2013. LNCS, vol. 8113, pp. 124–131. Springer, Heidelberg (2013)
Whittaker, E.W.D.: Statistical language modelling for automatic speech recognition of Russian and English. Ph.D. thesis, University of Cambridge (2000)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Kudinov, M. (2015). Recurrent Neural Networks for Hypotheses Re-Scoring. In: Ronzhin, A., Potapova, R., Fakotakis, N. (eds) Speech and Computer. SPECOM 2015. Lecture Notes in Computer Science(), vol 9319. Springer, Cham. https://doi.org/10.1007/978-3-319-23132-7_42
Download citation
DOI: https://doi.org/10.1007/978-3-319-23132-7_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23131-0
Online ISBN: 978-3-319-23132-7
eBook Packages: Computer ScienceComputer Science (R0)