Abstract
The probability estimates in stochastic language modelling often depend on some additional parameters apart from the training data. These parameters are typically related to the probabilities of events not seen in the training data and conventional maximum-likelihood methods therefore fail to determine them. We present a special form of cross validation, the leaving-one-out concept, to solve this problem. The application of this technique to several different modelling approaches reveals its flexibility and in some cases the simple way of computation. Experiments, performed on an English corpus of 1.1 million words, show the good generalization capability.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
R. O. Duda, P. E. Hart: Pattern Classification and Scene Analysis, Wiley, New York, 1973.
I J. Good: “The population frequencies of species and the estimation of population parameters”, Biometrika 40, pp. 237–264, Dec. 1953.
A. Nadas: “On Turing’s formula for word probabilities”, IEEE Trans, on Acoustics, Speech and Signal Proc, Vol. ASSP-33, pp.1414–1416, Dec. 1985.
S.M. Katz: “Estimation of probabilities from sparse data for the language model component of a speech recognizer”, IEEE Trans, on Acoustics, Speech and Signal Proc, Vol. SSP-35, pp. 400–401, March 1987.
H. Ney, U. Essen: “On smoothing techniques for bigram-based natural language modelling”, Proc. ICASSP, Vol. 2, pp. 825–828, May 1991.
F. Jelinek, R.L. Mercer: “Interpolated estimation of Markov source parameters from sarse data”, pp. 381–397, in E.S. Gelsema, L.N. Kanal (eds.): Pattern Recognition in Practice, North-Holland Publ. Company, Amsterdam, 1980.
R. Kneser, H. Ney: “Improved clustering techniques for class-based statistical language modelling”, Proc. Eurospeech, Vol. 2, pp. 973–976 Sept. 1993.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1995 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kneser, R., Essen, U., Ney, H. (1995). On the Use of the Leaving-One-Out Method in Statistical Language Modelling. In: Ayuso, A.J.R., Soler, J.M.L. (eds) Speech Recognition and Coding. NATO ASI Series, vol 147. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-57745-1_35
Download citation
DOI: https://doi.org/10.1007/978-3-642-57745-1_35
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-63344-7
Online ISBN: 978-3-642-57745-1
eBook Packages: Springer Book Archive