Non-parametric models

Abstract

This chapter presents a range of statistical techniques that are available for the analysis of word frequency distributions. Section 2.1 introduces some basic probabilistic concepts. Section 2.2 discusses the urn model, according to which word use is viewed as random selection from a population with fixed probabilities for words to occur. The binomial model and the Poisson approximation to the binomial model are defined here. Section 2.3 is concerned with the structural type distribution, which allows us to restate the Poisson model in integral form. Section 2.4 introduces the concept of the LNRE zone, the range of sample sizes where the sample relative frequencies are not good estimates of the corresponding population probabilities. The next section (2.5) focuses on the Good-Turing estimates, which adjust sample relative frequencies for the non-negligible frequency weight of the unseen words. Methods for calculating the frequency spectrum for any sample size given the frequency spectrum for a given sample size are presented in sections 2.6 and 3.2.