Proposal of a New ‘Constraint Measure’ for Text

Abstract

Zipf’s law is one of the most famous generalizations in text statistics. S. Mizutani introduced a new representation which approximates a text more precisely. We show that his representation is well approximated by the ratio N/L, where N is the number of words and L the number of different words in the target text. This ratio was applied to several examples of text; computer language, non-native English, and standard English. We found that his representation, which is approximated by the above ratio, is modified into a better measure of constraint (size of the domain expressed by the text) of the sentence, (math). This measure roughly classifies English text in the order of constraint. Combined with another parameter R, the correlation coefficient between word-length and logarithmic-scaled rank order, a stronger classifier can be proposed.