Document Length Normalization
Length normalization; Term frequency normalization
Document length normalization adjusts the term frequency or the relevance score in order to normalize the effect of document length on the document ranking.
The reasons for employing a document length normalization method in an IR system are quite subtle. In general, the effect observed on the ranking by the presence of many lengthy documents in a collection is to favor their retrieval with respect to shorter documents.
The same term usually occurs repeatedly in long documents.
The vocabulary of a long document is usually large.
In 1994, Robertson and Walker also studied the effect of document length in the context of the probabilistic model. They observed that:
Some documents may simply cover more material than others, […], a long document covers a similar scope to a...
- 1.Amati G. Probabilistic models for information retrieval based on divergence from randomness [PhD thesis]. University of Glasgow: Department of Computing Science; 2003.Google Scholar
- 3.Robertson SE, Walker S, Jones S, Hancock-Beaulieu M. Okapi at trec-3. In: Proceedings of the 3rd Text Retrieval Conference; 1994.Google Scholar
- 4.Singhal A, Buckley C, Mitra M. Pivoted document length normalization. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; 1996. p. 21–9.Google Scholar
- 5.Zhai C, Lafferty J. A study of smoothing methods for language models applied to ad hoc information retrieval. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; 2001. p. 334–42.Google Scholar