A Data-Compression Approach to the Monolingual GIRT Task: An Agnostic Point of View
In this paper we apply a data-compression IR method in the GIRT social science database, focusing on the monolingual task in German and English. For this purpose we use a recently proposed general scheme for context recognition and context classification of strings of characters (in particular texts) or other coded information. The key point of the method is the computation of a suitable measure of remoteness (or similarity) between two strings of characters. This measure of remoteness reflects the distance between the structures present in the two strings, i.e. between the two different distributions of elements of the compared sequences. The hypothesis is that the information-theory oriented measure of remoteness between two sequences could reflect their semantic distance. It is worth stressing the generality and versatility of our information-theoretic method which applies to any kind of corpora of character strings, whatever the type of coding used (i.e. language).
KeywordsRelative Entropy Semantic Distance Kolmogorov Complexity Reference Text Context Classification
Unable to display preview. Download preview PDF.
- 1.Rijsbergen, C.J.: Information Retrieval, 2nd edn. Butterworths, London (1979)Google Scholar
- 2.Croft, B. (ed.): Advances in Information Retrieval – Recent Research from the Centre for Intelligent Information Retrieval. Kluwer Academic Publishers, Dordrecht (2003)Google Scholar
- 3.Shannon, C.E.: A Mathematical Theory of Communication. The Bell System Technical Journal 27, 379–423, 623–656 (1948)Google Scholar
- 4.Zurek, W.H. (ed.): Complexity, Entropy and Physics of Information. Addison-Wesley, Redwood City (1990)Google Scholar
- 7.Benedetto, D., Caglioti, E., Loreto, V.: Language Trees and Zipping. Physical Review Letters 88, 048702–048705 (2002)Google Scholar
- 11.Benedetto, D., Caglioti, E., Loreto, V.: Zipping Out Relevant Information. Invited column “Computing Prescriptions”. The AIP/IEEE journal Computing in Science and Engineering, January-February issue (2003)Google Scholar