Testing the Effectiveness of Named Entities in Aligning Comparable English-Bengali Document Pair

Abstract

Named entities (NEs) play an important role in Cross Lingual Information Retrieval (CLIR). To verify whether documents in two different languages share information about same things, we may check if those two documents have fair number of NEs in common. Comparable documents generally share many named entities. In the present work, we test the effectiveness of named entities in aligning English-Bengali comparable document pairs. We develop an aligned corpus of English-Bengali document pairs using Wikipedia. We crawl English-Bengali document pairs by visiting the cross-lingual links found in the documents on Wikipedia. These document pairs are assumed to be comparable. To find the effectiveness of NE in aligning English-Bengali document pair, each English document is compared with all the other Bengali documents and the most similar Bengali document in terms of NE similarity is found. And then it is verified whether it is aligned successfully (since we already know the correct alignment). Rule based transliteration module is used to transliterate English named entities into Bengali named entities. Since, transliteration modules may not always produce exact transliterations; textual properties like longest common subsequence and minimum edit distance are adopted to check whether two Bengali words can be considered as alignment of each other. Our system achieved an accuracy of 45% for 100 English-Bengali document pairs.