Advertisement

A Hybrid Approach for Word Alignment with Statistical Modeling and Chunker

  • Jyoti SrivastavaEmail author
  • Sudip Sanyal
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9041)

Abstract

This paper presents a hybrid approach to improve word alignment with Statistical Modeling and Chunker for English-Hindi language pair. We first apply the standard word alignment technique to get an approximate alignment. The source and target language sentences are divided into chunks. The approximate word alignment is then used to align the chunks. The aligned chunks are then used to improve the original word alignment.

The statistical model used here is IBM Model 1. CRF Chunker is used to break the English sentences into chunks. A shallow parser is used to break Hindi sentences into chunks. This paper demonstrates an increment in F-measure by approximately 7% and reduction in Alignment Error Rate (AER) by approximately 7% in comparison to the performance of IBM Model 1 for word alignment. Experiments of this paper are based on TDIL corpus of 1000 sentences.

Keywords

Word alignment Chunk alignment Natural language processing Artificial Intelligence 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Aswani, N., Gaizauskas, R.: A hybrid approach to align sentences and words in english-hindi parallel corpora. In: Proceedings of the ACL Workshop on Building and Using Parallel Texts, ParaText 2005, pp. 57–64. Association for Computational Linguistics, Stroudsburg (2005), http://dl.acm.org/citation.cfm?id=1654449.1654458 CrossRefGoogle Scholar
  2. 2.
    Brown, P.F., Pietra, V.J.D., Pietra, S.A.D., Mercer, R.L.: The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics 19(2), 263–311 (1993)Google Scholar
  3. 3.
    Caseli, H., Ramisch, C., Nunes, M.G.V., Villavicencio, A.: Alignment-based extraction of multiword expressions. Language Resources and Evaluation 44(1-2), 59–77 (2010), http://dx.doi.org/10.1007/s10579-009-9097-9 CrossRefGoogle Scholar
  4. 4.
    Deng, Y., Kumar, S., Byrne, W.: Segmentation and alignment of parallel text for statistical machine translation. Natural Language Engineering 13(3), 235–260 (2007)CrossRefGoogle Scholar
  5. 5.
    Gale, W.A., Church, K.W.: Identifying word correspondence in parallel texts. In: Proceedings of the Workshop on Speech and Natural Language, HLT 1991, pp. 152–157. Association for Computational Linguistics, Stroudsburg (1991), http://dx.doi.org/10.3115/112405.112428 Google Scholar
  6. 6.
    Hutchins, W., Somers, H.: An introduction to machine translation. Academic Press (1992), http://books.google.co.in/books?id=0ZhrAAAAIAAJ
  7. 7.
    Kim, Y.-B., Ehara, T.: A method for partitioning of long japanese sentences with subject resolution in j/e machine translation. In: Proceedings of International Conference on Computer Processing of Oriental Languages, pp. 467–473 (1994)Google Scholar
  8. 8.
    Koehn, P.: Statistical Machine Translation. Cambridge University Press, New York (2010)zbMATHGoogle Scholar
  9. 9.
    Kolachina, P., Cancedda, N., Dymetman, M., Venkatapathy, S.: Prediction of learning curves in machine translation. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 22–30. Association for Computational Linguistics, Jeju Island (2012), http://www.aclweb.org/anthology/P12-1003
  10. 10.
    Lambert, P., de Gispert, A., Banchs, R.E., Mario, J.B.: Guidelines for word alignment evaluation and manual alignment. Language Resources and Evaluation 39(4), 267–285 (2005), http://dblp.uni-trier.de/db/journals/lre/lre39.html#LambertGBM05 CrossRefGoogle Scholar
  11. 11.
    Meng, B., Huang, S., Dai, X., Chen, J.: Segmenting long sentence pairs for statistical machine translation. In: International Conference on Asian Language Processing, IALP 2009, Singapore, pp. 53–58 (December 2009)Google Scholar
  12. 12.
    Och, F.J., Ney, H.: A systematic comparison of various statistical alignment models. Computational Linguistics 29(1), 19–51 (2003), http://dx.doi.org/10.1162/089120103321337421 CrossRefzbMATHGoogle Scholar
  13. 13.
    Ramanathan, A., Bhattacharyya, P., Visweswariah, K., Ladha, K., Gandhe, A.: Clause-based reordering constraints to improve statistical machine translation. In: Proceedings of 5th International Joint Conference on Natural Language Processing, Chiang Mai, Thailand, pp. 1351–1355 (November 2011)Google Scholar
  14. 14.
    Smadja, F., McKeown, K.R., Hatzivassiloglou, V.: Translating collocations for bilingual lexicons: a statistical approach. Computational Linguistics 22(1), 1–38 (1996), http://dl.acm.org/citation.cfm?id=234285.234287 Google Scholar
  15. 15.
    Srivastava, J., Sanyal, S.: A hybrid approach for word alignment in english-hindi parallel corpora with scarce resources. In: International Conference on Asian Language Processing (IALP), pp. 185–188 (2012)Google Scholar
  16. 16.
    Srivastava, J., Sanyal, S.: Segmenting long sentence pairs to improve word alignment in english-hindi parallel corpora. In: Isahara, H., Kanzaki, K. (eds.) JapTAL 2012. LNCS, vol. 7614, pp. 97–107. Springer, Heidelberg (2012), http://dblp.uni-trier.de/db/conf/tal/japtal2012.html#SrivastavaS12 CrossRefGoogle Scholar
  17. 17.
    Sudoh, K., Duh, K., Tsukada, H., Hirao, T., Nagata, M.: Divide and translate: Improving long distance reordering in statistical machine translation. In: Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Metrics MATR, pp. 418–427. Association for Computational Linguistics, Uppsala (2010), http://www.aclweb.org/anthology/W10-1762 Google Scholar
  18. 18.
    Sun, L., Jin, Y., Du, L., Sun, Y.: Word alignment of english-chinese bilingual corpus based on chunks. In: Proceedings of the 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora: Held in Conjunction with the 38th Annual Meeting of the Association for Computational Linguistics-Volume 13, pp. 110–116. Association for Computational Linguistics (2000)Google Scholar
  19. 19.
    Venkataramani, E., Gupta, D.: English-hindi automatic word alignment with scarce resources. In: Dong, M., Zhou, G., Qi, H., Zhang, M. (eds.) International Conference on Asian Language Processing (IALP), pp. 253–256. IEEE Computer Society (2010), http://dblp.uni-trier.de/db/conf/ialp/ialp2010.html#VenkataramaniG10
  20. 20.
    Watanabe, T., Sumita, E., Okuno, H.G.: Chunk-based statistical translation. In: Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, vol. 1, pp. 303–310. Association for Computational Linguistics (2003)Google Scholar
  21. 21.
    Xu, J., Zens, R., Ney, H.: Sentence segmentation using ibm word alignment model 1. In: In Proceedings of EAMT 2005 (10th Annual Conference of the European Association for Machine Translation), Budapest, Hungary, pp. 280–287 (May 2005)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Indian Institute of Information TechnologyAllahabadIndia

Personalised recommendations