Skip to main content
SpringerLink
Log in

Joint PoS Tagging and Stemming for Agglutinative Languages

  • Conference paper
  • First Online:
Computational Linguistics and Intelligent Text Processing (CICLing 2017)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10761))

  • 868 Accesses

Abstract

The number of word forms in agglutinative languages is theoretically infinite and this variety in word forms introduces sparsity in many natural language processing tasks. Part-of-speech tagging (PoS tagging) is one of these tasks that often suffers from sparsity. In this paper, we present an unsupervised Bayesian model using Hidden Markov Models (HMMs) for joint PoS tagging and stemming for agglutinative languages. We use stemming to reduce sparsity in PoS tagging. Two tasks are jointly performed to provide a mutual benefit in both tasks. Our results show that joint POS tagging and stemming improves PoS tagging scores. We present results for Turkish and Finnish as agglutinative languages and English as a morphologically poor language.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Available at http://www.ling.helsinki.fi/kieliteknologia/tutkimus/treebank/sources/.

References

  1. Adam, G., Asimakis, K., Bouras, C., Poulopoulos, V.: An efficient mechanism for stemming and tagging: the case of Greek language. In: Setchi, R., Jordanov, I., Howlett, R.J., Jain, L.C. (eds.) KES 2010. LNCS (LNAI), vol. 6278, pp. 389–397. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15393-8_44

    Chapter  Google Scholar 

  2. Biemann, C.: Unsupervised part-of-speech tagging employing efficient graph clustering. In: Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 7–12. Association for Computational Linguistics (2006)

    Google Scholar 

  3. Brants, T.: TNT: a statistical part-of-speech tagger. In: Proceedings of Sixth Conference on Applied Natural Language Processing, pp. 224–231. Association for Computational Linguistics (2000)

    Google Scholar 

  4. Brown, P.F., deSouza, P.V., Mercer, R.L., Pietra, V.J.D., Lai, J.C.: Class-based n-gram models of natural language. Comput. Linguist. 18(4), 467–479 (1992)

    Google Scholar 

  5. Brychcín, T., Konopík, M.: HPS: high precision stemmer. Inf. Process. Manage. 51(1), 68–91 (2015)

    Article  Google Scholar 

  6. Clark, A.: Inducing syntactic categories by context distribution clustering. In: Proceedings of the 2nd Workshop on Learning Language in Logic and the 4th Conference on Computational Natural Language Learning-Volume 7, pp. 91–94. Association for Computational Linguistics (2000)

    Google Scholar 

  7. Gao, J., Johnson, M.: A comparison of Bayesian estimators for unsupervised hidden Markov model PoS taggers. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 344–352. Association for Computational Linguistics (2008)

    Google Scholar 

  8. Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell. 6, 721–741 (1984)

    Article  Google Scholar 

  9. Goldsmith, J.: Unsupervised learning of the morphology of a natural language. Comput. Linguist. 27(2), 153–198 (2001)

    Article  MathSciNet  Google Scholar 

  10. Goldwater, S., Griffiths, T.: A fully Bayesian approach to unsupervised part-of-speech tagging. In: Annual Meeting-Association for Computational Linguistics, vol. 45, p. 744. Citeseer (2007)

    Google Scholar 

  11. Grönroos, S.A., Virpioja, S., Smit, P., Kurimo, M.: Morfessor FlatCat: an HMM-based method for unsupervised and semi-supervised learning of morphology. In: COLING, pp. 1177–1185 (2014)

    Google Scholar 

  12. Hankamer, J.: Finite state morphology and left to right phonology. In: Proceedings of the West Coast Conference on Formal Linguistics, vol. 5, pp. 41–52 (1986)

    Google Scholar 

  13. Johnson, M.: Why doesn’t EM find good HMM PoS-taggers? In: EMNLP-CoNLL, pp. 296–305 (2007)

    Google Scholar 

  14. Lovins, J.B.: Development of a stemming algorithm. Mech. Transl. Comput. Linguist. 11, 22 (1968)

    Google Scholar 

  15. Marcus, M.P., Marcinkiewicz, M.A., Santorini, B.: Building a large annotated corpus of English: the penn treebank. Comput. Linguist. 19(2), 313–330 (1993)

    Google Scholar 

  16. Mayfield, J., McNamee, P.: Single n-gram stemming. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval, pp. 415–416. ACM (2003)

    Google Scholar 

  17. Melucci, M., Orio, N.: A novel method for stemmer generation based on hidden Markov models. In: Proceedings of the Twelfth International Conference on Information and Knowledge Management, pp. 131–138. ACM (2003)

    Google Scholar 

  18. Merialdo, B.: Tagging English text with a probabilistic model. Computat. Linguist. 20(2), 155–171 (1994)

    Google Scholar 

  19. Mishra, U., Prakash, C.: MAULIK: an effective stemmer for Hindi language. Int. J. Comput. Sci. Eng. 4(5), 711 (2012)

    Google Scholar 

  20. Oflazer, K., Say, B., Hakkani-Tür, D.Z., Tür, G.: Building a Turkish treebank. In: Abeillé, A. (ed.) Treebanks. Text, Speech and Language Technology, vol. 20, pp. 261–277. Springer, Dordrecht (2003). https://doi.org/10.1007/978-94-010-0201-1_15

  21. Paik, J.H., Mitra, M., Parui, S.K., Järvelin, K.: GRAS: an effective and efficient stemming algorithm for information retrieval. ACM Trans. Inf. Syst. (TOIS) 29(4), 19 (2011)

    Article  Google Scholar 

  22. Paik, J.H., Pal, D., Parui, S.K.: A novel corpus-based stemming algorithm using co-occurrence statistics. In: Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 863–872. ACM (2011)

    Google Scholar 

  23. Petrov, S., Das, D., McDonald, R.: A universal part-of-speech tagset. arXiv preprint arXiv:1104.2086 (2011)

  24. Popat, P.P.K., Bhattacharyya, P.: Hybrid stemmer for Gujarati. In: Proceedings of the 23rd International Conference on Computational Linguistics, pp. 51–55 (2010)

    Google Scholar 

  25. Porter, M.F.: An algorithm for suffix stripping. Program 14(3), 130–137 (1980)

    Article  Google Scholar 

  26. Porter, M.F.: Snowball: a language for stemming algorithms (2001)

    Google Scholar 

  27. Rosenberg, A., Hirschberg, J.: V-measure: a conditional entropy-based external cluster evaluation measure. In: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, vol. 7, pp. 410–420 (2007)

    Google Scholar 

  28. Schütze, H.: Part-of-speech induction from scratch. In: Proceedings of the 31st Annual Meeting on Association for Computational Linguistics, pp. 251–258. Association for Computational Linguistics (1993)

    Google Scholar 

  29. Snyder, B., Naseem, T., Eisenstein, J., Barzilay, R.: Unsupervised multilingual learning for PoS tagging. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1041–1050. Association for Computational Linguistics (2008)

    Google Scholar 

  30. Van Gael, J., Vlachos, A., Ghahramani, Z.: The infinite HMM for unsupervised PoS tagging. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, vol. 2, pp. 678–687. Association for Computational Linguistics (2009)

    Google Scholar 

  31. Xu, J., Croft, W.B.: Corpus-based stemming using cooccurrence of word variants. ACM Trans. Inf. Syst. (TOIS) 16(1), 61–81 (1998)

    Article  Google Scholar 

Download references

Acknowledgments

This research is supported by the Scientific and Technological Research Council of Turkey (TUBITAK) with the project number EEEAG-115E464.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Hacettepe University, Beytepe, 06800, Ankara, Turkey

    Necva Bölücü & Burcu Can

Authors
  1. Necva Bölücü
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Burcu Can
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Necva Bölücü or Burcu Can .

Editor information

Editors and Affiliations

  1. CIC, Instituto Politécnico Nacional, Mexico City, Mexico

    Alexander Gelbukh

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bölücü, N., Can, B. (2018). Joint PoS Tagging and Stemming for Agglutinative Languages. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2017. Lecture Notes in Computer Science(), vol 10761. Springer, Cham. https://doi.org/10.1007/978-3-319-77113-7_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-77113-7_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-77112-0

  • Online ISBN: 978-3-319-77113-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation