Advertisement

The Implementation and Evaluation of a Lexicon-Based Stemmer

  • Gilberto Silva
  • Claudia Oliveira
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2857)

Abstract

This paper describes a stemming technique that depends principally on a target language’s lexicon, organised as an automaton of word strings. The clear distinction between the lexicon and the procedure itself allows the stemmer to be customised for any language with little or even no changes to the program’s source code. An implementation of the stemmer, with a medium sized Portuguese lexicon is evaluated using Paice’s [16] evaluation method.

Keywords

Retrieval Performance Lexical Entry Input Word Stem Group Deterministic Finite Automaton 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Adamson, G., Boreham, J.: The Use of an Association Measure Based on Character Structure to Identify Semantically Related Pairs of Words and Document Titles. Information Storage and Retrieval 10 (1974)Google Scholar
  2. 2.
    Aronoff, M., Anshen, F.: Morphology and the Lexicon: Lexicalization and Productivity. In: Spencer, A., Zwicky, A. (eds.) The Handbook of Morphology, Blackwell Publishers, Malden (1998)Google Scholar
  3. 3.
    Couto, M.: Representação de Léxicos através de Autômatos Finitos. MsC Dissertation. ICMC Universidade de São Paulo, São Carlos (1999)Google Scholar
  4. 4.
    Crystal, D.: An Encyclopedic Dictionary of Language and Languages. Penguin, London (1992)Google Scholar
  5. 5.
    Dawson, J.: Suffix Removal and Word Conflation. ALLC Bulletin, Michelmas (1974)Google Scholar
  6. 6.
    Frakes, W.B.: Stemming Algorithms. In: Frakes, W.B., Baeza-Yates, R. (eds.) Information Retrieval: Data Structures & Algorithms, Prentice-Hall, Englewood Cliffs (1992)Google Scholar
  7. 7.
    Fuller, M., Zobel, J.: Conflation-based Comparison of Stemming Algorithms. In: Proc. of the Third Australian Document Computing Symposium, Sydney, Australia (1998)Google Scholar
  8. 8.
    Hafer, M., Weiss, S.: Word Segmentation by Letter Succession Varieties. Information Storage and Retrieval 10 (1974)Google Scholar
  9. 9.
    Harman, D.: How Effective is Suffixing? Journal of the American Society for Information Science 42(1) (1991)Google Scholar
  10. 10.
    Hull, D.A.: Stemming Algorithms: A Case Study for Detailed Evaluation. Journal of the American Society for Information Science 47(1), 70–84 (1996)CrossRefGoogle Scholar
  11. 11.
    Kowaltolwski, T., Lucchesi, C., Stolfi, J.: Finite Automata and Efficient Lexicon Implementation. Technical Report, Instituto de Computação, Universidade de Campinas (1998)Google Scholar
  12. 12.
    Krovetz, R.: Viewing morphology as an inference process. In: ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 191-202 (1993)Google Scholar
  13. 13.
    Lovins, J. B.: Development of a Stemming Algorithm. Mechanical Translation and Computational Linguistics 11(1&2) (1968) Google Scholar
  14. 14.
    Pacheco, H.: Uma Ferramenta de Auxílio à Redação. MsC Dissertation. Universidade Federal de Minas Gerais (1996) Google Scholar
  15. 15.
    Paice, C.: Another Stemmer. ACM Sigir Forum 24(3) (1990)Google Scholar
  16. 16.
    Paice, C.: An Evaluation Method for Stemming Algorithms. In: ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 42–50 (1994)Google Scholar
  17. 17.
    Porter, M.: An Algorithm for Suffix Stripping. Program 14(3) (1980) Google Scholar
  18. 18.
    Pequeno Vocabulário Ortográfico da Língua Portuguesa. Academia Brasileira de Letras, Rio de Janeiro (1999)Google Scholar
  19. 19.
    Revuz, D.: Minimisation of acyclic deterministic automata in linear time. Theoretical Computer Science 92 (1992)Google Scholar
  20. 20.
    Salton, G.: Automatic Information Organization and Retrieval. McGraw Hill, New York (1968)Google Scholar
  21. 21.

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Gilberto Silva
    • 1
  • Claudia Oliveira
    • 2
  1. 1.Datasus – Centro de Tecnologia da Informação do Ministério da SaúdeRio de JaneiroBrazil
  2. 2.Departamento de Engenharia de ComputaçãoInstituto Militar de EngenhariaRio de JaneiroBrazil

Personalised recommendations