Advertisement

An Empirical Comparison of Text Categorization Methods

  • Ana Cardoso-Cachopo
  • Arlindo L. Oliveira
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2857)

Abstract

In this paper we present a comprehensive comparison of the performance of a number of text categorization methods in two different data sets. In particular, we evaluate the Vector and Latent Semantic Analysis (LSA) methods, a classifier based on Support Vector Machines (SVM) and the k-Nearest Neighbor variations of the Vector and LSA models.

We report the results obtained using the Mean Reciprocal Rank as a measure of overall performance, a commonly used evaluation measure for question answering tasks. We argue that this evaluation measure is also very well suited for text categorization tasks.

Our results show that overall, SVMs and k-NN LSA perform better than the other methods, in a statistically significant way.

Keywords

Support Vector Machine Information Retrieval Latent Semantic Analysis Vector Model Question Answering 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley, Reading (1999)Google Scholar
  2. 2.
    Berger, A., Caruana, R., Cohn, D., Freitag, D., Mittal, V.O.: Bridging the lexical chasm: statistical approaches to answer-finding. In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 2000, Athens, Greece, pp. 192–199 (2000)Google Scholar
  3. 3.
    Burges, C.J.C.: A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery 2(2), 121–167 (1998)CrossRefGoogle Scholar
  4. 4.
    Caron, J.: Experiments with LSA scoring: Optimal rank and basis. In: Proceedings of SIAM Computational Information Retrieval Workshop, Raleigh, NC, USA (October 2000)Google Scholar
  5. 5.
    Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines (2001), Software, available at http://www.csie.ntu.edu.tw/~cjlin/libsvm
  6. 6.
    Cooper, W.S.: Expected search length: a single measure of retrieval effectiveness based on weak ordering action of retrieval systems. Journal of the American Society for Information Science 19(1), 30–41 (1968)CrossRefGoogle Scholar
  7. 7.
    Cortes, C., Vapnik, V.: Support-vector networks. Machine Learning 20(3), 273–297 (1995)zbMATHGoogle Scholar
  8. 8.
    Creecy, R.M., Masand, B.M., Smith, S.J., Waltz, D.L.: Trading MIPS and memory for knowledge engineering: classifying census returns on the Connection Machine. Communications of the ACM 39(1), 48–63 (1996)Google Scholar
  9. 9.
    Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines. Cambridge University Press, Cambridge (2000)Google Scholar
  10. 10.
    Deerwester, S.C., Dumais, S.T., Landauer, T.K., Furnas, G.W., Harshman, R.A.: Indexing by latent semantic analysis. Journal of the American Society for Information Science 41(6), 391–407 (1990)CrossRefGoogle Scholar
  11. 11.
    Furnas, G.W., Deerwester, S.C., Dumais, S.T., Landauer, T.K., Harshman, R.A., Streeter, L.A., Lochbaum, K.E.: Information retrieval using a singular value decomposition model of latent semantic structure. In: Proceedings of the 11th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Grassau, France, June 1988, pp. 465–480 (1988)Google Scholar
  12. 12.
    Joachims, T.: Text categorization with support vector machines: learning with many relevant features. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, pp. 137–142. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  13. 13.
    Joachims, T.: Transductive inference for text classification using support vector machines. In: Bratko, I., Dzeroski, S. (eds.) Proceedings of the 16th International Conference on Machine Learning, Bled, Slovenia, 1999, pp. 200–209. Morgan Kaufmann Publishers, Inc., San Francisco (1999)Google Scholar
  14. 14.
    Jones, K.S., Willett, P. (eds.): Readings in Information Retrieval. Morgan Kaufmann Publishers, Inc., Los Altos (1997)Google Scholar
  15. 15.
    Masand, B., Linoff, G., Waltz, D.: Classifying news stories using memory-based reasoning. In: Belkin, N.J., Ingwersen, P., Pejtersen, A.M. (eds.) Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Copenhagen, Denmark, pp. 59–65. ACM Press, New York (1992)CrossRefGoogle Scholar
  16. 16.
    Salton, G.: The SMART Retrieval System. Prentice-Hall, Inc., New Jersey (1971)Google Scholar
  17. 17.
    Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Information Processing and Management 24(5), 513–523 (1988); Also reprinted in [14, pp. 323–328]CrossRefGoogle Scholar
  18. 18.
    Salton, G., Lesk, M.: Computer evaluation of indexing and text processing. Journal of the ACM 15(1), 8–36 (1968); Also reprinted in [14, pp. 60–84]zbMATHCrossRefGoogle Scholar
  19. 19.
    Sebastiani, F.: Machine learning in automated text categorization. ACM. Computing Surveys 34(1), 1–47 (2002)CrossRefGoogle Scholar
  20. 20.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Heidelberg (1995)zbMATHGoogle Scholar
  21. 21.
    Voorhees, E.M.: The TREC-8 question answering track report. In: Voorhees, E.M., Harman, D.K. (eds.) Proceedings of the 8th Text Retrieval Conference, Gaithersburg, Maryland, USA, November 1999, pp. 77–82 (1999)Google Scholar
  22. 22.
    Yang, Y.: Expert network: effective and efficient learning from human decisions in text categorisation and retrieval. In: Bruce Croft, W., van Rijsbergen, C.J. (eds.) Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland, pp. 13–22. Springer, Heidelberg (1994)Google Scholar
  23. 23.
    Yang, Y., Liu, X.: A re-examination of text categorization methods. In: Hearst, M.A., Gey, F., Tong, R. (eds.) Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Berkeley, CA, USA, August 1999, pp. 42–49 (1999)Google Scholar
  24. 24.
    Yang, Y., Pedersen, J.O.: A comparative study on feature selection in text categorization. In: Fisher, D.H. (ed.) Proceedings of the 14th International Conference on Machine Learning, Nashville, TN, USA, pp. 412–420. Morgan Kaufmann Publishers, Inc., San Francisco (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Ana Cardoso-Cachopo
    • 1
    • 2
  • Arlindo L. Oliveira
    • 1
    • 2
  1. 1.Departamento de Engenharia InformáticaInstituto Superior TécnicoLisboaPortugal
  2. 2.INESC-ID / ISTLisboaPortugal

Personalised recommendations