Skip to main content
SpringerLink
Log in

An enhanced short text categorization model with deep abundant representation

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Short text categorization is a crucial issue to many applications, e.g., Information Retrieval, Question-Answering System, MRI Database Construction and so forth. Many researches focus on data sparsity and ambiguity issues in short text categorization. To tackle these issues, we propose a novel short text categorization strategy based on abundant representation, which utilizes Bi-directional Recurrent Neural Network(Bi-RNN) with Long Short-Term Memory(LSTM) and topic model to catch more contextual and semantic information. Bi-RNN enriches contextual information, and topic model discovers more latent semantic information for abundant text representation of short text. Experimental results demonstrate that the proposed model is comparable to state-of-the-art neural network models and method proposed is effective.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7

Similar content being viewed by others

References

  1. Azhagusundari, B., Thanamani, D.A.S.: Feature selection based on information gain. International Journal of Innovative Technology &, Exploring Engineering 2(2), 18–21 (2013)

    Google Scholar 

  2. Bengio, Y., Schwenk, H., Senécal, J.S., Morin, F., Gauvain, J.L.: Neural probabilistic language models. Springer, Berlin (2006)

    Book  Google Scholar 

  3. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)

    MATH  Google Scholar 

  4. Ceri, S., Bozzon, A., Brambilla, M., Valle, E.D., Fraternali, P., Quarteroni, S.: An introduction to information retrieval. Web Information Retrieval, Springer, Berlin 2013, 96–102 (2013)

    MATH  Google Scholar 

  5. Chen, M., Jin, X., Shen, D.: Short text classification improved by learning multi-granularity topics. In: The 22Nd international joint conference on artificial intelligence, IJCAI 2011, Barcelona, July 16-22, pp 1776–1781 (2011)

  6. Collobert, R., Weston, J.: A unified architecture for natural language processing: deep neural networks with multitask learning. In: Machine learning, proceedings of the 25Th international conference, ICML 2008, Helsinki, June 5-9, pp 160–167 (2008)

  7. Ghahramani, Z.: An introduction to hidden markov models and bayesian networks. IJPRAI 15(1), 9–42 (2001)

    Google Scholar 

  8. Graves, A., Mohamed, A., Hinton, G.E.: Speech recognition with deep recurrent neural networks. In: IEEE international conference on acoustics, speech and signal processing, ICASSP 2013, Vancouver, May 26-31, pp 6645–6649 (2013)

  9. Han, E., Karypis, G., Kumar, V.: Text categorization using weight adjusted K-Nearest neighbor classification. In: The 5Th Pacific-Asia conference on knowledge discovery and data mining, PAKDD 2001, Hong Kong, April 16-18, pp 53–65 (2001)

  10. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  11. Hochreiter, S., Younger, A.S., Conwell, P.R.: Learning to learn using gradient descent. In: International conference on artificial neural networks, ICANN 2001, Vienna, August 21-25, pp 87–94 (2001)

  12. Hüsken, M., Stagge, P.: Recurrent neural networks for time series classification. Neurocomputing 50, 223–235 (2003)

    Article  Google Scholar 

  13. Joachims, T.: Text categorization with support vector machines: learning with many relevant features. In: The 10Th European conference on machine learning, ECML 1998, Chemnitz, April 21-23, pp 137–142 (1998)

    Chapter  Google Scholar 

  14. Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: The 52Nd annual meeting of the association for computational linguistics, ACL 2014, June 22-27, Baltimore, vol. 1: Long Papers, pp 655–665 (2014)

  15. Karbassi, A., Mohebi, B., Rezaee, S., Lestuzzi, P.: Damage prediction for regular reinforced concrete buildings using the decision tree algorithm. Comput. Struct. 130(1), 46–56 (2014)

    Article  Google Scholar 

  16. Kim, Y.: Convolutional neural networks for sentence classification. In: The 2014 conference on empirical methods in natural language processing, EMNLP 2014, Doha, October 25-29, pp 1746–1751 (2014)

  17. Lauer, F., Bloch, G.: Incorporating prior knowledge in support vector machines for classification: a review. Neurocomputing 71(7-9), 1578–1594 (2008)

    Article  Google Scholar 

  18. Le, Q.V., Mikolov, T.: Distributed representations of sentences and documents. In: The 31Th international conference on machine learning, ICML 2014, Beijing, June 21-26, pp 1188–1196 (2014)

  19. Lee, J.Y., Dernoncourt, F.: Sequential short-text classification with recurrent and convolutional neural networks. In: The 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, NAACL HLT 2016, San Diego, June 12-17, pp 515–520 (2016)

  20. Li, J., Cai, Y., Cai, Z., Leung, H., Yang, K.: Wikipedia based short text classification method. In: Database systems for advanced applications - DASFAA 2017 international workshops: BDMS, BDQM, SeCoP, and DMMOOC, Suzhou, March 27-30, pp 275–286 (2017)

    Chapter  Google Scholar 

  21. Li, J., Rao, Y., Jin, F., Chen, H., Xiang, X.: Multi-label maximum entropy model for social emotion classification over short text. Neurocomputing 210, 247–256 (2016)

    Article  Google Scholar 

  22. Li, L., Zhong, L., Xu, G., Kitsuregawa, M.: A feature-free search query classification approach using semantic distance. Expert Systems with Applications 39 (12), 10,739–10,748 (2012)

    Article  Google Scholar 

  23. Li, X., Roth, D.: Learning question classifiers. In: 19Th international conference on computational linguistics, COLING 2002, Taipei, August 24 - September 1, pp 556–562 (2002)

  24. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv:1301.3781 (2013)

  25. Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., Khudanpur, S.: Recurrent neural network based language model. In: The 11Th annual conference of the international speech communication association, INTERSPEECH 2010, Makuhari, September 26-30, pp 1045–1048 (2010)

  26. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: The 27Th annual conference on neural information processing systems, Lake Tahoe, Nevada, December 5-8, pp 3111–3119 (2013)

  27. Nikolentzos, G., Meladianos, P., Rousseau, F., Vazirginannis, M., Stavrakas, Y.: Multivariate gaussian document representation from word embeddings for text categorization. In: European chapter of the association for computational linguistics, EACL 2017, Barcelona, April 3-7, pp 450–355 (2017)

  28. Paccanaro, A., Hinton, G.E.: Learning distributed representations of concepts using linear relational embedding. IEEE Trans. Knowl. Data Eng. 13(2), 232–244 (2001)

    Article  Google Scholar 

  29. Papadakis, G., Giannakopoulos, G., Paliouras, G.: Graph vs. bag representation models for the topic classification of Web documents. World Wide Web 19(5), 887–920 (2016)

    Article  Google Scholar 

  30. Phan, X.H., Nguyen, M.L., Horiguchi, S.: Learning to classify short and sparse text & Web with hidden topics from large-scale data collections. In: The 17Th international conference on World Wide Web, WWW 2008, Beijing, April 21-25, pp 91–100 (2008)

  31. Salton, G., Wong, A., Yang, C.: A vector space model for automatic indexing. Commun. ACM 18(11), 613–620 (1975)

    Article  Google Scholar 

  32. Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)

    Article  Google Scholar 

  33. Socher, R., Lin, C.C., Ng, A.Y., Manning, C.D.: Parsing natural scenes and natural language with recursive neural networks. In: The 28Th international conference on machine learning, ICML 2011, Bellevue, June 28 - July 2, pp 129–136 (2011)

  34. Sriram, B., Fuhry, D., Demir, E., Ferhatosmanoglu, H., Demirbas, M.: Short text classification in twitter to improve information filtering. In: The 33Rd international ACM SIGIR conference on research and development in information retrieval, SIGIR 2010, Geneva, July 19-23, pp 841–842 (2010)

  35. Toh, K., Lu, J., Yau, W.: Global feedforward neural network learning for classification and regression. In: Energy minimization methods in computer vision and pattern recognition, third international workshop, EMM-CVPR 2001, Sophia Antipolis, September 3-5, pp 407–422 (2001)

    Google Scholar 

  36. Troussas, C., Virvou, M., Espinosa, K.J., Llaguno, K., Caro, J.: Sentiment analysis of facebook statuses using naive bayes classifier for language learning. In: The 4Th international conference on information, intelligence, systems and applications, IISA 2013, Piraeus, July 10-12, pp 1–6 (2013)

  37. Wang, P., Xu, B., Xu, J., Tian, G., Liu, C., Hao, H.: Semantic expansion using word embedding clustering and convolutional neural network for improving short text classification. Neurocomputing 174, 806–814 (2016)

    Article  Google Scholar 

  38. Wang, P., Xu, J., Xu, B., Liu, C., Zhang, H., Wang, F., Hao, H.: Semantic clustering and convolutional neural network for short text categorization. In: The 53Rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing of the asian federation of natural language processing, ACL 2015, Beijing, vol. 2: Short Papers, July 26-31, pp 352–357 (2015)

  39. Wu, Z., Zhu, H., Li, G., Cui, Z., Huang, H., Li, J., Chen, E., Xu, G.: An efficient wikipedia semantic matching approach to text document classification. Inform. Sci. 393, 15–28 (2017)

    Article  MathSciNet  Google Scholar 

  40. Yan, X., Guo, J., Lan, Y., Cheng, X.: A Biterm topic model for short texts. In: The 22Nd international World Wide Web conference, WWW 2013, Rio De Janeiro, May 13-17, pp 1445–1456 (2013)

  41. Yang, Y., Pedersen, J.O.: A comparative study on feature selection in text categorization. In: The 14Th international conference on machine learning, ICML 1997, Nashville, July 8-12, pp 412–420 (1997)

  42. Yao, L., Sheng, Q.Z., Ngu, A.H.H., Gao, B.J., Li, X., Wang, S.: Multi-label classification via learning a unified object-label graph with sparse representation. World Wide Web 19(6), 1125–1149 (2016)

    Article  Google Scholar 

  43. Zhang, Y., Dong, Z., Wu, L., Wang, S.: A hybrid method for MRI brain image classification. Expert Systems with Applications 38(8), 10,049–10,053 (2011)

    Article  Google Scholar 

  44. Zheng, W., Tang, H., Qian, Y.: Collaborative work with linear classifier and extreme learning machine for fast text categorization. World Wide Web 18(2), 235–252 (2015)

    Article  Google Scholar 

  45. Zhou, C., Sun, C., Liu, Z., Lau, F.C.M.: A c-LSTM neural network for text classification. arXiv:1511.08630 (2015)

Download references

Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments. This work is partially supported by National Natural Science Foundation of China under Grant 41571382, U1636116, 11431006, 61472191, 61772278, the Natural Science Research of Jiangsu Higher Education Institutions of China under Grant 15KJA420001, and the Research Fund for International Young Scientists under Grant 61650110510.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Nanjing Normal University, Nanjing, China

    Yanhui Gu, Min Gu, Junsheng Zhou & Weiguang Qu

  2. School of Geography Science, Nanjing Normal University, Nanjing, China

    Yi Long

  3. Advanced Analytics Institute, University of Technology Sydney, Sydney, Australia

    Guandong Xu

  4. Institute of Big Data, College of Computer and Control Engineering, Institute of Statistics, Nankai University, Tianjin, China

    Zhenglu Yang

Authors
  1. Yanhui Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Min Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi Long
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guandong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhenglu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Junsheng Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Weiguang Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Min Gu, Zhenglu Yang or Weiguang Qu.

Additional information

This article belongs to the Topical Collection: Special Issue on Deep Mining Big Social Data

Guest Editors: Xiaofeng Zhu, Gerard Sanroma, Jilian Zhang, and Brent C. Munsell

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gu, Y., Gu, M., Long, Y. et al. An enhanced short text categorization model with deep abundant representation. World Wide Web 21, 1705–1719 (2018). https://doi.org/10.1007/s11280-018-0542-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-018-0542-9

Keywords

Search

Navigation