An enhanced short text categorization model with deep abundant representation

  • Yanhui Gu
  • Min Gu
  • Yi Long
  • Guandong Xu
  • Zhenglu Yang
  • Junsheng Zhou
  • Weiguang Qu
Part of the following topical collections:
  1. Special Issue on Deep Mining Big Social Data


Short text categorization is a crucial issue to many applications, e.g., Information Retrieval, Question-Answering System, MRI Database Construction and so forth. Many researches focus on data sparsity and ambiguity issues in short text categorization. To tackle these issues, we propose a novel short text categorization strategy based on abundant representation, which utilizes Bi-directional Recurrent Neural Network(Bi-RNN) with Long Short-Term Memory(LSTM) and topic model to catch more contextual and semantic information. Bi-RNN enriches contextual information, and topic model discovers more latent semantic information for abundant text representation of short text. Experimental results demonstrate that the proposed model is comparable to state-of-the-art neural network models and method proposed is effective.


Short text categorization Topic model Bi-directional LSTM 



We would like to thank the anonymous reviewers for their insightful comments. This work is partially supported by National Natural Science Foundation of China under Grant 41571382, U1636116, 11431006, 61472191, 61772278, the Natural Science Research of Jiangsu Higher Education Institutions of China under Grant 15KJA420001, and the Research Fund for International Young Scientists under Grant 61650110510.


  1. 1.
    Azhagusundari, B., Thanamani, D.A.S.: Feature selection based on information gain. International Journal of Innovative Technology &, Exploring Engineering 2(2), 18–21 (2013)Google Scholar
  2. 2.
    Bengio, Y., Schwenk, H., Senécal, J.S., Morin, F., Gauvain, J.L.: Neural probabilistic language models. Springer, Berlin (2006)CrossRefGoogle Scholar
  3. 3.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)zbMATHGoogle Scholar
  4. 4.
    Ceri, S., Bozzon, A., Brambilla, M., Valle, E.D., Fraternali, P., Quarteroni, S.: An introduction to information retrieval. Web Information Retrieval, Springer, Berlin 2013, 96–102 (2013)zbMATHGoogle Scholar
  5. 5.
    Chen, M., Jin, X., Shen, D.: Short text classification improved by learning multi-granularity topics. In: The 22Nd international joint conference on artificial intelligence, IJCAI 2011, Barcelona, July 16-22, pp 1776–1781 (2011)Google Scholar
  6. 6.
    Collobert, R., Weston, J.: A unified architecture for natural language processing: deep neural networks with multitask learning. In: Machine learning, proceedings of the 25Th international conference, ICML 2008, Helsinki, June 5-9, pp 160–167 (2008)Google Scholar
  7. 7.
    Ghahramani, Z.: An introduction to hidden markov models and bayesian networks. IJPRAI 15(1), 9–42 (2001)Google Scholar
  8. 8.
    Graves, A., Mohamed, A., Hinton, G.E.: Speech recognition with deep recurrent neural networks. In: IEEE international conference on acoustics, speech and signal processing, ICASSP 2013, Vancouver, May 26-31, pp 6645–6649 (2013)Google Scholar
  9. 9.
    Han, E., Karypis, G., Kumar, V.: Text categorization using weight adjusted K-Nearest neighbor classification. In: The 5Th Pacific-Asia conference on knowledge discovery and data mining, PAKDD 2001, Hong Kong, April 16-18, pp 53–65 (2001)Google Scholar
  10. 10.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  11. 11.
    Hochreiter, S., Younger, A.S., Conwell, P.R.: Learning to learn using gradient descent. In: International conference on artificial neural networks, ICANN 2001, Vienna, August 21-25, pp 87–94 (2001)Google Scholar
  12. 12.
    Hüsken, M., Stagge, P.: Recurrent neural networks for time series classification. Neurocomputing 50, 223–235 (2003)CrossRefzbMATHGoogle Scholar
  13. 13.
    Joachims, T.: Text categorization with support vector machines: learning with many relevant features. In: The 10Th European conference on machine learning, ECML 1998, Chemnitz, April 21-23, pp 137–142 (1998)Google Scholar
  14. 14.
    Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: The 52Nd annual meeting of the association for computational linguistics, ACL 2014, June 22-27, Baltimore, vol. 1: Long Papers, pp 655–665 (2014)Google Scholar
  15. 15.
    Karbassi, A., Mohebi, B., Rezaee, S., Lestuzzi, P.: Damage prediction for regular reinforced concrete buildings using the decision tree algorithm. Comput. Struct. 130(1), 46–56 (2014)CrossRefGoogle Scholar
  16. 16.
    Kim, Y.: Convolutional neural networks for sentence classification. In: The 2014 conference on empirical methods in natural language processing, EMNLP 2014, Doha, October 25-29, pp 1746–1751 (2014)Google Scholar
  17. 17.
    Lauer, F., Bloch, G.: Incorporating prior knowledge in support vector machines for classification: a review. Neurocomputing 71(7-9), 1578–1594 (2008)CrossRefGoogle Scholar
  18. 18.
    Le, Q.V., Mikolov, T.: Distributed representations of sentences and documents. In: The 31Th international conference on machine learning, ICML 2014, Beijing, June 21-26, pp 1188–1196 (2014)Google Scholar
  19. 19.
    Lee, J.Y., Dernoncourt, F.: Sequential short-text classification with recurrent and convolutional neural networks. In: The 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, NAACL HLT 2016, San Diego, June 12-17, pp 515–520 (2016)Google Scholar
  20. 20.
    Li, J., Cai, Y., Cai, Z., Leung, H., Yang, K.: Wikipedia based short text classification method. In: Database systems for advanced applications - DASFAA 2017 international workshops: BDMS, BDQM, SeCoP, and DMMOOC, Suzhou, March 27-30, pp 275–286 (2017)Google Scholar
  21. 21.
    Li, J., Rao, Y., Jin, F., Chen, H., Xiang, X.: Multi-label maximum entropy model for social emotion classification over short text. Neurocomputing 210, 247–256 (2016)CrossRefGoogle Scholar
  22. 22.
    Li, L., Zhong, L., Xu, G., Kitsuregawa, M.: A feature-free search query classification approach using semantic distance. Expert Systems with Applications 39 (12), 10,739–10,748 (2012)CrossRefGoogle Scholar
  23. 23.
    Li, X., Roth, D.: Learning question classifiers. In: 19Th international conference on computational linguistics, COLING 2002, Taipei, August 24 - September 1, pp 556–562 (2002)Google Scholar
  24. 24.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv:1301.3781 (2013)
  25. 25.
    Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., Khudanpur, S.: Recurrent neural network based language model. In: The 11Th annual conference of the international speech communication association, INTERSPEECH 2010, Makuhari, September 26-30, pp 1045–1048 (2010)Google Scholar
  26. 26.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: The 27Th annual conference on neural information processing systems, Lake Tahoe, Nevada, December 5-8, pp 3111–3119 (2013)Google Scholar
  27. 27.
    Nikolentzos, G., Meladianos, P., Rousseau, F., Vazirginannis, M., Stavrakas, Y.: Multivariate gaussian document representation from word embeddings for text categorization. In: European chapter of the association for computational linguistics, EACL 2017, Barcelona, April 3-7, pp 450–355 (2017)Google Scholar
  28. 28.
    Paccanaro, A., Hinton, G.E.: Learning distributed representations of concepts using linear relational embedding. IEEE Trans. Knowl. Data Eng. 13(2), 232–244 (2001)CrossRefGoogle Scholar
  29. 29.
    Papadakis, G., Giannakopoulos, G., Paliouras, G.: Graph vs. bag representation models for the topic classification of Web documents. World Wide Web 19(5), 887–920 (2016)CrossRefGoogle Scholar
  30. 30.
    Phan, X.H., Nguyen, M.L., Horiguchi, S.: Learning to classify short and sparse text & Web with hidden topics from large-scale data collections. In: The 17Th international conference on World Wide Web, WWW 2008, Beijing, April 21-25, pp 91–100 (2008)Google Scholar
  31. 31.
    Salton, G., Wong, A., Yang, C.: A vector space model for automatic indexing. Commun. ACM 18(11), 613–620 (1975)CrossRefzbMATHGoogle Scholar
  32. 32.
    Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)CrossRefGoogle Scholar
  33. 33.
    Socher, R., Lin, C.C., Ng, A.Y., Manning, C.D.: Parsing natural scenes and natural language with recursive neural networks. In: The 28Th international conference on machine learning, ICML 2011, Bellevue, June 28 - July 2, pp 129–136 (2011)Google Scholar
  34. 34.
    Sriram, B., Fuhry, D., Demir, E., Ferhatosmanoglu, H., Demirbas, M.: Short text classification in twitter to improve information filtering. In: The 33Rd international ACM SIGIR conference on research and development in information retrieval, SIGIR 2010, Geneva, July 19-23, pp 841–842 (2010)Google Scholar
  35. 35.
    Toh, K., Lu, J., Yau, W.: Global feedforward neural network learning for classification and regression. In: Energy minimization methods in computer vision and pattern recognition, third international workshop, EMM-CVPR 2001, Sophia Antipolis, September 3-5, pp 407–422 (2001)Google Scholar
  36. 36.
    Troussas, C., Virvou, M., Espinosa, K.J., Llaguno, K., Caro, J.: Sentiment analysis of facebook statuses using naive bayes classifier for language learning. In: The 4Th international conference on information, intelligence, systems and applications, IISA 2013, Piraeus, July 10-12, pp 1–6 (2013)Google Scholar
  37. 37.
    Wang, P., Xu, B., Xu, J., Tian, G., Liu, C., Hao, H.: Semantic expansion using word embedding clustering and convolutional neural network for improving short text classification. Neurocomputing 174, 806–814 (2016)CrossRefGoogle Scholar
  38. 38.
    Wang, P., Xu, J., Xu, B., Liu, C., Zhang, H., Wang, F., Hao, H.: Semantic clustering and convolutional neural network for short text categorization. In: The 53Rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing of the asian federation of natural language processing, ACL 2015, Beijing, vol. 2: Short Papers, July 26-31, pp 352–357 (2015)Google Scholar
  39. 39.
    Wu, Z., Zhu, H., Li, G., Cui, Z., Huang, H., Li, J., Chen, E., Xu, G.: An efficient wikipedia semantic matching approach to text document classification. Inform. Sci. 393, 15–28 (2017)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Yan, X., Guo, J., Lan, Y., Cheng, X.: A Biterm topic model for short texts. In: The 22Nd international World Wide Web conference, WWW 2013, Rio De Janeiro, May 13-17, pp 1445–1456 (2013)Google Scholar
  41. 41.
    Yang, Y., Pedersen, J.O.: A comparative study on feature selection in text categorization. In: The 14Th international conference on machine learning, ICML 1997, Nashville, July 8-12, pp 412–420 (1997)Google Scholar
  42. 42.
    Yao, L., Sheng, Q.Z., Ngu, A.H.H., Gao, B.J., Li, X., Wang, S.: Multi-label classification via learning a unified object-label graph with sparse representation. World Wide Web 19(6), 1125–1149 (2016)CrossRefGoogle Scholar
  43. 43.
    Zhang, Y., Dong, Z., Wu, L., Wang, S.: A hybrid method for MRI brain image classification. Expert Systems with Applications 38(8), 10,049–10,053 (2011)CrossRefGoogle Scholar
  44. 44.
    Zheng, W., Tang, H., Qian, Y.: Collaborative work with linear classifier and extreme learning machine for fast text categorization. World Wide Web 18(2), 235–252 (2015)CrossRefGoogle Scholar
  45. 45.
    Zhou, C., Sun, C., Liu, Z., Lau, F.C.M.: A c-LSTM neural network for text classification. arXiv:1511.08630 (2015)

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Computer Science and TechnologyNanjing Normal UniversityNanjingChina
  2. 2.School of Geography ScienceNanjing Normal UniversityNanjingChina
  3. 3.Advanced Analytics InstituteUniversity of Technology SydneySydneyAustralia
  4. 4.Institute of Big Data, College of Computer and Control Engineering, Institute of StatisticsNankai UniversityTianjinChina

Personalised recommendations