Abstract
Most sentence embedding models typically represent each sentence only using word surface, which makes these models indiscriminative for ubiquitous homonymy and polysemy. In order to enhance discriminativeness, we employ concept conceptualization model to assign associated concepts for each sentence in the text corpus, and learn conceptual sentence embedding (CSE). Hence, the sentence representations are more expressive than some widely-used document representation models such as latent topic models, especially for short text. In the experiments, we evaluate the CSE models on two tasks, text classification and information retrieval. The experimental results show that the proposed models outperform typical sentence embedding models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Probase data is available at http://probase.msra.cn/dataset.aspx.
- 2.
- 3.
References
Harris, Z.S.: Distributional structure. Synth. Lang. Libr. 10, 146–162 (1954)
Le, Q., V., Mikolov, T.: Distributed representations of sentences and documents (2014). arXiv preprint arXiv:1405.4053
Liu, Y., Liu, Z., Chua, T.S., et al.: Topical word embeddings. In: Twenty-Ninth AAAI Conference on Artificial Intelligence (2015)
Ma, M., Huang, L., Xiang, B., et al.: Dependency-based convolutional neural networks for sentence embedding (2015)
Palangi, H., Deng, L., Shen, Y., et al.: Deep sentence embedding using long short-term memory networks. Arxiv. 24(4) 694–707 (2015)
Wieting, J., Bansal, M., Gimpel, K., et al.: Towards universal paraphrastic sentence embeddings (2015). arXiv preprint arXiv:1511.08198
Mikolov, T., Chen, K., Corrado, G., et al.: Efficient estimation of word representations in vector space (2013). arXiv preprint arXiv:1301.3781
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323(6088), 533–536 (1986)
Wang, M., Lu, Z., Li, H., et al.: Syntax-based deep matching of short texts (2015). arXiv preprint arXiv:1503.02427
Severyn, A., Moschitti, A., Tsagkias, M., et al.: A syntax-aware re-ranker for microblog retrieval. In: 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp 1067–1070. ACM Press (2014)
Wang, Z., Zhao, K., Wang, H., et al.: Query understanding through knowledge-based conceptualization. In: 24th International Joint Conference on Artificial Intelligence, pp. 3264–3270. AAAI Press (2015)
Song, Y., Wang, S., Wang, H.: Open domain short text conceptualization: a generative+descriptive modeling approach. In: 24th International Conference on Artificial Intelligence, pp. 3820–3826. AAAI Press (2015)
Morin, F., Bengio, Y.: Hierarchical probabilistic neural network language model. In: International Workshop on Artificial Intelligence and Statistics, pp. 246–252 (2005)
Mikolov, T., Sutskever, I., Chen, K., et al.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)
Huang, P.S., He, X., Gao, J., et al.: Learning deep structured semantic models for web search using clickthrough data. In: 22nd ACM International Conference on Conference on Information & Knowledge Management, pp. 2333–2338. ACM Press (2013)
Wu, W., Li, H., Wang, H., et al.: Probase: a probabilistic taxonomy for text understanding. In: 2012 ACM SIGMOD International Conference on Management of Data, pp. 481–492. ACM Press (2012)
Ounis, I., Macdonald, C., Lin, J., Soboroff, I.: Overview of the TREC-2011 microblog track. In: TREC 2011 (2011)
Soboroff, I., Ounis, I., Lin, J.: Overview of the TREC-2012 microblog track. In: TREC 2012 (2012)
Li, X., Roth, D.: Learning question classifiers. In: 19th International Conference on Computational Linguistics, pp. 1–7. Association for Computational Linguistics (2002)
Fan, R.E., Chang, K.W., Hsieh, C.J., et al.: LIBLINEAR: a library for large linear classification. J. Mach. Learn. Res. 9, 1871–1874 (2008)
Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, New York (1986)
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)
Smola, A., Narayanamurthy, S.: An architecture for parallel topic models. VLDB Endowment 3(1–2), 703–710 (2010)
Ahmed, A., Aly, M., Gonzalez, J., et al.: Scalable inference in latent variable models. In: Fifth ACM International Conference on Web Search and Data Mining, pp. 123–132. ACM Press (2012)
Acknowledgement
The work was supported by National Natural Science Foundation of China (Grant Nos. 61132009, 61201351), and National Hi-Tech Research & Development Program (863 Program, Grant No. 2015AA015404).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Wang, Y., Huang, H., Feng, C., Zhou, Q., Gu, J. (2016). Conceptual Sentence Embeddings. In: Cui, B., Zhang, N., Xu, J., Lian, X., Liu, D. (eds) Web-Age Information Management. WAIM 2016. Lecture Notes in Computer Science(), vol 9658. Springer, Cham. https://doi.org/10.1007/978-3-319-39937-9_30
Download citation
DOI: https://doi.org/10.1007/978-3-319-39937-9_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-39936-2
Online ISBN: 978-3-319-39937-9
eBook Packages: Computer ScienceComputer Science (R0)