Abstract
Word embedding, which represents individual words with semantically rich numerical vectors, has made it possible to successfully apply deep learning to NLP tasks such as semantic role modeling, question answering, and machine translation. As math text consists of natural text as well as math expressions that similarly exhibit linear correlation and contextual characteristics, word embedding can be applied to math documents as well. On the other hand, math terms also show characteristics (e.g., abstractions) that are different from textual words. Accordingly, it is worthwhile to explore the use and effectiveness of word embedding in math language processing and MKM.
In this paper, we present exploratory investigations of math embedding by testing it on some basic tasks such as (1) math-term similarity, (2) analogy, (3) basic numerical concept-modeling using a novel approach based on computing the (weighted) centroid of the keywords that characterize a concept, and (4) math search, especially query expansion using the weighted centroid of the query keywords and then expanding the query with new keywords that are most similar to the centroid. Due to lack of benchmarks, our investigations were done using carefully selected illustrations on the DLMF. We draw from our investigations some general observations and lessons that form a trajectory for future statistically significant testing on large benchmarks. Our preliminary results and observations show that math embedding holds much promise but also point to the need for more robust embedding.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bordes, A., et al.: Joint learning of words and meaning representations for open-text semantic parsing. In: AISTATS (2012)
Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. In: EMNLP (2015)
Cer, D., et al.: Universal sentence encoder. CoRR arXiv:1803.11175 (2018)
Chelba, C., et al.: One billion word benchmark for measuring progress in statistical language modeling. In: INTERSPEECH (2014)
Chiu, J., Nichols, E.: Named entity recognition with bidirectional LSTM-CNNs. Trans. Assoc. Comput. Linguist. 4, 357–370 (2016)
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: EMNLP (2014)
Clark, C., Gardner, M.: Simple and effective multi-paragraph reading comprehension. In: ACL 2018, Melbourne, Australia, 15–20 July 2018, pp. 845–855 (2018)
Chen, Q., et al.: Enhanced LSTM for natural language inference. In: ACL (2017)
Devlin, J., et al.: Fast and robust neural network joint models for statistical machine translation. In: Proceedings of the ACL (2014)
Gao, L., et al.: Preliminary exploration of formula embedding for mathematical information retrieval: can mathematical formulae be embedded like a natural language? arXiv:1707.05154 (2017)
Gong, Y., Luo, H., Zhang, J.: Natural language inference over interaction space. In: ICLR (2018)
He, L., Lee, K., Lewis, M., Zettlemoyer, L.S.: Deep semantic role labeling: what works and what’s next. In: ACL (2017)
Iacobacci, I., Pilehvar, M.T., Navigli, R.: Embeddings for word sense disambiguation: an evaluation study. In: ACL (2016)
Kim, Y.: Convolutional neural networks for sentence classification. In: EMNLP, Doha, Qatar, pp. 1746–1751, October 2014
Krishna, A., Youssef, A., et al.: Query Expansion for Patent Searching using Word Embedding and Professional Crowdsourcing (in submission)
Kstovski, K., Blei, D.M.: Equation embeddings. arXiv:1803.09123, March 2018
Lai, S., Liu, K., He, S., Zhao, J.: How to generate a good word embedding. IEEE Intell. Syst. 31(6), 5–14 (2016)
Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: International Conference on Machine Learning, pp. 1188–1196, January 2014
Lee, K., He, L., Lewis, M., Zettlemoyer, L.S.: End-to-end neural coreference resolution. In: EMNLP (2017)
Liu, X., Shen, Y., Duh, K., Gao, J.-F.: Stochastic answer networks for machine reading comprehension. arXiv:1712.03556
Mikolov, T., et al.: Efficient estimation of word representations in vector space. In: Workshops Track, International Conference on Learning Representations (2013)
Mikolov, T., et al.: Distributed representations of words and phrases and their compositionality. In: NIPS, pp. 3111–3119 (2013)
Nickel, M., Kiela, D.: Poincare embeddings for learning hierarchical representations. In: Advances in NIPS (2017)
Olver, F.W.J., et al., (eds.): NIST Digital Library of Mathematical Functions. https://dlmf.nist.gov/, Release 1.0.18 of 27-03-2018
Palmer, M., et al.: The proposition bank: an annotated corpus of semantic roles. Comput. Linguist. 31, 71–106 (2005)
Piotr, B., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. ACL 5, 135–146 (2017)
Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of NAACL-HLT, pp. 2227–2237 (2018)
Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. InL EMNLP, 25–29 October 2014, pp. 1532–1543 (2014)
Raganato, A., Bovi, C.D., Navigli, R.: Neural sequence learning models for word sense disambiguation. In: EMNLP (2017)
Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: EMNLP (2016)
Rudolph, M., Ruiz, F., Athey, S., Blei, D.M.: Structured embedding models for grouped data. In: NIPS, pp. 250–260 (2017)
Socher, R., et al.: Recursive deep models for semantic compositionality over a sentiment treebank. In: EMNLP (2013)
Wiseman, S., Rush, A.M., Shieber, S.M.: Learning global features for coreference resolution. In: HLT-NAACL (2016)
Zhou, J., Xu, W.: End-to-end learning of semantic role labeling using recurrent neural networks. In: ACL (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 This is a U.S. government work and not under copyright protection in the United States; foreign copyright protection may apply
About this paper
Cite this paper
Youssef, A., Miller, B.R. (2019). Explorations into the Use of Word Embedding in Math Search and Math Semantics. In: Kaliszyk, C., Brady, E., Kohlhase, A., Sacerdoti Coen, C. (eds) Intelligent Computer Mathematics. CICM 2019. Lecture Notes in Computer Science(), vol 11617. Springer, Cham. https://doi.org/10.1007/978-3-030-23250-4_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-23250-4_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-23249-8
Online ISBN: 978-3-030-23250-4
eBook Packages: Computer ScienceComputer Science (R0)