Abstract
This work investigates a method for enriching pre-trained word embeddings with domain-specific information using a small, custom word embedding. For a classification task on text containing out-of-vocabulary expert jargon, this new approach improves the prediction accuracy when using popular models such as Word2Vec (71.5% to 76.6%), GloVe (73.5% to 77.2%), and fastText (75.8% to 79.6%). Furthermore, an analysis of the approach demonstrates that expert knowledge is improved in terms of discrimination and inconsistency. Another advantage of this word embedding augmentation technique is that it is computationally inexpensive and leverages the general syntactic information encoded in large pre-trained word embeddings.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The code and training data for the best performing approach described in this work (fastText pre-trained model with fastText-based custom embedding) are available at: https://github.com/lemay-ai/sidecar/.
- 2.
Google News Vectors Binary File (2019) for Word2Vec can be found at https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit.
- 3.
More information on this can be found at https://www.auditmap.ai/.
- 4.
- 5.
- 6.
For some interesting background on programming jargon in particular, see http://catb.org/jargon/html/distinctions.html.
- 7.
- 8.
- 9.
- 10.
References
Anders Ericsson, K., Charness, N.: Expert performance: its structure and acquisition. Am. Psychol. 49, 725–747 (1994)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606 (2016)
Dong, J., Huang, J.: Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus. CoRR abs/1802.02614 (2018). http://arxiv.org/abs/1802.02614
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
Kiros, R., Zhu, Y., Salakhutdinov, R.R., Zemel, R., Urtasun, R., Torralba, A., Fidler, S.: Skip-thought vectors. In: Advances in Neural Information Processing Systems, pp. 3294–3302 (2015)
Limsopatham, N., Collier, N.: Modelling the combination of generic and target domain embeddings in a convolutional neural network for sentence classification. Association for Computational Linguistics (2016)
McBride, M.F., Burgman, M.A.: What is expert knowledge, how is such knowledge gathered, and how do we use it to address questions in landscape ecology? In: Perera, A., Drew, C., Johnson, C. (eds.) Expert Knowledge and Its Application in Landscape Ecology, pp. 11–38. Springer, New York (2012)
Merriam-Webster Online: Merriam-Webster Online Dictionary (2009). http://www.merriam-webster.com
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014). http://www.aclweb.org/anthology/D14-1162
Saal, F.E., Downey, R.G., Lahey, M.A.: Rating the ratings: assessing the psychometric quality of rating data. Psychol. Bull. 88(2), 413 (1980)
Shanteau, J., Weiss, D., Thomas, R., Pounds, J.: Performance-based assessment of expertise: how to decide if someone is an expert or not. Eur. J. Oper. Res. 136(2), 253–263 (2002)
Shapiro, D.: Composing recommendations using computer screen images: a deep learning recommender system for PC users. Ph.D. thesis, Université d’Ottawa/University of Ottawa (2017)
Shapiro, D., Qassoud, H., Lemay, M., Bolic, M.: Visual deep learning recommender system for personal computer users. In: The Second International Conference on Applications and Systems of Visual Paradigms, VISUAL 2017, pp. 1–10 (2017). https://www.thinkmind.org/index.php?view=article&articleid=visual_2017_1_10_70006
Speer, R., Chin, J.: An ensemble method to produce high-quality word embeddings. arXiv preprint arXiv:1604.01692 (2016)
Xu, C., Bai, Y., Bian, J., Gao, B., Wang, G., Liu, X., Liu, T.Y.: RC-NET: a general framework for incorporating knowledge into word representations. In: Proceedings of the 23rd ACM International Conference on Information and Knowledge Management, pp. 1219–1228. ACM (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Lemay, M., Shapiro, D., MacPherson, M.K., Yee, K., Qassoud, H., Bolic, M. (2020). Sidecar: Augmenting Word Embedding Models with Expert Knowledge. In: Arai, K., Kapoor, S., Bhatia, R. (eds) Advances in Information and Communication. FICC 2020. Advances in Intelligent Systems and Computing, vol 1130. Springer, Cham. https://doi.org/10.1007/978-3-030-39442-4_39
Download citation
DOI: https://doi.org/10.1007/978-3-030-39442-4_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-39441-7
Online ISBN: 978-3-030-39442-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)