Improving Performance of NMT Using Semantic Concept of WordNet Synset

  • Fangxu Liu
  • JinAn XuEmail author
  • Gouyi Miao
  • Yufeng Chen
  • Yujie Zhang
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 954)


Neural machine translation (NMT) has shown promising progress in recent years. However, for reducing the computational complexity, NMT typically needs to limit its vocabulary scale to a fixed or relatively acceptable size, which leads to the problem of rare word and out-of-vocabulary (OOV). In this paper, we present that the semantic concept information of word can help NMT learn better semantic representation of word and improve the translation accuracy. The key idea is to utilize the external semantic knowledge base WordNet to replace rare words and OOVs with their semantic concepts of WordNet synsets. More specifically, we propose two semantic similarity models to obtain the most similar concepts of rare words and OOVs. Experimental results on 4 translation tasks (We verify the effectiveness of our method on four translation tasks, including English-to- German, German-to-English, English-to-Chinese and Chinese-to-English.) show that our method outperforms the baseline RNNSearch by 2.38–2.88 BLEU points. Furthermore, the proposed hybrid method by combining BPE and our proposed method can also gain 0.39–0.97 BLEU points improvement over BPE. Experiments and analysis presented in this study also demonstrate that the proposed method can significantly improve translation quality of OOVs in NMT.


NMT Semantic concept of synset Rare words Unknown words 



The research work has been supported by the National Nature Science Foundation of China (Contract 61370130, 61473294 and 61502149), and Beijing Natural Science Foundation under Grant No. 4172047, and the Fundamental Research Funds for the Central Universities (2015JBM033), and the International Science and Technology Cooperation Program of China under grant No. 2014DFA11350.


  1. 1.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: ICLR 2015 (2015)Google Scholar
  2. 2.
    Vaswani, A., et al.: Attention is all you need. CoRR abs/1706.03762 (2017)Google Scholar
  3. 3.
    Wu, Y., Schuster, M., Chen, Z., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation (2016)Google Scholar
  4. 4.
    Luong, M.T., Sutskever, I., Le, Q.V., et al.: Addressing the rare word problem in neural machine translation. Bull. Univ. Agric. Sci. Vet. Med. Cluj-Napoca. Vet. Med. 27(2), 82–86 (2014)Google Scholar
  5. 5.
    Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. Computer Science (2015)Google Scholar
  6. 6.
    Miller, G.A.: WordNet: a lexical database for English. Commun. ACM 38(11), 39–41 (1995)CrossRefGoogle Scholar
  7. 7.
    Palangi, H., Palangi, H., Deng, L., Shen, Y.: Deep sentence embedding using long short-term memory networks: analysis and application to information retrieval. IEEE/ACM Trans. Audio Speech Lang. Process. 24(4), 694–707 (2015)CrossRefGoogle Scholar
  8. 8.
    Papineni, K., Roukos, S., et al.: BLEU: a method for automatic valuation of machine translation. In: Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, July 2002, pp. 311–318 (2002)Google Scholar
  9. 9.
    Ziemski, M., Junczys-Dowmunt, M., Pouliquen, B.: The United Nations Parallel Corpus, Language Resources and Evaluation (LREC 2016), Portorož, Slovenia, May 2016 (2016)Google Scholar
  10. 10.
    Stolcke, A.: SRILM—an extensible language modeling toolkit. In: International Conference on Spoken Language Processing, pp. 901–904 (2002)Google Scholar
  11. 11.
    Li, X., Zhang, J., Zong, C.: Towards zero unknown word in neural machine translation. In: International Joint Conference on Artificial Intelligence, pp. 2852–2858. AAAI Press (2016)Google Scholar
  12. 12.
    Li, S., Xu, J., Miao, G., Zhang, Y., Chen, Y.: A semantic concept based unknown words processing method in neural machine translation. In: Huang, X., Jiang, J., Zhao, D., Feng, Y., Hong, Yu. (eds.) NLPCC 2017. LNCS (LNAI), vol. 10619, pp. 233–242. Springer, Cham (2018). Scholar
  13. 13.
    Sutskever, I., et al.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  • Fangxu Liu
    • 1
  • JinAn Xu
    • 1
    Email author
  • Gouyi Miao
    • 1
  • Yufeng Chen
    • 1
  • Yujie Zhang
    • 1
  1. 1.School of Computer and Information TechnologyBeijing Jiaotong UniversityBeijingChina

Personalised recommendations