Advertisement

Cached Embedding with Random Selection: Optimization Technique to Improve Training Speed of Character-Aware Embedding

  • Yaofei Yang
  • Hua-Ping ZhangEmail author
  • Linfang Wu
  • Xin Liu
  • Yangsen Zhang
Conference paper
  • 308 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12033)

Abstract

Embedding is widely used in most natural language processing. e.g., neural machine translation, text classification, text abstraction and sentiment analysis etc. Word-based embedding is faster and character-based embedding performs better. In this paper, we explore a way to combine these two embeddings to bridge the gap between word-based and character-based embedding in speed and performance. In the experiments and analysis of Hybrid Embedding, we found it’s difficult to make these two different embeddings generate the same embedding vector, but we still obtain a comparable result. According to the results of analysis, we explore a form of character-based embedding called Cached Embedding that can achieve almost the same performance and reduce the extra training time by almost half compared to character-based embedding.

Keywords

Cached Embedding Word embedding Char-aware embedding Time reduction Training speed Linguist Natural language processing 

Notes

Acknowledgements

This work was supported by National Science Foundation of China (Grant No. 61772075), National Science Foundation of China (Grant No. 61772081), Scientific Research Project of Beijing Educational Committee (Grant No. KM201711232022), Beijing Municipal Education Committee (Grant No. SZ20171123228), Beijing Institute of Computer Technology and Application (Grant by Extensible Knowledge Graph Construction Technique Project).

References

  1. 1.
    Cherry, C.A., Foster, G., Bapna, A., Firat, O., Macherey, W.: Revisiting character-based neural machine translation with capacity and compression. In: Empirical Methods in Natural Language Processing (2018)Google Scholar
  2. 2.
    Chung, J., Cho, K., Bengio, Y.: A character-level decoder without explicit segmentation for neural machine translation. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany (volume 1: Long Papers), pp. 1693–1703. Association for Computational Linguistics (2016)Google Scholar
  3. 3.
    Graves, A.: Generating sequences with recurrent neural networks. arXiv:1308.0850 [cs], August 2013
  4. 4.
    Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580 [cs]. July 2012
  5. 5.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  6. 6.
    Kiefer, J., Wolfowitz, J.: Stochastic estimation of the maximum of a regression function. Ann. Math. Stat. 23(3), 462–466 (1952)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Kim, Y., Jernite, Y., Sontag, D., Rush, A.M.: Character-aware neural language models. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI 2016, pp. 2741–2749. AAAI Press (2016)Google Scholar
  8. 8.
    Kudo, T.: Subword regularization: improving neural network translation models with multiple subword candidates. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia (vol. 1: Long Papers), pp. 66–75. Association for Computational Linguistics (2018)Google Scholar
  9. 9.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML 2010, pp. 807–814. Omnipress, USA (2010)Google Scholar
  10. 10.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (2002)Google Scholar
  11. 11.
    Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany (vol. 1: Long Papers), pp. 1715–1725. Association for Computational Linguistics, August 2016Google Scholar
  12. 12.
    Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv:1505.00387 [cs], May 2015
  13. 13.
    Zaremba, W., Sutskever, I., Vinyals, O.: Recurrent neural network regularization. arXiv:1409.2329 [cs], September 2014
  14. 14.
    Zhang, W., Feng, Y., Meng, F., You, D., Liu, Q.: Bridging the gap between training and inference for neural machine translation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4334–4343, July 2019Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Yaofei Yang
    • 2
  • Hua-Ping Zhang
    • 1
    Email author
  • Linfang Wu
    • 3
  • Xin Liu
    • 4
  • Yangsen Zhang
    • 2
  1. 1.Beijing Institute of TechnologyBeijingChina
  2. 2.Beijing Information Science and Technology UniversityBeijingChina
  3. 3.Hebei University of Science and TechnologyShijiazhuangChina
  4. 4.Beijing Institute of Information TechnologyBeijingChina

Personalised recommendations