Skip to main content
SpringerLink
Log in

Cross Aggregation of Multi-head Attention for Neural Machine Translation

  • Conference paper
  • First Online:
Natural Language Processing and Chinese Computing (NLPCC 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11838))

Abstract

Transformer based encoder has been the state-of-the-art model for the latest neural machine translation, which relies on the key design called self-attention. Multi-head attention of self-attention network (SAN) plays a significant role in extracting information of the given input from different subspaces among each pair of tokens. However, that information captured by each token on a specific head, which is explicitly represented by the attention weights, is independent from other heads and tokens, which means it does not take the global structure into account. Besides, since SAN does not apply an RNN-like network structure, its ability of modeling relative position and sequential information is weakened. In this paper, we propose a method named Cross Aggregation with an iterative routing-by-agreement algorithm to alleviate these problems. Experimental results on the machine translation task show that our method help the model outperform the strong Transformer baseline significantly.

This paper was partially supported by National Key Research and Development Program of China (No. 2017YFB0304100) and key projects of National Natural Science Foundation of China (No. U1836222 and No. 61733011).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015) (2015)

    Google Scholar 

  2. Cai, D., Zhao, H.: Neural word segmentation learning for Chinese. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), pp. 409–420 (2016)

    Google Scholar 

  3. Cai, D., Zhao, H., Zhang, Z., Xin, Y., Wu, Y., Huang, F.: Fast and accurate neural word segmentation for Chinese. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), pp. 608–615 (2017)

    Google Scholar 

  4. Cai, J., He, S., Li, Z., Zhao, H.: A full end-to-end semantic role labeler, syntactic-agnostic over syntactic-aware? In: Proceedings of the 27th International Conference on Computational Linguistics (COLING 2018), pp. 2753–2765 (2018)

    Google Scholar 

  5. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), pp. 1724–1734 (2014)

    Google Scholar 

  6. Chorowski, J.K., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-based models for speech recognition. In: Proceedings of the 29th Conference on Neural Information Processing Systems (NIPS 2015), pp. 577–585 (2015)

    Google Scholar 

  7. Dou, Z.Y., Tu, Z., Wang, X., Wang, L., Shi, S., Zhang, T.: Dynamic layer aggregation for neural machine translation with routing-by-agreement. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI 2019), pp. 86–93 (2019)

    Google Scholar 

  8. Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N.: Convolutional sequence to sequence learning. In: Proceedings of the 34th International Conference on Machine Learning (ICML 2017), pp. 1243–1252 (2017)

    Google Scholar 

  9. Guan, C., Cheng, Y., Zhao, H.: Semantic role labeling with associated memory network. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019), pp. 3361–3371 (2019)

    Google Scholar 

  10. Hassan, H., et al.: Achieving human parity on automatic Chinese to English news translation. arXiv preprint arXiv:1803.05567 (2018)

  11. He, S., Li, Z., Zhao, H., Bai, H.: Syntax for semantic role labeling, to be, or not to be. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), pp. 2061–2071 (2018)

    Google Scholar 

  12. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  13. Huang, Y., Zhao, H.: Chinese pinyin aided IME, input what you have not keystroked yet. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pp. 2923–2929 (2018)

    Google Scholar 

  14. Klein, G., Kim, Y., Deng, Y., Senellart, J., Rush, A.M.: OpenNMT: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810 (2017)

  15. Li, H., Zhang, Z., Ju, Y., Zhao, H.: Neural character-level dependency parsing for Chinese. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI 2018), pp. 5205–5212 (2018)

    Google Scholar 

  16. Li, J., Yang, B., Dou, Z.Y., Wang, X., Lyu, M.R., Tu, Z.: Information aggregation for multi-head attention with routing-by-agreement. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019), pp. 3566–3575 (2019)

    Google Scholar 

  17. Li, P., Zhang, X., Jia, W., Zhao, H.: GAN driven semi-distant supervision for relation extraction. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019), pp. 3026–3035 (2019)

    Google Scholar 

  18. Li, Z., Cai, J., He, S., Zhao, H.: Seq2seq dependency parsing. In: Proceedings of the 27th International Conference on Computational Linguistics (COLING 2018), pp. 3203–3214 (2018)

    Google Scholar 

  19. Li, Z., Cai, J., Zhao, H.: Effective representation for easy-first dependency parsing. In: PRICAI 2019: Trends in Artificial Intelligence - 15th Pacific Rim International Conference on Artificial Intelligence (2019)

    Google Scholar 

  20. Li, Z., et al.: A unified syntax-aware framework for semantic role labeling. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pp. 2401–2411 (2018)

    Google Scholar 

  21. Li, Z., He, S., Zhang, Z., Zhao, H.: Joint learning of POS and dependencies for multilingual universal dependency parsing. In: Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies (CoNLL 2018), pp. 65–73 (2018)

    Google Scholar 

  22. Li, Z., et al.: Dependency or span, end-to-end uniform semantic role labeling. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI 2019), pp. 6730–6737 (2019)

    Google Scholar 

  23. Lin, Z., Feng, M., et al.: A structured self-attentive sentence embedding. In: Proceedings of the 5th International Conference on Learning Representations (ICLR 2017) (2017)

    Google Scholar 

  24. Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pp. 1412–1421 (2015)

    Google Scholar 

  25. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002), pp. 311–318 (2002)

    Google Scholar 

  26. Peters, M., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018), pp. 2227–2237 (2018)

    Google Scholar 

  27. Raganato, A., Tiedemann, J.: An analysis of encoder representations in transformer-based machine translation. In: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 287–297 (2018)

    Google Scholar 

  28. Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), pp. 3856–3866 (2017)

    Google Scholar 

  29. Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), pp. 1715–1725 (2016)

    Google Scholar 

  30. Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018), pp. 464–468 (2018)

    Google Scholar 

  31. Shen, T., Zhou, T., Long, G., Jiang, J., Pan, S., Zhang, C.: DiSAN: directional self-attention network for RNN/CNN-free language understanding. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI 2018), pp. 5446–5455 (2018)

    Google Scholar 

  32. Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), pp. 5998–6008 (2017)

    Google Scholar 

  33. Wang, R., Finch, A., Utiyama, M., Sumita, E.: Sentence embedding for neural machine translation domain adaptation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), pp. 560–566 (2017)

    Google Scholar 

  34. Wang, R., Utiyama, M., Finch, A., Liu, L., Chen, K., Sumita, E.: Sentence selection and weighting for neural machine translation domain adaptation. IEEE/ACM Trans. Audio Speech Lang. Process. 26(10), 1727–1741 (2018)

    Article  Google Scholar 

  35. Wang, R., Utiyama, M., Liu, L., Chen, K., Sumita, E.: Instance weighting for neural machine translation domain adaptation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pp. 1482–1488 (2017)

    Google Scholar 

  36. Wang, R., Utiyama, M., Sumita, E.: Dynamic sentence sampling for efficient training of neural machine translation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), pp. 298–304 (2018)

    Google Scholar 

  37. Wang, X., Cai, D., Li, L., Xu, G., Zhao, H., Si, L.: Unsupervised learning helps supervised neural word segmentation, pp. 7200–7207 (2019)

    Google Scholar 

  38. Wu, Y., Zhao, H.: Finding better subword segmentation for neural machine translation. In: The 17th China National Conference on Computational Linguistics (CCL 2018), pp. 53–64 (2018)

    Google Scholar 

  39. Wu, Y., Zhao, H., Tong, J.J.: Multilingual universal dependency parsing from raw text with low-resource language enhancement. In: Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies (CoNLL 2018), pp. 74–80 (2018)

    Google Scholar 

  40. Wu, Y., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)

  41. Xiao, F., Li, J., Zhao, H., Wang, R., Chen, K.: Lattice-based transformer encoder for neural machine translation. In: Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL 2019), pp. 3090–3097 (2019)

    Google Scholar 

  42. Yang, B., Li, J., Wong, D.F., Chao, L.S., Wang, X., Tu, Z.: Context-aware self-attention networks. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI 2019), pp. 387–394 (2019)

    Google Scholar 

  43. Yang, B., Tu, Z., Wong, D.F., Meng, F., Chao, L.S., Zhang, T.: Modeling localness for self-attention networks. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pp. 4449–4458 (2018)

    Google Scholar 

  44. Yang, B., Wang, L., Wong, D.F., Chao, L.S., Tu, Z.: Convolutional self-attention networks. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019), pp. 4040–4045 (2019)

    Google Scholar 

  45. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2016), pp. 1480–1489 (2016)

    Google Scholar 

  46. Zhang, H., Zhao, H.: Minimum divergence vs. maximum margin: an empirical comparison on seq2seq models. In: Proceedings of the 7th International Conference on Learning Representations (ICLR 2019) (2019)

    Google Scholar 

  47. Zhang, Z., Wang, R., Utiyama, M., Sumita, E., Zhao, H.: Exploring recombination for efficient decoding of neural machine translation. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pp. 4785–4790 (2018)

    Google Scholar 

  48. Zhang, Z., Huang, Y., Zhao, H.: Subword-augmented embedding for cloze reading comprehension. In: Proceedings of the 27th International Conference on Computational Linguistics (COLING 2018), pp. 1802–1814 (2018)

    Google Scholar 

  49. Zhang, Z., Huang, Y., Zhao, H.: Open vocabulary learning for neural Chinese pinyin IME. In: Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL 2019), pp. 1584–1594 (2019)

    Google Scholar 

  50. Zhang, Z., Huang, Y., Zhu, P., Zhao, H.: Effective character-augmented word embedding for machine reading comprehension. In: 7th CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC 2018), pp. 27–39 (2018)

    Google Scholar 

  51. Zhang, Z., Zhao, H., Ling, K., Li, J., Li, Z., He, S.: Effective subword segmentation for text comprehension. Speech, and Language Processing, IEEE/ACM Transactions on Audio (2019)

    Google Scholar 

  52. Zhou, J., Zhao, H.: Head-driven phrase structure grammar parsing on Penn treebank. In: Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL 2019), pp. 2396–2408 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Juncheng Cao, Hai Zhao & Kai Yu

  2. Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China

    Juncheng Cao, Hai Zhao & Kai Yu

  3. MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China

    Juncheng Cao, Hai Zhao & Kai Yu

Authors
  1. Juncheng Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hai Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kai Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hai Zhao .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Jie Tang

  2. National University of Singapore, Singapore, Singapore

    Min-Yen Kan

  3. Peking University, Beijing, China

    Dongyan Zhao

  4. Peking University, Beijing, China

    Sujian Li

  5. Zhengzhou University, Zhengzhou, China

    Hongying Zan

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cao, J., Zhao, H., Yu, K. (2019). Cross Aggregation of Multi-head Attention for Neural Machine Translation. In: Tang, J., Kan, MY., Zhao, D., Li, S., Zan, H. (eds) Natural Language Processing and Chinese Computing. NLPCC 2019. Lecture Notes in Computer Science(), vol 11838. Springer, Cham. https://doi.org/10.1007/978-3-030-32233-5_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32233-5_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32232-8

  • Online ISBN: 978-3-030-32233-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation