Skip to main content
SpringerLink
Log in

Improved Graph-Based Dependency Parsing via Hierarchical LSTM Networks

  • Conference paper
  • First Online:
Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data (NLP-NABD 2016, CCL 2016)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10035))

Abstract

In this paper, we propose a neural graph-based dependency parsing model which utilizes hierarchical LSTM networks on character level and word level to learn word representations, allowing our model to avoid the problem of limited-vocabulary and capture both distributional and compositional semantic information. Our model achieves state-of-the-art accuracy on Chinese Penn Treebank and competitive accuracy on English Penn Treebank with only first-order features. Moreover, our model shows effectiveness in recovering dependencies involving out-of-vocabulary words.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    In our experiments, all words occurring less than 10 times in the corpus are treated as unknown words.

  2. 2.

    http://nlp.stanford.edu/software/lex-parser.shtml.

  3. 3.

    http://stp.lingfil.uu.se/nivre/research/Penn2Malt.html.

  4. 4.

    Following previous work, a token is a punctuation if its POS tag is {“" : , .}.

References

  1. Marcus, M.P., Marcinkiewicz, M.A., Santorini, B.: Building a large annotated corpus of English: The Penn Treebank. Comput. Linguist. 19(2), 313–330 (1993)

    Google Scholar 

  2. Xue, N., Xia, F., Chiou, F.-D., Palmer, M.: The penn Chinese treebank: phrase structure annotation of a large corpus. Nat. Lang. Eng. 11(02), 207–238 (2005)

    Article  Google Scholar 

  3. Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., Collins, M.: Globally normalized transition-based neural networks. In: Proceedings of the 54rd Annual Meeting of the Association for Computational Linguistics (2016)

    Google Scholar 

  4. Ballesteros, M., Dyer, C., Smith, N.A.: Improved transition-based parsing by modeling characters instead of words with LSTMs. Computer Science (2015a)

    Google Scholar 

  5. Ballesteros, M., Dyer, C., Smith, N.A.: Improved transition-based parsing by modeling characters instead of words with LSTMs. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP, Lisbon, Portugal, 17–21 September 2015, pp. 349–359 (2015)

    Google Scholar 

  6. Kuhn, J., Bohnet, B.: The best of both worlds: a graph-based completion model for transition-based parsers. In: Conference of the European Chapter of the Association for Computational Linguistics

    Google Scholar 

  7. Chen, D., Manning, C.D.: A fast and accurate dependency parser using neural networks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 740–750 (2014)

    Google Scholar 

  8. Cross, J., Huang, L.: Incremental parsing with minimal features using bi-directional LSTM. In: Proceedings of the 54rd Annual Meeting of the Association for Computational Linguistics (2016)

    Google Scholar 

  9. Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12, 2121–2159 (2011)

    MathSciNet  MATH  Google Scholar 

  10. Dyer, C., Ballesteros, M., Ling, W., Matthews, A., Smith, N.A.: Transition-based dependency parsing with stack long short-term memory. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pp. 334–343 (2015)

    Google Scholar 

  11. Graff, D., Kong, J., Chen, K., Maeda, K.: English Gigaword. Linguistic Data Consortium, Philadelphia (2003)

    Google Scholar 

  12. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors (2012). arXiv preprint arXiv:1207.0580

  13. Ling, W., Dyer, C., Black, A., Trancoso, I.: Two/too simple adaptations of word2vec for syntax problems. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2015a)

    Google Scholar 

  14. Ling, W., Dyer, C., Black, A.W., Trancoso, I., Fermandez, R., Amir, S., Marujo, L., Luís, T.: Compositional character models for open vocabulary word representation. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 1520–1530 (2015b)

    Google Scholar 

  15. De Marneffe, M.C., Maccartney, B., Manning, C.D.: Generating typed dependency parses from phrase structure parses. LREC 6, 449–454 (2006)

    Google Scholar 

  16. Pei, W., Ge, T., Chang, B.: An effective neural network model for graph-based dependency parsing. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pp. 313–322 (2015)

    Google Scholar 

  17. Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-rich part-of-speech tagging with a cyclic dependency network. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pp. 173–180. Association for Computational Linguistics (2003)

    Google Scholar 

  18. Wang, W., Chang, B.: Graph-based dependency parsing with bidirectional LSTM. In: Proceedings of the 54rd Annual Meeting of the Association for Computational Linguistics (2016)

    Google Scholar 

  19. Weiss, D., Alberti, C., Collins, M., Petrov, S.: Structured training for neural network transition-based parsing. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (2015)

    Google Scholar 

  20. Yamada, H., Matsumoto, Y.: Statistical dependency analysis with support vector machines. In: Proceedings of IWPT, vol. 3, pp. 195–206 (2003)

    Google Scholar 

  21. Zhang, Y., Clark, S.: A tale of two parsers: investigating and combining graph-based and transition-based dependency parsing. In: Conference on Empirical Methods in Natural Language Processing, pp. 562–571 (2008)

    Google Scholar 

  22. Zhang, H., McDonald, R.T.: Enforcing structural diversity in cube-pruned dependency parsing. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pp. 656–661 (2014)

    Google Scholar 

  23. Zhang, Y., Nivre, J.: Transition-based dependency parsing with rich non-local features. In: The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 188–193 (2011)

    Google Scholar 

  24. Zhang, Y., Weiss, D.: Stack-propagation: improved representation learning for syntax. In: Proceedings of the 54rd Annual Meeting of the Association for Computational Linguistics (2016)

    Google Scholar 

Download references

Acknowledgments

This work is supported by National Key Basic Research Program of China under Grant No. 2014CB340504 and National Natural Science Foundation of China under Grant No. 61273318. The Corresponding author of this paper is Baobao Chang.

Author information

Authors and Affiliations

  1. Key Laboratory of Computational Linguistics, Ministry of Education, School of Electronics Engineering and Computer Science, Peking University, No. 5 Yiheyuan Road, Haidian District, Beijing, 100871, China

    Wenhui Wang & Baobao Chang

  2. Collaborative Innovation Center for Language Ability, Xuzhou, 221009, China

    Wenhui Wang & Baobao Chang

Authors
  1. Wenhui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Baobao Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Baobao Chang .

Editor information

Editors and Affiliations

  1. Tsinghua University , Beijing, China

    Maosong Sun

  2. Fudan University , Shanghai, China

    Xuanjing Huang

  3. Dalian University of Technology , Dalian, China

    Hongfei Lin

  4. Tsinghua University , Beijing, China

    Zhiyuan Liu

  5. Tsinghua University , Beijing, China

    Yang Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Wang, W., Chang, B. (2016). Improved Graph-Based Dependency Parsing via Hierarchical LSTM Networks. In: Sun, M., Huang, X., Lin, H., Liu, Z., Liu, Y. (eds) Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data. NLP-NABD CCL 2016 2016. Lecture Notes in Computer Science(), vol 10035. Springer, Cham. https://doi.org/10.1007/978-3-319-47674-2_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-47674-2_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-47673-5

  • Online ISBN: 978-3-319-47674-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation