Split First and Then Rephrase: Hierarchical Generation for Sentence Simplification

  • Mengru WangEmail author
  • Hiroaki Ozaki
  • Yuta Koreeda
  • Kohsuke Yanai
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1215)


Split-and-rephrase is a strategy known to be used when humans need to break down a complex sentence into a meaning preserving sequence of shorter ones. Recent work proposed to model split-and-rephrase as a supervised sequence generation problem. However, different from other types of sequence generations, the task of split-and-rephrase inevitably introduces overlaps across splits to compensate for the missing context caused by separating a sentence. Serving as the baseline of this task, the vanilla SEQ2SEQ model usually suffers from inappropriate duplication because of the lack of a mechanism to plan how the source sentence should be split into shorter units. This work demonstrates that the problem of inappropriate duplication can be tackled by explicitly modeling the hierarchy within split-and-rephrase: Our model first introduces a separator network capable of selecting semantic components from the source sentence to form a representation for each split. Then, a decoder generates each split on the basis of its representation. Analyses demonstrate that with the aid of the separator, a model can effectively learn attention to avoid duplication and detect clues for splitting a sentence. Experimental results on the WikiSplit corpus show that our model outperforms the non-hierarchical SEQ2SEQ model by 1.4 points in terms of duplication rate and by 0.3 points in terms of coverage rate.


Text simplification Hierarchical text generation Split-and-rephrase 


  1. 1.
    Narayan, S., Gardent, C., Cohen, S.B., Shimorina, A.: Split and rephrase. In: Proceedings of EMNLP, pp. 606–616 (2017)Google Scholar
  2. 2.
    Braud, C., Lacroix, O., Søgaard, A.: Cross-lingual and cross-domain discourse segmentation of entire documents. In: Proceedings of ACL, vol. 2, pp. 237–243 (2017)Google Scholar
  3. 3.
    Koehn, P., Knowles, R.: Six challenges for neural machine translation. In: Proceedings of the 1st Workshop on NMT, pp. 28–39 (2017)Google Scholar
  4. 4.
    Pouget-Abadie, J., Bahdanau, D., van Merrienboer, B., Cho, K., Bengio, Y.: Overcoming the curse of sentence length for neural machine translation using automatic segmentation. In: Proceedings of SSST-8, pp. 78–85 (2014)Google Scholar
  5. 5.
    Chandrasekar, R., Doran, C., Srinivas, B.: Motivations and methods for text simplification. In: Proceedings of COLING, vol. 2, pp. 1041–1044 (1996)Google Scholar
  6. 6.
    Tomita, M.: Efficient Parsing for Natural Language: A Fast Algorithm for Practical Systems. SECS, vol. 8. Springer, Boston (1986). Scholar
  7. 7.
    Chandrasekar, R., Bangalore, S.: Automatic induction of rules for text simplification. Knowl.-Based Syst. 10, 183–190 (1997)CrossRefGoogle Scholar
  8. 8.
    McDonald, R., Nivre, J.: Analyzing and integrating dependency parsers. Comput. Linguist. 37(1), 197–230 (2011)CrossRefGoogle Scholar
  9. 9.
    Jelínek, T.: Improvements to dependency parsing using automatic simplification of data. In: Proceedings of LREC, pp. 73–77 (2014)Google Scholar
  10. 10.
    Zhang, Y., Zhong, V., Chen, D., Angeli, G., Manning, C.D.: Position-aware attention and supervised data improve slot filling. In: Proceedings of EMNLP, pp. 35–45 (2017)Google Scholar
  11. 11.
    Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE TSP 45(11), 2673–2681 (1997)Google Scholar
  12. 12.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  13. 13.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: Proceedings of ICLR (2015)Google Scholar
  14. 14.
    Press, O., Wolf, L.: Using the output embedding to improve language models. In: Proceedings of EACL, vol. 2, pp. 157–163 (2017)Google Scholar
  15. 15.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of ACL, pp. 311–318 (2002)Google Scholar
  16. 16.
    Botha, J.A., Faruqui, M., Alex, J., Baldridge, J., Das, D.: Learning to split and rephrase from Wikipedia edit history. In: Proceedings of EMNLP, pp. 732–737 (2018)Google Scholar
  17. 17.
    Aharoni, R., Goldberg, Y.: Split and rephrase: better evaluation and stronger baselines. In: Proceedings of ACL, vol. 2, pp. 719–724 (2018)Google Scholar
  18. 18.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of ICLR (2015)Google Scholar
  19. 19.
    Dyer, C., Chahuneau, V., Smith, N.A.: A simple, fast, and effective reparameterization of IBM model 2. In: Proceedings of NAACL-HLT, pp. 644–648 (2013)Google Scholar
  20. 20.
    Vu, T., Hu, B., Munkhdalai, T., Yu, H.: Sentence simplification with memory-augmented neural networks. In: Proceedings of NAACL-HLT, pp. 79–85 (2018)Google Scholar
  21. 21.
    Zuo, S., Xu, Z.: A hierarchical neural network for sequence-to-sequences learning, vol. arXiv preprint arXiv:1811.09575 (2018)
  22. 22.
    Brown, P.F., Pietra, S.A.D., Pietra, V.J.D., Mercer, R.L., Mohanty, S.: Dividing and conquering long sentences in a translation system. In: Proceedings of the Workshop on Speech and Natural Language, pp. 267–271 (1992)Google Scholar
  23. 23.
    Li, J., Luong, T., Jurafsky, D.: A hierarchical neural autoencoder for paragraphs and documents. In: Proceedings of ACL and IJCNLP, vol. 1, pp. 1106–1115 (2015)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  • Mengru Wang
    • 1
    Email author
  • Hiroaki Ozaki
    • 1
  • Yuta Koreeda
    • 1
  • Kohsuke Yanai
    • 1
  1. 1.Hitachi, Ltd., Research & Development GroupKokubunji-shiJapan

Personalised recommendations