Abstract
Conditional Variational Autoencoder (CVAE) has shown promising performance in text generation. However, CVAE is inadequate to generate sentences that are highly coherent to its condition due to error accumulation in decoding and KL-vanishing problem. In this paper, we propose an Edit-CVAE (ECVAE) in which we attempt to exploit information-related data to address the problem by (1) explicitly editing the generated sentence. (2) enriching the latent representation. While maintaining the diversity and information consistency. Experiment results on dialogue and Chinese poetry generation show that our method substantially increases generative coherence while maintaining the diversity and information consistency.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
OST is clollected from www.opensubtitles.org.
- 3.
Poetry is from https://github.com/chinese-poetry/chinese-poetry.
- 4.
We use RUBER from https://github.com/liming-vie/RUBER.
- 5.
References
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv:1312.6114 (2013)
Doersch, C.: Tutorial on variational autoencoders. arXiv:1606.05908 (2016)
Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. In: NeurIPS (2015)
Li, J., et al.: Generating classical chinese poems via conditional variational autoencoder and adversarial training. In: EMNLP (2018)
Serban, I.V., et al.: A hierarchical latent variable encoder-decoder model for generating dialogues. In: AAAI (2017)
Yang, X., et al.: Generating thematic Chinese poetry with conditional variational autoencoder. CoRR (2017)
Hu, Z., et al.: Toward controlled generation of text. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR.org (2017)
Shen, X., et al.: A conditional variational framework for dialog generation. arXiv preprint arXiv:1705.00316 (2017)
Zhao, T., Zhao, R., Eskenazi, M.: Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. arXiv preprint arXiv:1703.10960 (2017)
Bowman, S.R., et al.: Generating Sentences from a Continuous Space. In: CoNLL (2016)
Shen, X., et al.: Improving variational encoder-decoders in dialogue generation. In: AAAI (2018)
Xiao, Y., Zhao, T., Wang, W.Y.: Dirichlet variational autoencoder for text modeling. arXiv preprint arXiv:1811.00135 (2018)
Serban, I.V., et al.: Piecewise latent variables for neural variational text processing. In: Proceedings of the 2nd Workshop on Structured Prediction for Natural Language Processing (2017)
Xu, X., et al.: Better conversations by modeling, filtering, and optimizing for coherence and diversity. In: EMNLP (2018)
Dieng, A.B., et al.: Avoiding latent variable collapse with generative skip models. arXiv preprint arXiv:1807.04863 (2018)
Jiang, S., de Rijke, M.: Why are sequence-to-sequence models so dull? Understanding the low-diversity problem of chatbots. In: Proceedings of the 2018 EMNLP Workshop SCAI (2018)
Goyal, A., et al.: Z-forcing: training stochastic recurrent networks. In: NeurIPS (2017)
Du, J., et al.: Variational autoregressive decoder for neural response generation. In: EMNLP (2018)
Yang, Z., et al.: Improved variational autoencoders for text modeling using dilated convolutions. arXiv preprint arXiv:1702.08139 (2017)
Kim, Y., et al.: Semi-amortized variational autoencoders. arXiv preprint arXiv:1802.02550 (2018)
Guu, K., et al.: Generating sentences by editing prototypes. Trans. Assoc. Comput. Linguist. 6, 437–450 (2018)
Wu, Y., et al.: Response generation by context-aware prototype editing. arXiv preprint arXiv:1806.07042 (2018)
Cao, Z., et al.: Retrieve, rerank and rewrite: soft template based neural summarization. In: ACL (2018)
Gu, J., et al.: Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393 (2016)
See, A., Liu, P.J., Manning, C.D.: Get to the point: summarization with pointer-generator networks. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (2017)
Song, Y., et al.: Two are better than one: an ensemble of retrieval-and generation-based dialog systems. arXiv preprint arXiv:1610.07149 (2016)
Song, Y., et al.: An Ensemble of Retrieval-Based and Generation-Based Human-Computer Conversation Systems (2018)
Wang, H., et al.: A dataset for research on short-text conversations. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (2013)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Tao, C., et al.: Ruber: an unsupervised method for automatic evaluation of open-domain dialog systems. In: AAAI (2018)
Li, J., et al.: A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055 (2015)
Tao, C., et al.: Get the point of my utterance! learning towards effective responses with multi-head attention mechanism. In: IJCAI (2018)
Pereyra, G., et al.: Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548 (2017)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Neurips (2014)
Vaswani, A., et al.: Attention is all you need. In: Neurips (2017)
Acknowledgments
This work was supported by the National Key Research and Development Program of China (No. 2017YFC0804001), the National Science Foundation of China (NSFC No. 61672058; NSFC No. 61876196).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Tao, Z., Si, W., Li, J., Zhao, D., Yan, R. (2019). Boosting Variational Generative Model via Condition Enhancing and Lexical-Editing. In: Nayak, A., Sharma, A. (eds) PRICAI 2019: Trends in Artificial Intelligence. PRICAI 2019. Lecture Notes in Computer Science(), vol 11670. Springer, Cham. https://doi.org/10.1007/978-3-030-29908-8_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-29908-8_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29907-1
Online ISBN: 978-3-030-29908-8
eBook Packages: Computer ScienceComputer Science (R0)