Abstract
We train multi-task (variational) autoencoders on linguistic tasks and analyze the learned hidden sentence representations. The representations change significantly when translation and part-of-speech decoders are added. The more decoders are attached, the better the models cluster sentences according to their syntactic similarity, as the representation space becomes less entangled. We compare standard unconstrained autoencoders to variational autoencoders and find significant differences. We achieve better disentanglement with the standard autoencoder, which goes against recent work on variational autoencoders in the visual domain.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bengio, Y., Courville, A.C., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013). https://doi.org/10.1109/TPAMI.2013.50
Burgess, C.P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., Lerchner, A.: Understanding disentangling in \(\beta \)-VAE. arXiv preprint arXiv:180403599 (2018)
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., Lerchner, A.: \(\beta \)-VAE: Learning basic visual concepts with a constrained variational framework (2016)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. CoRR abs/1312.6114. http://arxiv.org/abs/1312.6114 (2013)
Koehn, P.: Europarl: A Parallel Corpus for Statistical Machine Translation (2005)
Liu, X., Gao, J., He, X., Deng, L., Duh, K., Wang, Y.: Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In: NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 912–921 (2015)
Luong, M., Le, Q.V., Sutskever, I., Vinyals, O., Kaiser, L.: Multi-task sequence to sequence learning. CoRR abs/1511.06114 (2015)
Niehues, J., Cho, E.: Exploiting linguistic resources for neural machine translation using multi-task learning. In: Proceedings of the Second Conference on Machine Translation, WMT 2017, pp. 80–89 (2017)
Shen, T., Lei, T., Barzilay, R., Jaakkola, T.: Style transfer from non-parallel text by cross-alignment. In: Advances in Neural Information Processing Systems, pp. 6833–6844 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Brunner, G., Wang, Y., Wattenhofer, R., Weigelt, M. (2019). Disentangling the Latent Space of (Variational) Autoencoders for NLP. In: Lotfi, A., Bouchachia, H., Gegov, A., Langensiepen, C., McGinnity, M. (eds) Advances in Computational Intelligence Systems. UKCI 2018. Advances in Intelligent Systems and Computing, vol 840. Springer, Cham. https://doi.org/10.1007/978-3-319-97982-3_13
Download citation
DOI: https://doi.org/10.1007/978-3-319-97982-3_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-97981-6
Online ISBN: 978-3-319-97982-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)