Context-Dependent Token-Wise Variational Autoencoder for Topic Modeling

  • Tomonari MasadaEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11609)


This paper proposes a new variational autoencoder (VAE) for topic models. The variational inference (VI) for Bayesian models approximates the true posterior distribution by maximizing a lower bound of the log marginal likelihood of observations. We can implement VI as VAE by using a neural network called encoder to obtain parameters of approximate posterior. Our contribution is three-fold. First, we marginalize out per-document topic probabilities by following the proposal by Mimno et al. This marginalizing out has not been considered in the existing VAE proposals for topic modeling to the best of our knowledge. Second, after marginalizing out topic probabilities, we need to approximate the posterior probabilities of token-wise topic assignments. However, this posterior is categorical and thus cannot be approximated by continuous distributions like Gaussian. Therefore, we adopt the Gumbel-softmax trick. Third, while we can sample token-wise topic assignments with the Gumbel-softmax trick, we should consider document-wide contextual information for a better approximation. Therefore, we feed to our encoder network a concatenation of token-wise information and document-wide information, where the former is implemented as word embedding and the latter as the document-wide mean of word embeddings. The experimental results showed that our VAE improved the existing VAE proposals for a half of the data sets in terms of perplexity or of normalized pairwise mutual information (NPMI).


Topic modeling Deep learning Variational autoencoder 



This work was supported by JSPS KAKENHI Grant Number JP18K11440.


  1. 1.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)zbMATHGoogle Scholar
  2. 2.
    Bouma, G.: Normalized (Pointwise) Mutual information in collocation extraction. In: Proceedings of the Biennial GSCL Conference From Form to Meaning: Processing Texts Automatically, vol. 2009, pp. 31–40 (2009)Google Scholar
  3. 3.
    Dieng, A.B., Wang, C., Gao, J., Paisley, J.W.: TopicRNN: a recurrent neural network with long-range semantic dependency. CoRR abs/1611.01702 (2016).
  4. 4.
    Goodfellow, I.J., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27 (NIPS), pp. 2672–2680 (2014)Google Scholar
  5. 5.
    Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proc. Nat. Acad. Sci. 101(suppl 1), 5228–5235 (2004)CrossRefGoogle Scholar
  6. 6.
    Hoffman, M.D., Blei, D.M., Bach, F.R.: Online learning for latent Dirichlet allocation. Adv. Neural Inf. Process. Syst. (NIPS) 23, 856–864 (2010)Google Scholar
  7. 7.
    Jang, E., Gu, S., Poole, B.: Categorical reparameterization with Gumbel-softmax. CoRR abs/1611.01144 (2016),
  8. 8.
    Kim, Y., Wiseman, S., Miller, A.C., Sontag, D., Rush, A.M.: Semi-amortized variational autoencoders. In: Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 2683–2692 (2018)Google Scholar
  9. 9.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014).
  10. 10.
    Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. CoRR abs/1312.6114 (2013).
  11. 11.
    Masada, T., Takasu, A.: Adversarial learning for topic models. In: Proceedings of the 14th International Conference on Advanced Data Mining and Applications (ADMA), pp. 292–302 (2018)Google Scholar
  12. 12.
    Mescheder, L.M., Nowozin, S., Geiger, A.: Adversarial variational Bayes: unifying variational autoencoders and generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 2391–2400 (2017)Google Scholar
  13. 13.
    Miao, Y., Grefenstette, E., Blunsom, P.: Discovering discrete latent topics with neural variational inference. In: Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 2410–2419 (2017)Google Scholar
  14. 14.
    Miao, Y., Yu, L., Blunsom, P.: Neural variational inference for text processing. In: Proceedings of the 33rd International Conference on Machine Learning (ICML), pp. 1727–1736 (2016)Google Scholar
  15. 15.
    Mimno, D., Hoffman, M.D., Blei, D.M.: Sparse stochastic inference for latent Dirichlet allocation. In: Proceedings of the 29th International Conference on Machine Learning (ICML), pp. 1515–1522 (2012)Google Scholar
  16. 16.
    Mnih, A., Hinton, G.E.: A scalable hierarchical distributed language model. Adv. Neural Inf. Process. Syst. (NIPS) 21, 1081–1088 (2008)Google Scholar
  17. 17.
    Press, O., Wolf, L.: Using the output embedding to improve language models. CoRR abs/1608.05859 (2016).
  18. 18.
    Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models. In: Proceedings of the 31st International Conference on Machine Learning (ICML), pp. II-1278-II-1286 (2014)Google Scholar
  19. 19.
    Roeder, G., Wu, Y., Duvenaud, D.K.: Sticking the landing: simple, lower-variance gradient estimators for variational inference. Adv. Neural Inf. Process. Syst. (NIPS) 30, 6928–6937 (2017)Google Scholar
  20. 20.
    Salimans, T., Knowles, D.A.: Fixed-form variational posterior approximation through stochastic linear regression. CoRR abs/1206.6679 (2012).
  21. 21.
    Sønderby, C.K., Raiko, T., Maaløe, L., Sønderby, S.K., Winther, O.: Ladder variational autoencoders. Adv. Neural Inf. Process. Syst. (NIPS) 29, 3738–3746 (2016)Google Scholar
  22. 22.
    Srivastava, A., Sutton, C.: Autoencoding variational inference for topic models. CoRR abs/1703.01488 (2017).
  23. 23.
    Teh, Y.W., Newman, D., Welling, M.: A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. Adv. Neural Inf. Process. Syst. (NIPS) 19, 1353–1360 (2006)Google Scholar
  24. 24.
    Titsias, M.K., Lázaro-Gredilla, M.: Doubly stochastic variational Bayes for non-conjugate inference. In: Proceedings of the 31st International Conference on Machine Learning (ICML), pp. II-1971-II-1980 (2014)Google Scholar
  25. 25.
    Tokui, S., Sato, I.: Reparameterization trick for discrete variables. CoRR abs/1611.01239 (2016).
  26. 26.
    Wang, W., et al.: Topic compositional neural language model. In: Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 356–365 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Nagasaki UniversityNagasaki-shiJapan

Personalised recommendations