Context-Dependent Token-Wise Variational Autoencoder for Topic Modeling

Abstract

This paper proposes a new variational autoencoder (VAE) for topic models. The variational inference (VI) for Bayesian models approximates the true posterior distribution by maximizing a lower bound of the log marginal likelihood of observations. We can implement VI as VAE by using a neural network called encoder to obtain parameters of approximate posterior. Our contribution is three-fold. First, we marginalize out per-document topic probabilities by following the proposal by Mimno et al. This marginalizing out has not been considered in the existing VAE proposals for topic modeling to the best of our knowledge. Second, after marginalizing out topic probabilities, we need to approximate the posterior probabilities of token-wise topic assignments. However, this posterior is categorical and thus cannot be approximated by continuous distributions like Gaussian. Therefore, we adopt the Gumbel-softmax trick. Third, while we can sample token-wise topic assignments with the Gumbel-softmax trick, we should consider document-wide contextual information for a better approximation. Therefore, we feed to our encoder network a concatenation of token-wise information and document-wide information, where the former is implemented as word embedding and the latter as the document-wide mean of word embeddings. The experimental results showed that our VAE improved the existing VAE proposals for a half of the data sets in terms of perplexity or of normalized pairwise mutual information (NPMI).