Advertisement

Adversarial Learning for Topic Models

  • Tomonari MasadaEmail author
  • Atsuhiro Takasu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11323)

Abstract

This paper proposes adversarial learning for topic models. Adversarial learning we consider here is a method of density ratio estimation using a neural network called discriminator. In generative adversarial networks (GANs) we train discriminator for estimating the density ratio between the true data distribution and the generator distribution. Also in variational inference (VI) for Bayesian probabilistic models we can train discriminator for estimating the density ratio between the approximate posterior distribution and the prior distribution. With the adversarial learning in VI we can adopt implicit distribution as an approximate posterior. This paper proposes adversarial learning for latent Dirichlet allocation (LDA) to improve the expressiveness of the approximate posterior. Our experimental results showed that the quality of extracted topics was improved in terms of test perplexity.

Keywords

Topic models Adversarial learning Variational inference 

References

  1. 1.
    Asuncion, A., Welling, M., Smyth, P., Teh, Y.W.: On smoothing and inference for topic models. In: Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2009, pp. 27–34 (2009)Google Scholar
  2. 2.
    Blei, D.M.: Probabilistic topic models. Commun. ACM 55(4), 77–84 (2012)CrossRefGoogle Scholar
  3. 3.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)zbMATHGoogle Scholar
  4. 4.
    Chen, S.F., Goodman, J.: An empirical study of smoothing techniques for language modeling. In: Proceedings of the 34th Annual Meeting on Association for Computational Linguistics, ACL 1996, pp. 310–318 (1996)Google Scholar
  5. 5.
    Dieng, A.B., Wang, C., Gao, J., Paisley, J.W.: TopicRNN: a recurrent neural network with long-range semantic dependency. CoRR abs/1611.01702 (2016). http://arxiv.org/abs/1611.01702
  6. 6.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, pp. 249–256 (2010)Google Scholar
  7. 7.
    Goodfellow, I.J., et al.: Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS 2014, vol. 2, pp. 2672–2680 (2014)Google Scholar
  8. 8.
    Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proc. Natl. Acad. Sci. 101(Suppl. 1), 5228–5235 (2004)CrossRefGoogle Scholar
  9. 9.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: Proceedings of the 2015 IEEE International Conference on Computer Vision, ICCV 2015, pp. 1026–1034 (2015)Google Scholar
  10. 10.
    Huszár, F.: Variational inference using implicit distributions. CoRR abs/1702.08235 (2017). http://arxiv.org/abs/1702.08235
  11. 11.
    Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. CoRR abs/1312.6114 (2013). http://arxiv.org/abs/1312.6114
  12. 12.
    Lea, C., Vidal, R., Reiter, A., Hager, G.D.: Temporal convolutional networks: a unified approach to action segmentation. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 47–54. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-49409-8_7CrossRefGoogle Scholar
  13. 13.
    Mescheder, L.M., Nowozin, S., Geiger, A.: Adversarial variational Bayes: unifying variational autoencoders and generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, pp. 2391–2400 (2017)Google Scholar
  14. 14.
    Miao, Y., Yu, L., Blunsom, P.: Neural variational inference for text processing. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning, ICML 2016, vol. 48, pp. 1727–1736 (2016)Google Scholar
  15. 15.
    Mohamed, S., Lakshminarayanan, B.: Learning in implicit generative models. CoRR abs/1610.03483 (2016). http://arxiv.org/abs/1610.03483
  16. 16.
    Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models. In: Proceedings of the 31st International Conference on International Conference on Machine Learning, ICML 2014, vol. 32, pp. II-1278–II-1286 (2014)Google Scholar
  17. 17.
    Shu, R., Bui, H.H., Zhao, S., Kochenderfer, M.J., Ermon, S.: Amortized inference regularization. CoRR abs/1805.08913 (2018). http://arxiv.org/abs/1805.08913
  18. 18.
    Srivastava, A., Sutton, C.: Autoencoding variational inference for topic models. CoRR abs/1703.01488 (2017). http://arxiv.org/abs/1703.01488
  19. 19.
    Titsias, M.K., Lázaro-Gredilla, M.: Doubly stochastic variational Bayes for non-conjugate inference. In: Proceedings of the 31st International Conference on International Conference on Machine Learning, ICML 2014, vol. 32, pp. II-1971–II-1980 (2014)Google Scholar
  20. 20.
    Uehara, M., Sato, I., Suzuki, M., Nakayama, K., Matsuo, Y.: Generative adversarial nets from a density ratio estimation perspective. CoRR abs/1610.02920 (2016). http://arxiv.org/abs/1610.02920

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Nagasaki UniversityNagasaki-shiJapan
  2. 2.National Institute of InformaticsChiyoda-kuJapan

Personalised recommendations