A Semi-supervised Hidden Markov Topic Model Based on Prior Knowledge
Abstract
A topic model is an unsupervised model to automatically discover the topics discussed in a collection of documents. Most of the existing topic models only use bag-of-words representations or single-word distributions and do not consider relations between words in the model. As a consequence, these models may generate topics which are not in good agreement with human-judged topic coherence. To mitigate this issue, we present a topic model which employs topically-related knowledge from prior topics and words’ co-occurrence/relations in the collection. To incorporate the prior knowledge, we leverage a two-staged semi-supervised Markov topic model. In the first stage, we estimate a transition matrix and a low-dimensional vocabulary for the final topic model. In the second stage, we produce the final topic model where the topic assignment is performed following a Markov chain process. Experiments on real text documents from a major compensation agency demonstrate improvements of both the PMI score measure and the topic coherence.
Keywords
Topic modelling Hidden Markov model Latent Dirichlet allocation Topic coherenceNotes
Acknowledgement
This project was funded by the Capital Market Cooperative Research Centre in combination with the Transport Accident Commission of Victoria. Acknowledgements and thanks to industry supervisors David Attwood (Lead Research Partnerships) and Bernie Kruger (Data Science Lead). This research has received ethics approval from University of Technology Sydney (UTS HREC REF NO. ETH16-0968).
References
- 1.Blei, D.M., Lafferty, J.D.: A correlated topic model of science. Ann. Appl. Stat. 1(1), 17–35 (2007)MathSciNetCrossRefGoogle Scholar
- 2.Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)MATHGoogle Scholar
- 3.Blei, D.M., Lafferty, J.D.: Dynamic topic models. In: International Conference on Machine Learning (ICML) (2006)Google Scholar
- 4.Blei, D.M., McAuliffe, J.D.: Supervised topic models. In: Advances in neural information processing systems, vol. 20, pp. 121–128. MIT Press, Cambridge (2008)Google Scholar
- 5.Chen, Z., Liu, B.: Topic modeling using topics from many domains, lifelong learning and big data. In: Proceedings of the 31st International Conference on Machine Learning (ICML), vol. 32, pp. 703–711 (2014)Google Scholar
- 6.Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. J. Assoc. Inf. Sci. Technol. 41(6), 391–407 (1990)Google Scholar
- 7.Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proc. Nat. Acad. Sci. 101, 5228–5235 (2004)CrossRefGoogle Scholar
- 8.Gruber, A., Rosen-Zvi, M., Weiss, Y.: Hidden topic Markov models. In: Proceedings of the 11th International Conference on Artificial Intelligence and Statistics (AISTATS-07), pp. 163–170 (2007)Google Scholar
- 9.Hofmann, T.: Probabilistic latent semantic indexing. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 50–57 (1999)Google Scholar
- 10.Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. Mach. Learn. 42, 177–196 (2001)CrossRefGoogle Scholar
- 11.Hsu, W.S., Poupart, P.: Online Bayesian moment matching for topic modeling with unknown number of topics. In: Proceedings of the 30th International Conference on Neural Information Processing Systems (2016)Google Scholar
- 12.Hu, Y., Boyd-Graber, J., Satinoff, B.: Interactive topic modeling. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pp. 248–257 (2011)Google Scholar
- 13.Lacoste-Julien, S., Sha, F., Jordan, M.I.: DiscLDA: discriminative learning for dimensionality reduction and classification. In: Proceedings of the 21st International Conference on Neural Information Processing Systems, pp. 897–904 (2008)Google Scholar
- 14.Newman, D., Bonilla, E.V., Buntine, W.L.: Improving topic coherence with regularized topic models. In: Proceedings of the 24th International Conference on Neural Information Processing Systems (2011)Google Scholar
- 15.Papadimitriou, C.H., Raghavan, P., Tamaki, H., Vempala, S.: Latent semantic indexing: a probabilistic analysis. J. Comput. Syst. Sci. 61(2), 217–235 (2000)MathSciNetCrossRefGoogle Scholar
- 16.Qiang, J., Chen, P., Wang, T., Wu, X.: Topic modeling over short texts by incorporating word embeddings. In: Kim, J., Shim, K., Cao, L., Lee, J.-G., Lin, X., Moon, Y.-S. (eds.) PAKDD 2017. LNCS (LNAI), vol. 10235, pp. 363–374. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-57529-2_29CrossRefGoogle Scholar
- 17.Teh, Y.W., Newman, D., Welling, M.: A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In: Proceedings of the 19th International Conference on Neural Information Processing Systems, pp. 1353–1360 (2006)Google Scholar
- 18.Wang, C., Blei, D.M., Heckerman, D.: Continuous time dynamic topic models. In: Proceedings of the 24th Conference on Uncertainty in Artificial Intelligence (2008)Google Scholar
- 19.Wei, L., Blei, D.M., McCallum, A.: Nonparametric Bayes pachinko allocation. In: Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence, pp. 243–250 (2007)Google Scholar
- 20.Wood, J., Tan, P., Wang, W., Arnold, C.: Source-LDA: enhancing probabilistic topic models using prior knowledge sources. In: IEEE 33rd International Conference on Data Engineering (ICDE) (2017)Google Scholar
- 21.Xuerui, X., McCallum, A.: Topics over time: a non-Markov continuous-time model of topical trends. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 424–433 (2006)Google Scholar