A Semi-supervised Hidden Markov Topic Model Based on Prior Knowledge

  • Sattar Seifollahi
  • Massimo Piccardi
  • Ehsan Zare Borzeshi
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 845)


A topic model is an unsupervised model to automatically discover the topics discussed in a collection of documents. Most of the existing topic models only use bag-of-words representations or single-word distributions and do not consider relations between words in the model. As a consequence, these models may generate topics which are not in good agreement with human-judged topic coherence. To mitigate this issue, we present a topic model which employs topically-related knowledge from prior topics and words’ co-occurrence/relations in the collection. To incorporate the prior knowledge, we leverage a two-staged semi-supervised Markov topic model. In the first stage, we estimate a transition matrix and a low-dimensional vocabulary for the final topic model. In the second stage, we produce the final topic model where the topic assignment is performed following a Markov chain process. Experiments on real text documents from a major compensation agency demonstrate improvements of both the PMI score measure and the topic coherence.


Topic modelling Hidden Markov model Latent Dirichlet allocation Topic coherence 



This project was funded by the Capital Market Cooperative Research Centre in combination with the Transport Accident Commission of Victoria. Acknowledgements and thanks to industry supervisors David Attwood (Lead Research Partnerships) and Bernie Kruger (Data Science Lead). This research has received ethics approval from University of Technology Sydney (UTS HREC REF NO. ETH16-0968).


  1. 1.
    Blei, D.M., Lafferty, J.D.: A correlated topic model of science. Ann. Appl. Stat. 1(1), 17–35 (2007)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)zbMATHGoogle Scholar
  3. 3.
    Blei, D.M., Lafferty, J.D.: Dynamic topic models. In: International Conference on Machine Learning (ICML) (2006)Google Scholar
  4. 4.
    Blei, D.M., McAuliffe, J.D.: Supervised topic models. In: Advances in neural information processing systems, vol. 20, pp. 121–128. MIT Press, Cambridge (2008)Google Scholar
  5. 5.
    Chen, Z., Liu, B.: Topic modeling using topics from many domains, lifelong learning and big data. In: Proceedings of the 31st International Conference on Machine Learning (ICML), vol. 32, pp. 703–711 (2014)Google Scholar
  6. 6.
    Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. J. Assoc. Inf. Sci. Technol. 41(6), 391–407 (1990)Google Scholar
  7. 7.
    Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proc. Nat. Acad. Sci. 101, 5228–5235 (2004)CrossRefGoogle Scholar
  8. 8.
    Gruber, A., Rosen-Zvi, M., Weiss, Y.: Hidden topic Markov models. In: Proceedings of the 11th International Conference on Artificial Intelligence and Statistics (AISTATS-07), pp. 163–170 (2007)Google Scholar
  9. 9.
    Hofmann, T.: Probabilistic latent semantic indexing. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 50–57 (1999)Google Scholar
  10. 10.
    Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. Mach. Learn. 42, 177–196 (2001)CrossRefGoogle Scholar
  11. 11.
    Hsu, W.S., Poupart, P.: Online Bayesian moment matching for topic modeling with unknown number of topics. In: Proceedings of the 30th International Conference on Neural Information Processing Systems (2016)Google Scholar
  12. 12.
    Hu, Y., Boyd-Graber, J., Satinoff, B.: Interactive topic modeling. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pp. 248–257 (2011)Google Scholar
  13. 13.
    Lacoste-Julien, S., Sha, F., Jordan, M.I.: DiscLDA: discriminative learning for dimensionality reduction and classification. In: Proceedings of the 21st International Conference on Neural Information Processing Systems, pp. 897–904 (2008)Google Scholar
  14. 14.
    Newman, D., Bonilla, E.V., Buntine, W.L.: Improving topic coherence with regularized topic models. In: Proceedings of the 24th International Conference on Neural Information Processing Systems (2011)Google Scholar
  15. 15.
    Papadimitriou, C.H., Raghavan, P., Tamaki, H., Vempala, S.: Latent semantic indexing: a probabilistic analysis. J. Comput. Syst. Sci. 61(2), 217–235 (2000)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Qiang, J., Chen, P., Wang, T., Wu, X.: Topic modeling over short texts by incorporating word embeddings. In: Kim, J., Shim, K., Cao, L., Lee, J.-G., Lin, X., Moon, Y.-S. (eds.) PAKDD 2017. LNCS (LNAI), vol. 10235, pp. 363–374. Springer, Cham (2017). Scholar
  17. 17.
    Teh, Y.W., Newman, D., Welling, M.: A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In: Proceedings of the 19th International Conference on Neural Information Processing Systems, pp. 1353–1360 (2006)Google Scholar
  18. 18.
    Wang, C., Blei, D.M., Heckerman, D.: Continuous time dynamic topic models. In: Proceedings of the 24th Conference on Uncertainty in Artificial Intelligence (2008)Google Scholar
  19. 19.
    Wei, L., Blei, D.M., McCallum, A.: Nonparametric Bayes pachinko allocation. In: Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence, pp. 243–250 (2007)Google Scholar
  20. 20.
    Wood, J., Tan, P., Wang, W., Arnold, C.: Source-LDA: enhancing probabilistic topic models using prior knowledge sources. In: IEEE 33rd International Conference on Data Engineering (ICDE) (2017)Google Scholar
  21. 21.
    Xuerui, X., McCallum, A.: Topics over time: a non-Markov continuous-time model of topical trends. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 424–433 (2006)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  • Sattar Seifollahi
    • 1
    • 2
  • Massimo Piccardi
    • 1
  • Ehsan Zare Borzeshi
    • 2
  1. 1.Faculty of Engineering and Information TechnologyUniversity of Technology SydneySydneyAustralia
  2. 2.Capital Markets Cooperative Research Centre (CMCRC)SydneyAustralia

Personalised recommendations