Abstract
Several authors in recent years have proposed discrete analogues to principle component analysis intended to handle discrete or positive only data, for instance suited to analyzing sets of documents. Methods include non-negative matrix factorization, probabilistic latent semantic analysis, and latent Dirichlet allocation. This paper begins with a review of the basic theory of the variational extension to the expectation-maximization algorithm, and then presents discrete component finding algorithms in that light. Experiments are conducted on both bigram word data and document bag-of-word to expose some of the subtleties of this new class of algorithms.
Chapter PDF
References
Tipping, M., Bishop, C.: Mixtures of probabilistic principal component analysers. Neural Computation 11 (1999) 443–482
Hofmann, T.: Probabilistic latent semantic indexing. In: Research and Development in Information Retrieval. (1999) 50–57
Lee, D., Seung, H.: Learning the parts of objects by non-negative matrix factorization. Nature 401 (1999) 788–791
Blei, D., Ng, A., Jordan, M.: Latent Dirichlet allocation. In: NIPS*14. (2002) to appear.
Collins, M., Dasgupta, S., Schapire, R.: A generalization of principal component analysis to the exponential family. In: NIPS*13. (2001)
Hall, K., Hofmann, T.: Learning curved multinomial subfamilies for natural language processing and information retrieval. In: ICML 2000. (2000)
Sjolander, K., Karplus, K., Brown, M., Hughey, R., Krogh, A., Mian, I. S., Haussler, D.: Dirichlet mixtures: A method for improved detection of weak but signicant protein sequence homology. Computer Applications in the Biosciences 12 (1996) 327–345
Brand, M.: Structure learning in conditional probability models via an entropic prior and parameter extinction. Neural Computation 11 (1999) 1155–1182
Roweis, S.: EM algorithms for PCA and SPCA. In: NIPS*10. (1998)
Ghahramani, Z., Beal, M.: Propagation algorithms for variational Bayesian learning. In: NIPS. (2000) 507–513
Buntine, W.: Computation with the exponential family and graphical models. unpublished handouts, NATO Summer School on Graphical Models, Erice, Italy (1996)
Minka, T.: Estimating a Dirichlet distribution. Course notes (2000)
Jordan, M., Ghahramani, Z., Jaakkola, T., Saul, L.: An introduction to variational methods for graphical models. Machine Learning 37 (1999) 183–233
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Buntine, W. (2002). Variational Extensions to EM and Multinomial PCA. In: Elomaa, T., Mannila, H., Toivonen, H. (eds) Machine Learning: ECML 2002. ECML 2002. Lecture Notes in Computer Science(), vol 2430. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36755-1_3
Download citation
DOI: https://doi.org/10.1007/3-540-36755-1_3
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44036-9
Online ISBN: 978-3-540-36755-0
eBook Packages: Springer Book Archive