Abstract
The users often have additional knowledge when Bayesian nonparametric models (BNP) are employed, e.g. for clustering there may be prior knowledge that some of the data instances should be in the same cluster (must-link constraint) or in different clusters (cannot-link constraint), and similarly for topic modeling some words should be grouped together or separately because of an underlying semantic. This can be achieved by imposing appropriate sampling probabilities based on such constraints. However, the traditional inference technique of BNP models via Gibbs sampling is time consuming and is not scalable for large data. Variational approximations are faster but many times they do not offer good solutions. Addressing this we present a small-variance asymptotic analysis of the MAP estimates of BNP models with constraints. We derive the objective function for Dirichlet process mixture model with constraints and devise a simple and efficient K-means type algorithm. We further extend the small-variance analysis to hierarchical BNP models with constraints and devise a similar simple objective function. Experiments on synthetic and real data sets demonstrate the efficiency and effectiveness of our algorithms.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Antoniak, C.E.: Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems. The Annals of Statistics 2(6), 1152–1174 (1974)
Teh, Y.W., Jordan, M.I., Beal, M.J., Blei, D.M.: Hierarchical dirichlet processes. JASA 101, 1566–1581 (2006)
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)
Wagsta, K., Cardie, C., Schroedl, S.: Constrained k-means clustering with background knowledge. In: Proc. 18th International Conf. on Machine Learning, pp. 577–584 (2001)
Wagstaff, K., Cardie, C.: Clustering with instance-level constraints. In: Proceedings of the Seventeenth International Conference on Machine Learning, pp. 1103–1110 (2000)
Basu, S., Davidson, I., Wagstaff, K.L. (eds.) Constrained Clustering: Advances in Algorithms, Theory, and Applications, 1st ed. Chapman and Hall/CRC, August 2008
Vlachos, A., Korhonen, A., Ghahramani, Z.: Unsupervised and constrained Dirichlet process mixture models for verb clustering. In: GEMS 2009 (2009)
Orbanz, P., Buhmann, J.M.: Nonparametric bayesian image segmentation. Int. J. Comput. Vision 77(1–3), 25–45 (2008)
Ross, J., Dy, J.: Nonparametric mixture of gaussian processes with constraints. In: ICML, JMLR Workshop and Conference Proceedings. vol. 28, no. 3, pp. 1346–1354 (2013)
Aldous, D.: Exchangeability and related topics. In: Ecole d’Ete de Probabilities de Saint-Flour XIII 1983, pp. 1–198. Springer (1985)
Shental, N., Bar-hillel, A., Hertz, T., Weinshall, D.: Computing gaussian mixture models with em using equivalence constraints. In: NIPS. MIT Press (2003)
Jiang, K., Kulis, B., Jordan, M.I.: Small-variance asymptotics for exponential family dirichlet process mixture models. In: NIPS, pp. 3167–3175 (2012)
Agarwal, A., Daumé III, H.: A geometric view of conjugate priors. Mach. Learn. 81(1), 99–113 (2010)
Banerjee, A., Merugu, S., Dhillon, I.S., Ghosh, J.: Clustering with bregman divergences. J. Mach. Learn. Res. 6, 1705–1749 (2005)
Neal, R.M.: Markov chain sampling methods for dirichlet process mixture models. JCGS 9(2), 249–265 (2000)
Geman, S., Geman, D.: Stochastic relaxation, gibbs distributions, and the bayesian restoration of images, TPAMI, vol. PAMI-6, no. 6, pp. 721–741 (1984)
Kulis, B., Jordan, M.I.: Revisiting k-means: New algorithms via bayesian nonparametrics. In: ICML (2012)
Broderick, T., Kulis, B., Jordan, M.I.: Mad-bayes: Map-based asymptotic derivations from bayes. In: ICML (2013)
Klein, D., Kamvar, S.D., Manning, C.D.: From instance-level constraints to space-level constraints: making the most of prior knowledge in data clustering. In: ICML (2002)
Davidson, I., Wagstaff, K.L., Basu, S.: Measuring constraint-set utility for partitional clustering algorithms. In: ECML/PKDD (2006)
Bache, K., Lichman, M.: UCI machine learning repository (2013). http://archive.ics.uci.edu/ml
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Li, C., Rana, S., Phung, D., Venkatesh, S. (2015). Small-Variance Asymptotics for Bayesian Nonparametric Models with Constraints. In: Cao, T., Lim, EP., Zhou, ZH., Ho, TB., Cheung, D., Motoda, H. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2015. Lecture Notes in Computer Science(), vol 9078. Springer, Cham. https://doi.org/10.1007/978-3-319-18032-8_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-18032-8_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-18031-1
Online ISBN: 978-3-319-18032-8
eBook Packages: Computer ScienceComputer Science (R0)