Non-parametric Mixture Models for Clustering

  • Pavan Kumar Mallapragada
  • Rong Jin
  • Anil Jain
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6218)

Abstract

Mixture models have been widely used for data clustering. However, commonly used mixture models are generally of a parametric form (e.g., mixture of Gaussian distributions or GMM), which significantly limits their capacity in fitting diverse multidimensional data distributions encountered in practice. We propose a non-parametric mixture model (NMM) for data clustering in order to detect clusters generated from arbitrary unknown distributions, using non-parametric kernel density estimates. The proposed model is non-parametric since the generative distribution of each data point depends only on the rest of the data points and the chosen kernel. A leave-one-out likelihood maximization is performed to estimate the parameters of the model. The NMM approach, when applied to cluster high dimensional text datasets significantly outperforms the state-of-the-art and classical approaches such as K-means, Gaussian Mixture Models, spectral clustering and linkage methods.

Keywords

Cluster Algorithm Mixture Model Gaussian Mixture Model Latent Dirichlet Allocation Spectral Cluster 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Jain, A.K.: Data clustering: 50 years beyond k-means. Pattern Recognition Letters 31, 651–666 (2010)CrossRefGoogle Scholar
  2. 2.
    McLachlan, G.L., Peel, D.: Finite Mixture Models. Wiley, Chichester (2000)MATHCrossRefGoogle Scholar
  3. 3.
    Figueiredo, M.A.T., Jain, A.K.: Unsupervised learning of finite mixture models. TPAMI 24, 381–396 (2002)Google Scholar
  4. 4.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. Journal of Machine Learning Research 3, 993–1022 (2003)MATHCrossRefGoogle Scholar
  5. 5.
    Jain, A.K., Dubes, R.C.: Algorithms for Clustering Data. Prentice Hall, Englewood Cliffs (1988)MATHGoogle Scholar
  6. 6.
    Jarvis, R.A., Patrick, E.A.: Clustering using a similarity measure based on shared near neighbors. IEEE Transactions on Computers 22 (1973)Google Scholar
  7. 7.
    Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proc. KDD, pp. 226–231 (1996)Google Scholar
  8. 8.
    Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 603–619 (2002)CrossRefGoogle Scholar
  9. 9.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)MATHGoogle Scholar
  10. 10.
    Andreetto, M., Zelnik Manor, L., Perona, P.: Non-parametric probabilistic image segmentation. In: Proceedings of the ICCV, pp. 1–8 (2007)Google Scholar
  11. 11.
    Shawe-Taylor, J., Dolia, A.N.: A framework for probability density estimation. In: Proc. AISTATS, pp. 468–475 (2007)Google Scholar
  12. 12.
    Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. Wiley, Chichester (2001)MATHGoogle Scholar
  13. 13.
    Wand, M.P., Jones, M.C.: Kernel Smoothing (Monographs on Statistics and Applied Probability, December 1994. Chapman & Hall/CRC, Boca Raton (1994)Google Scholar
  14. 14.
    Csiszar, I., Tusnady, G.: Information geometry and alternating minimization procedures. Statistics and Decision (1984)Google Scholar
  15. 15.
    Jaakkola, T.S.: Tutorial on variational approximation methods. In: Advanced Mean Field Methods: Theory and Practice, pp. 129–159 (2000)Google Scholar
  16. 16.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 888–905 (2000)CrossRefGoogle Scholar
  17. 17.
    Slonim, N., Tishby, N.: Agglomerative information bottleneck. In: Advances in NIPS (2000)Google Scholar
  18. 18.
    Nadler, B., Galun, M.: Fundamental limitations of spectral clustering. In: NIPS 19, Citeseer, pp. 1017–1025 (2007)Google Scholar
  19. 19.
    Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: Analysis and an algorithm. In: Advances in NIPS, pp. 849–856. MIT Press, Cambridge (2001)Google Scholar
  20. 20.
    Banerjee, A., Langford, J.: An objective evaluation criterion for clustering. In: Proceedings of the KDD, pp. 515–520 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Pavan Kumar Mallapragada
    • 1
  • Rong Jin
    • 1
  • Anil Jain
    • 1
  1. 1.Department of Computer Science and EngineeringMichigan State UniversityEast Lansing

Personalised recommendations