Advertisement

A Binary Optimization Approach for Constrained K-Means Clustering

  • Huu M. LeEmail author
  • Anders Eriksson
  • Thanh-Toan Do
  • Michael Milford
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11364)

Abstract

K-Means clustering still plays an important role in many computer vision problems. While the conventional Lloyd method, which alternates between centroid update and cluster assignment, is primarily used in practice, it may converge to solutions with empty clusters. Furthermore, some applications may require the clusters to satisfy a specific set of constraints, e.g., cluster sizes, must-link/cannot-link. Several methods have been introduced to solve constrained K-Means clustering. Due to the non-convex nature of K-Means, however, existing approaches may result in sub-optimal solutions that poorly approximate the true clusters. In this work, we provide a new perspective to tackle this problem by considering constrained K-Means as a special instance of Binary Optimization. We then propose a novel optimization scheme to search for feasible solutions in the binary domain. This approach allows us to solve constrained K-Means clustering in such a way that multiple types of constraints can be simultaneously enforced. Experimental results on synthetic and real datasets show that our method provides better clustering accuracy with faster run time compared to several existing techniques.

References

  1. 1.
    Althoff, T., Ulges, A., Dengel, A.: Balanced clustering for content-based image browsing. Series of the Gesellschaft fur Informatik, pp. 27–30 (2011)Google Scholar
  2. 2.
    Bertacco, L., Fischetti, M., Lodi, A.: A feasibility pump heuristic for general mixed-integer problems. Discret. Optim. 4(1), 63–76 (2007)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bradley, P., Bennett, K., Demiriz, A.: Constrained K-Means Clustering, pp. 1–8. Microsoft Research, Redmond (2000)Google Scholar
  4. 4.
    Fard, M.M., Thonet, T., Gaussier, E.: Deep \(k\)-means: jointly clustering with \(k\)-means and learning representations. arXiv preprint arXiv:1806.10069 (2018)
  5. 5.
    Fischetti, M., Glover, F., Lodi, A.: The feasibility pump. Math. Program. 104(1), 91–104 (2005)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Ge, T., He, K., Ke, Q., Sun, J.: Optimized product quantization for approximate nearest neighbor search. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2946–2953. IEEE (2013)Google Scholar
  7. 7.
    Geißler, B., Morsi, A., Schewe, L., Schmidt, M.: Penalty alternating direction methods for mixed-integer optimization: a new view on feasibility pumps. SIAM J. Optim. 27(3), 1611–1636 (2017)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Gersho, A., Gray, R.M.: Vector Quantization and Signal Compression, vol. 159. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-1-4615-3626-0CrossRefzbMATHGoogle Scholar
  9. 9.
    Jegou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 117–128 (2011)CrossRefGoogle Scholar
  10. 10.
    Johnson, S.C.: Hierarchical clustering schemes. Psychometrika 32(3), 241–254 (1967)CrossRefGoogle Scholar
  11. 11.
    Kalantidis, Y., Avrithis, Y.: Locally optimized product quantization for approximate nearest neighbor search. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2321–2328 (2014)Google Scholar
  12. 12.
    Khan, S.S., Ahmad, A.: Cluster center initialization algorithm for k-means clustering. Pattern Recogn. Lett. 25(11), 1293–1302 (2004)CrossRefGoogle Scholar
  13. 13.
    Le Tan, D.K., Le, H., Hoang, T., Do, T.T., Cheung, N.M.: DeepVQ: a deep network architecture for vector quantization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2579–2582 (2018)Google Scholar
  14. 14.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  15. 15.
    Li, Z., Liu, J.: Constrained clustering by spectral kernel learning. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 421–427. IEEE (2009)Google Scholar
  16. 16.
    Liu, H., Han, J., Nie, F., Li, X.: Balanced clustering with least square regression (2017)Google Scholar
  17. 17.
    MacQueen, J., et al.: Some methods for classification and analysis of multivariate observations. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Oakland, CA, USA, vol. 1, pp. 281–297 (1967)Google Scholar
  18. 18.
    Mahajan, M., Nimbhorkar, P., Varadarajan, K.: The planar k-means problem is NP-hard. Theor. Comput. Sci. 442, 13–21 (2012)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Malinen, M.I., Fränti, P.: Balanced K-means for clustering. In: Fränti, P., Brown, G., Loog, M., Escolano, F., Pelillo, M. (eds.) S+SSPR 2014. LNCS, vol. 8621, pp. 32–41. Springer, Heidelberg (2014).  https://doi.org/10.1007/978-3-662-44415-3_4CrossRefGoogle Scholar
  20. 20.
    Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12(Oct), 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  21. 21.
    Pena, J.M., Lozano, J.A., Larranaga, P.: An empirical comparison of four initialization methods for the k-means algorithm. Pattern Recogn. Lett. 20(10), 1027–1040 (1999)CrossRefGoogle Scholar
  22. 22.
    Rifkin, R.M., Lippert, R.A.: Notes on regularized least squares (2007)Google Scholar
  23. 23.
    Wagstaff, K., Cardie, C., Rogers, S., Schroedl, S.: Constrained k-means clustering with background knowledge. In: Proceedings of the Eighteenth International Conference on Machine Learning, pp. 577–584. Citeseer (2001)Google Scholar
  24. 24.
    Wright, S., Nocedal, J.: Numerical optimization. Springer Sci. 35(67–68), 7 (1999)zbMATHGoogle Scholar
  25. 25.
    Yang, B., Fu, X., Sidiropoulos, N.D., Hong, M.: Towards k-means-friendly spaces: simultaneous deep learning and clustering. arXiv preprint arXiv:1610.04794 (2016)
  26. 26.
    Zhu, S., Wang, D., Li, T.: Data clustering with size constraints. Knowl.-Based Syst. 23(8), 883–889 (2010)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Huu M. Le
    • 1
    Email author
  • Anders Eriksson
    • 1
  • Thanh-Toan Do
    • 2
  • Michael Milford
    • 1
  1. 1.Queensland University of TechnologyBrisbaneAustralia
  2. 2.University of LiverpoolLiverpoolEngland

Personalised recommendations