Abstract
Clustering is an unsupervised classification technique that identifies some inherent structure present in a set of objects based on a similarity measure. Clustering methods can be derived from statistical models or competitive learning and correspondingly they can be classified into generative (or model-based) and discriminative (or similarity-based) approaches. A clustering problem can also be modeled as a COP. Clustering neural networks are statistical models, where a probability density function (pdf) for data is estimated by learning its parameters. In this chapter, our emphasis is placed on a number of competitive learning-based neural networks and clustering algorithms. We describe the SOM, learning vector quantization (LVQ), and ART models, as well as C-means, subtractive, and fuzzy clustering algorithms.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The link distance between two points A and B inside a polygon P is defined to be the minimum number of edges required to connect A and B inside P.
References
Alex, N., Hasenfuss, A., & Hammer, B. (2009). Patch clustering for massive data sets. Neurocomputing, 72, 1455–1469.
Anderson, I. A., Bezdek, J. C., & Dave, R. (1982). Polygonal shape description of plane boundaries. In L. Troncale (Ed.), Systems science and science (Vol. 1, pp. 295–301). Louisville, KY: SGSR.
Angelov, P. P., & Filev, D. P. (2004). An approach to online identification of Takagi–Sugeno fuzzy models. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 34(1), 484–498.
Aoki, T., & Aoyagi, T. (2007). Self-organizing maps with asymmetric neighborhood function. Neural Computation, 19, 2515–2535.
Atiya, A. F. (2005). Estimating the posterior probabilities using the \(K\)-nearest neighbor rule. Neural Computation, 17, 731–740.
Baraldi, A., & Blonda, P. (1999). A survey of fuzzy clustering algorithms for pattern recognition—Part II. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 29(6), 786–801.
Bax, E. (2012). Validation of \(k\)-nearest neighbor classifiers. IEEE Transactions on Information Theory, 58(5), 3225–3234.
Beliakov, G., & Li, G. (2012). Improving the speed and stability of the \(k\)-nearest neighbors method. Pattern Recognition Letters, 33, 1296–1301.
Beni, G., & Liu, X. (1994). A least biased fuzzy clustering method. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(9), 954–960.
Berglund, E., & Sitte, J. (2006). The parameterless self-organizing map algorithm. IEEE Transactions on Neural Networks, 17(2), 305–316.
Bermejo, S. (2006). The regularized LVQ1 algorithm. Neurocomputing, 70, 475–488.
Bezdek, J. (1981). Pattern recognition with fuzzy objective function algorithms. New York: Plenum Press.
Biehl, M., Ghosh, A., & Hammer, B. (2007). Dynamics and generalization ability of LVQ algorithms. Journal of Machine Learning Research, 8, 323–360.
Brodal, A. (1981). Neurological anatomy in relation to clinical medicine (3rd ed.). New York: Oxford University Press.
Burke, L. I. (1991). Clustering characterization of adaptive resonance. Neural Networks, 4(4), 485–491.
Carpenter, G., Grossberg, S., & Rosen, D. B. (1991). ART 2-A: An adaptive resonance algorithm for rapid category learning and recognition. Neural Networks, 4, 493–504.
Carpenter, G. A., & Grossberg, S. (1987). A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing, 37, 54–115.
Carpenter, G. A., & Grossberg, S. (1987). ART 2: Self-organization of stable category recognition codes for analog input patterns. Applied Optics, 26, 4919–4930.
Carpenter, G. A., Grossberg, S., & Reynolds, J. H. (1991). ARTMAP: Supervised real-time learning and classification of nonstationary data by a self-organizing neural network. Neural Networks, 4(5), 565–588.
Carpenter, G. A., Grossberg, S., Markuzon, N., Reynolds, J. H., & Rosen, D. B. (1992). Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Transactions on Neural Networks, 3, 698–713.
Cheng, Y. (1997). Convergence and ordering of Kohonen’s batch map. Neural Computation, 9, 1667–1676.
Chinrunrueng, C., & Sequin, C. H. (1995). Optimal adaptive \(k\)-means algorithm with dynamic adjustment of learning rate. IEEE Transactions on Neural Networks, 6(1), 157–169.
Chiu, S. (1994). Fuzzy model identification based on cluster estimation. Journal of Intelligent and Fuzzy Systems, 2(3), 267–278.
Chiu, S. L. (1994). A cluster estimation method with extension to fuzzy model identification. In Proceedings of the IEEE International Conference on Fuzzy Systems (Vol. 2, pp. 1240–1245). Orlando, FL.
Cho, J., Paiva, A. R. C., Kim, S.-P., Sanchez, J. C., & Principe, J. C. (2007). Self-organizing maps with dynamic learning for signal reconstruction. Neural Networks, 20, 274–284.
Choy, C. S. T., & Siu, W. C. (1998). Fast sequential implementation of “neural-gas” network for vector quantization. IEEE Transactions on Communications, 46(3), 301–304.
Cottrell, M., Hammer, B., Hasenfuss, A., & Villmann, T. (2006). Batch and median neural gas. Neural Networks, 19, 762–771.
Cover, T. M., & Hart, P. E. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13, 21–27.
Dave, R. N., & Krishnapuram, R. (1997). Robust clustering methods: A unified view. IEEE Transactions on Fuzzy Systems, 5(2), 270–293.
de Carvalho, F. A. T., Tenorio, C. P., & Cavalcanti, N. L, Jr. (2006). Partitional fuzzy clustering methods based on adaptive quadratic distances. Fuzzy Sets and Systems, 157, 2833–2857.
Demartines, P., & Herault, J. (1997). Curvilinear component analysis: A self-organizing neural network for nonlinear mapping of data sets. IEEE Transactions on Neural Networks, 8(1), 148–154.
Desarbo, W. S., Carroll, J. D., Clark, L. A., & Green, P. E. (1984). Synthesized clustering: A method for amalgamating clustering bases with differential weighting variables. Psychometrika, 49, 57–78.
Ding, C., & He, X. (2004). Cluster structure of \(k\)-means clustering via principal component analysis. In Proceedings of the 8th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining (PAKDD), LNCS (Vol. 3056, pp. 414–418). Sydney, Australia; Berlin: Springer.
Du, K.-L., & Swamy, M. N. S. (2006). Neural networks in a softcomputing framework. London: Springer.
Du, K.-L. (2010). Clustering: A neural network approach. Neural Networks, 23(1), 89–107.
Dunn, J. C. (1974). Some recent investigations of a new fuzzy partitioning algorithm and its applicatiopn to pattern classification problems. Journal of Cybernetics, 4, 1–15.
Estevez, P. A., & Figueroa, C. J. (2006). Online data visualization using the neural gas network. Neural Networks, 19, 923–934.
Fayed, H. A., & Atiya, A. F. (2009). A novel template reduction approach for the \(K\)-nearest neighbor method. IEEE Transactions on Neural Networks, 20(5), 890–896.
Fix, E., & Hodges, J. L., Jr. (1951). Discriminatory analysis—Nonparametric discrimination: Consistency properties. Project No. 2-49-004, Report No. 4. Randolph Field, TX: USAF School of Aviation (Reprinted in International Statistical Review, 57(3), 238–247 (1989)).
Flanagan, J. A. (1996). Self-organization in Kohonen’s SOM. Neural Networks, 9(7), 1185–1197.
Fort, J. C. (1988). Solving a combinatorial problem via self-organizing process: An application of Kohonen-type neural networks to the travelling salesman problem. Biology in Cybernetics, 59, 33–40.
Fritzke, B. (1997). The LBG-U method for vector quantization—An improvement over LBG inspired from neural networks. Neural Processing Letters, 5(1), 35–45.
Fukushima, K. (2010). Neocognitron trained with winner-kill-loser rule. Neural Networks, 23, 926–938.
Gath, I., & Geva, A. B. (1989). Unsupervised optimal fuzzy clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7), 773–781.
Gersho, A. (1979). Asymptotically optimal block quantization. IEEE Transactions on Information Theory, 25(4), 373–380.
Ghosh, A., Biehl, M., & Hammer, B. (2006). Performance analysis of LVQ algorithms: A statistical physics approach. Neural Networks, 19, 817–829.
Gottlieb, L.-A., Kontorovich, A., & Krauthgamer, R. (2014). Efficient classification for metric data. IEEE Transactions on Information Theory, 60(9), 5750–5759.
Grossberg, S. (1976). Adaptive pattern classification and universal recording: I. Parallel development and coding of neural feature detectors; II. Feedback, expectation, olfaction, and illusions. Biological Cybernetics, 23, 121–34, 187–202.
Guo, Y., & Sengur, A. (2015). NCM: Neutrosophic \(c\)-means clustering algorithm. Pattern Recognition, 48(8), 2710–2724.
Gustafson, D. E., & Kessel, W. (1979). Fuzzy clustering with a fuzzy covariance matrix. In Proceedings of the IEEE Conference on Decision and Control (pp. 761–766). San Diego, CA.
Hall, L. O., & Goldgof, D. B. (2011). Convergence of the single-pass and online fuzzy \(C\)-means algorithms. IEEE Transactions on Fuzzy Systems, 19(4), 792–794.
Hammer, B., Strickert, M., & Villmann, T. (2005). Supervised neural gas with general similarity measure. Neural Processing Letters, 21(1), 21–44.
Hara, T., & Hirose, A. (2004). Plastic mine detecting radar system using complex-valued self-organizing map that deals with multiple-frequency interferometric images. Neural Networks, 17, 1201–1210.
Hart, P. E. (1968). The condensed nearest neighbor rule. IEEE Transactions on Information Theory, 14(3), 515–516.
Hathaway, R. J., & Bezdek, J. C. (1993). Switching regression models and fuzzy clustering. IEEE Transactions on Fuzzy Systems, 1, 195–204.
Hathaway, R. J., & Bezdek, J. C. (2000). Generalized fuzzy \(c\)-means clustering strategies using \(L_p\) norm distances. IEEE Transactions on Fuzzy Systems, 8(5), 576–582.
He, J., Tan, A. H., & Tan, C. L. (2004). Modified ART 2A growing network capable of generating a fixed number of nodes. IEEE Transactions on Neural Networks, 15(3), 728–737.
Huang, J. Z., Ng, M. K., Rong, H., & Li, Z. (2005). Automated variable weighting in k-means type clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(5), 657–668.
Huang, P., & Zhang, D. (2010). Locality sensitive \(C\)-means clustering algorithms. Neurocomputing, 73, 2935–2943.
Hueter, G. J. (1988). Solution of the traveling salesman problem with an adaptive ring. In Proceedings of International Conference on Neural Networks (pp. 85–92). San Diego, CA.
Hwang, C., & Rhee, F. (2007). Uncertain fuzzy clustering: Interval type-2 fuzzy approach to \(C\)-means. IEEE Transactions on Fuzzy Systems, 15(1), 107–120.
Kanungo, T., Mount, D. M., Netanyahu, N. S., Piatko, C. D., Silverman, R., & Wu, A. Y. (2002). An efficient \(k\)-means clustering algorithm: Analysis and implementation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7), 881–892.
Karayiannis, N. B., & Randolph-Gips, M. M. (2003). Soft learning vector quantization and clustering algorithms based on non-Euclidean norms: Multinorm algorithms. IEEE Transactions on Neural Networks, 14(1), 89–102.
Kaymak, U., & Setnes, M. (2002). Fuzzy clustering with volume prototypes and adaptive cluster merging. IEEE Transactions on Fuzzy Systems, 10(6), 705–712.
Kersten, P. R. (1999). Fuzzy order statistics and their application to fuzzy clustering. IEEE Transactions on Fuzzy Systems, 7(6), 708–712.
Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological Cybernetics, 43, 59–69.
Kohonen, T. (1989). Self-organization and associative memory. Berlin: Springer.
Kohonen, T. (1990). The self-organizing map. Proceedings of the IEEE, 78, 1464–1480.
Kohonen, T. (1990). Improved versions of learning vector quantization. In Proceedings of the International Joint Conference on Neural Networks (IJCNN) (Vol. 1, pp. 545–550). San Diego, CA.
Kohonen, T. (1990). Derivation of a class of training algorithms. IEEE Transactions on Neural Networks, 1, 229–232.
Kohonen, T. (1996). Emergence of invariant-feature detectors in the adaptive-subspace self-organizing map. Biological Cybernetics, 75, 281–291.
Kohonen, T. (2001). Self-organizing maps (3rd ed.). Berlin: Springer.
Kohonen, T., Kangas, J., Laaksonen, J., & Torkkola, K. (1992). LVQPAK: A program package for the correct application of learning vector quantization algorithms. In Proceedings of the International Joint Conference on Neural Networks (IJCNN) (Vol. 1, pp. 725–730). Baltimore, MD.
Kohonen, T. (2006). Self-organizing neural projections. Neural Networks, 19, 723–733.
Kolen, J., & Hutcheson, T. (2002). Reducing the time complexity of the fuzzy \(C\)-means algorithm. IEEE Transactions on Fuzzy Systems, 10(2), 263–267.
Krishnapuram, R., & Kim, J. (2000). Clustering algorithms based on volume criteria. IEEE Transactions on Fuzzy Systems, 8(2), 228–236.
Kusumoto, H., & Takefuji, Y. (2006). \(O(\log _2 M)\) self-organizing map algorithm without learning of neighborhood vectors. IEEE Transactions on Neural Networks, 17(6), 1656–1661.
Laaksonen, J., & Oja, E. (1996). Classification with learning k-nearest neighbors. In Proceedings of the International Conference on Neural Networks (pp. 1480–1483). Washington, DC.
Lai, J. Z. C., Huang, T.-J., & Liaw, Y.-C. (2009). A fast \(k\)-means clustering algorithm using cluster center displacement. Pattern Recognition, 42, 2551–2556.
Leski, J. (2003). Towards a robust fuzzy clustering. Fuzzy Sets and Systems, 137, 215–233.
Li, M. J., Ng, M. K., Cheung, Y.-M., & Huang, J. Z. (2008). Agglomerative fuzzy \(K\)-means clustering algorithm with selection of number of clusters. IEEE Transactions on Knowledge and Data Engineering, 20(11), 1519–1534.
Likas, A., Vlassis, N., & Verbeek, J. J. (2003). The global \(k\)-means clustering algorithm. Pattern Recognition, 36(2), 451–461.
Linda, O., & Manic, M. (2012). General type-2 fuzzy \(C\)-means algorithm for uncertain fuzzy clustering. IEEE Transactions on Fuzzy Systems, 20(5), 883–897.
Linde, Y., Buzo, A., & Gray, R. M. (1980). An algorithm for vector quantizer design. IEEE Transactions on Communications, 28, 84–95.
Lippman, R. P. (1987). An introduction to computing with neural nets. IEEE ASSP Magazine, 4(2), 4–22.
Lo, Z. P., & Bavarian, B. (1991). On the rate of convergence in topology preserving neural networks. Biological Cybernetics, 65, 55–63.
Luttrell, S. P. (1990). Derivation of a class of training algorithms. IEEE Transactions on Neural Networks, 1, 229–232.
Luttrell, S. P. (1994). A Bayesian analysis of self-organizing maps. Neural Computation, 6, 767–794.
MacQueen, J. B. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability (pp. 281–297). Berkeley, CA: University of California Press.
Martinetz, T. M. (1993). Competitive Hebbian learning rule forms perfectly topology preserving maps. In Proceedings of the International Conference on Artificial Neural Networks (ICANN) (pp. 427–434). Amsterdam, The Netherlands.
Martinetz, T. M., Berkovich, S. G., & Schulten, K. J. (1993). Neural-gas network for vector quantization and its application to time-series predictions. IEEE Transactions on Neural Networks, 4(4), 558–569.
Martinetz, T. M., & Schulten, K. J. (1994). Topology representing networks. Neural Networks, 7, 507–522.
Moore, B. (1988). ART and pattern clustering. In D. Touretzky, G. Hinton & T. Sejnowski (Eds.), Proceedings of the 1988 Connectionist Model Summer School (pp. 174–183). San Mateo, CA: Morgan Kaufmann.
Mulder, S. A., & Wunsch, D. C, I. I. (2003). Million city traveling salesman problem solution by divide and conquer clustering with adaptive resonance neural networks. Neural Networks, 16, 827–832.
Obermayer, K., Ritter, H., & Schulten, K. (1991). Development and spatial structure of cortical feature maps: A model study. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems (Vol. 3, pp. 11–17). San Mateo, CA: Morgan Kaufmann.
Odorico, R. (1997). Learning vector quantization with training count (LVQTC). Neural Networks, 10(6), 1083–1088.
Pal, N. R., Bezdek, J. C., & Tsao, E. C. K. (1993). Generalized clustering networks and Kohonen’s self-organizing scheme. IEEE Transactions on Neural Networks, 4(2), 549–557.
Pal, N. R., & Chakraborty, D. (2000). Mountain and subtractive clustering method: Improvements and generalizations. International Journal of Intelligent Systems, 15, 329–341.
Patane, G., & Russo, M. (2001). The enhanced LBG algorithm. Neural Networks, 14(9), 1219–1237.
Pedrycz, W., & Waletzky, J. (1997). Fuzzy clustering with partial supervision. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 27(5), 787–795.
Richardt, J., Karl, F., & Muller, C. (1998). Connections between fuzzy theory, simulated annealing, and convex duality. Fuzzy Sets and Systems, 96, 307–334.
Rodriguez, A., & Laio, A. (2014). Clustering by fast search and find of density peaks. Science, 344(6191), 1492–1496.
Rose, K. (1998). Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proceedings of the IEEE, 86(11), 2210–2239.
Rose, K., Gurewitz, E., & Fox, G. C. (1990). A deterministic annealing approach to clustering. Pattern Recognition Letters, 11(9), 589–594.
Sato, A., & Yamada, K. (1995). Generalized learning vector quantization. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems (Vol. 7, pp. 423–429). Cambridge, MA: MIT Press.
Serrano-Gotarredona, T., & Linares-Barranco, B. (1996). A modified ART 1 algorithm more suitable for VLSI implementations. Neural Networks, 9(6), 1025–1043.
Seo, S., & Obermayer, K. (2003). Soft learning vector quantization. Neural Computation, 15, 1589–1604.
Shalev-Shwartz, S., & Ben-David, S. (2014). Understanding machine learning: From theory to algorithms. Cambridge, UK: Cambridge University Press.
Su, Z., & Denoeux, T. (2019). BPEC: Belief-peaks evidential clustering. IEEE Transactions on Fuzzy Systems, 27(1), 111–123.
Tsekouras, G., Sarimveis, H., Kavakli, E., & Bafas, G. (2004). A hierarchical fuzzy-clustering approach to fuzzy modeling. Fuzzy Sets and Systems, 150(2), 245–266.
Verzi, S. J., Heileman, G. L., Georgiopoulos, M., & Anagnostopoulos, G. C. (2003). Universal approximation with fuzzy ART and fuzzy ARTMAP. In Proceedings of the International Joint Conference on Neural Networks (IJCNN) (Vol. 3, pp. 1987–1892). Portland, OR.
von der Malsburg, C. (1973). Self-organizing of orientation sensitive cells in the striata cortex. Kybernetik, 14, 85–100.
Wilson, D. L. (1972). Asymptotic properties of nearest neighbor rules using edited data. IEEE Transactions on Systems, Man, and Cybernetics, 2(3), 408–420.
Witoelar, A., Biehl, M., Ghosh, A., & Hammer, B. (2008). Learning dynamics and robustness of vector quantization and neural gas. Neurocomputing, 71, 1210–1219.
Wu, K.-L., Yang, M.-S., & Hsieh, J.-N. (2010). Mountain \(c\)-regressions method. Pattern Recognition, 43, 86–98.
Yager, R. R., & Filev, D. (1994). Approximate clustering via the mountain method. IEEE Transactions on Systems, Man, and Cybernetics, 24(8), 1279–1284.
Yair, E., Zeger, K., & Gersho, A. (1992). Competitive learning and soft competition for vector quantizer design. IEEE Transactions on Signal Processing, 40(2), 294–309.
Yang, M. S. (1993). On a class of fuzzy classification maximum likelihood procedures. Fuzzy Sets and Systems, 57, 365–375.
Yang, M.-S., Wu, K.-L., Hsieh, J.-N., & Yu, J. (2008). Alpha-cut implemented fuzzy clustering algorithms and switching regressions. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 38(3), 588–603.
Yin, H., & Allinson, N. M. (1995). On the distribution and convergence of the feature space in self-organizing maps. Neural Computation, 7(6), 1178–1187.
Yu, J., Cheng, Q., & Huang, H. (2004). Analysis of the weighting exponent in the FCM. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 34(1), 634–639.
Yu, J., & Yang, M. S. (2005). Optimality test for generalized FCM and its application to parameter selection. IEEE Transactions on Fuzzy Systems, 13(1), 164–176.
Zhao, W.-L., Deng, C.-H., & Ngo, C.-W. (2018). \(k\)-means: A revisit. Neurocomputing, 291, 195–206.
Zheng, H., Lefebvre, G., & Laurent, C. (2008). Fast-learning adaptive-subspace self-organizing map: An application to saliency-based invariant image feature construction. IEEE Transactions on Neural Networks, 19(5), 746–757.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2019 Springer-Verlag London Ltd., part of Springer Nature
About this chapter
Cite this chapter
Du, KL., Swamy, M.N.S. (2019). Clustering I: Basic Clustering Models and Algorithms. In: Neural Networks and Statistical Learning. Springer, London. https://doi.org/10.1007/978-1-4471-7452-3_9
Download citation
DOI: https://doi.org/10.1007/978-1-4471-7452-3_9
Published:
Publisher Name: Springer, London
Print ISBN: 978-1-4471-7451-6
Online ISBN: 978-1-4471-7452-3
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)