A Data-Clustering Algorithm on Distributed Memory Multiprocessors

  • Inderjit S. Dhillon
  • Dharmendra S. Modha
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1759)


To cluster increasingly massive data sets that are common today in data and text mining, we propose a parallel implementation of the k-means clustering algorithm based on the message passing model. The proposed algorithm exploits the inherent data-parallelism in the k-means algorithm. We analytically show that the speedup and the scaleup of our algorithm approach the optimal as the number of data points increases. We implemented our algorithm on an IBM POWERparallel SP2 with a maximum of 16 nodes. On typical test data sets, we observe nearly linear relative speedups, for example, 15.62 on 16 nodes, and essentially linear scaleup in the size of the data set and in the number of clusters desired. For a 2 gigabyte test data set, our implementation drives the 16 node SP2 at more than 1.8 gigaflops.


Execution Time Association Rule Parallelization Strategy Relative Speedup Proposed Parallel Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agrawal, R., Shafer, J.C.: Parallel mining of association rules: Design, implementation, and experience. IEEE Trans. Knowledge and Data Eng. 8 (1996) 962–969CrossRefGoogle Scholar
  2. 2.
    Chattratichat, J., Darlington, J., Ghanem, M., Guo, Y., Hüning, H., Köhler, M., Sutiwaraphun, J., To, H.W., Yang, D.: Large scale data mining: Challenges and responses. In Pregibon, D., Uthurusamy, R., eds.: Proceedings Third International Conference on Knowledge Discovery and Data Mining, Newport Beach, CA, AAAI Press (1997) 61–64Google Scholar
  3. 3.
    Cheung, D.W., Xiao, Y.: Effect of data distribution in parallel mining of associations. Data Mining and Knowledge Discovery (1999) to appear.Google Scholar
  4. 4.
    Han, E.H., Karypis, G., Kumar, V.: Scalable parallel data mining for association rules. In: SIGMOD Record: Proceedings of the 1997 ACM-SIGMOD Conference on Management of Data, Tucson, AZ, USA. (1997) 277–288Google Scholar
  5. 5.
    Joshi, M.V., Karypis, G., Kumar, V.: ScalParC: A new scalable and efficient parallel classification algorithm for mining large datasets. In: Proceedings of the First Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Processing, Orlando, FL, USA. (1998) 573–579Google Scholar
  6. 6.
    Kargupta, H., Hamzaoglu, I., Stafford, B., Hanagandi, V., Buescher, K.: PADMA: Parallel data mining agents for scalable text classification. In: Proceedings of the High Performance Computing, Atlanta, GA, USA. (1997) 290–295Google Scholar
  7. 7.
    Shafer, J., Agrawal, R., Mehta, M.: A scalable parallel classifier for data mining. In: Proc. 22nd International Conference on VLDB, Mumbai, India. (1996)Google Scholar
  8. 8.
    Srivastava, A., Han, E.H., Kumar, V., Singh, V.: Parallel formulations of decision-tree classification algorithms. In: Proc. 1998 International Conference on Parallel Processing. (1998)Google Scholar
  9. 9.
    Zaki, M.J., Ho, C.T., Agrawal, R.: Parallel classification for data mining on shared-memory multiprocessors. In: 15th IEEE Intl. Conf. on Data Engineering. (1999)Google Scholar
  10. 10.
    Zaki, M.J., Parthasarathy, S., Ogihara, M., Li, W.: New parallel algorithms for fast discovery of association rule. Data Mining and Knowledge Discovery 1 (1997) 343–373CrossRefGoogle Scholar
  11. 11.
    Stolorz, P., Musick, R.: Scalable High Performance Computing for Knowledge Discovery and Data Mining. Kluwer Academic Publishers (1997)Google Scholar
  12. 12.
    Freitas, A.A., Lavington, S.H.: Mining Very Large Databases with Parallel Processing. Kluwer Academic Publishers (1998)Google Scholar
  13. 13.
    Hartigan, J.A.: Clustering Algorithms. Wiley (1975)Google Scholar
  14. 14.
    Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., Uthurusamy, R.: Advances in Knowledge Discovery and Data Mining. AAAI/MIT Press (1996)Google Scholar
  15. 15.
    Fukunaga, K., Narendra, P.M.: A branch and bound algorithm for computing k-nearest neighbors. IEEE Trans. Comput. (1975) 750–753Google Scholar
  16. 16.
    Cheeseman, P., Stutz, J.: Bayesian classification (autoclass): Theory and results. In Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., Uthurusamy, R., eds.: Advances in Knowledge Discovery and Data Mining, AAAI/MIT Press (1996) 153–180Google Scholar
  17. 17.
    Smyth, P., Ghil, M., Ide, K., Roden, J., Fraser, A.: Detecting atmospheric regimes using cross-validated clustering. In Pregibon, D., Uthurusamy, R., eds.: Proceedings Third International Conference on Knowledge Discovery and Data Mining, Newport Beach, CA, AAAI Press (1997) 61–64Google Scholar
  18. 18.
    Gersho, A., Gray, R.M.: Vector quantization and signal compression. Kluwer Academic Publishers (1992)Google Scholar
  19. 19.
    Shaw, C.T., King, G.P.: Using cluster analysis to classify time series. Physica D 58 (1992) 288–298CrossRefGoogle Scholar
  20. 20.
    Dhillon, I.S., Modha, D.S., Spangler, W.S.: Visualizing class structure of multidimensional data. In Weisberg, S., ed.: Proceedings of the 30th Symposium on the Interface: Computing Science and Statistics, Minneapolis, MN. (1998)Google Scholar
  21. 21.
    Dhillon, I.S., Modha, D.S., Spangler, W.S.: Visualizing class structure of high-dimensional data with applications. Submitted for publication (1999)Google Scholar
  22. 22.
    Broder, A.Z., Glassman, S.C., Manasse, M.S., Zweig, G.: Syntactic clustering of the web. Technical Report 1997-015, Digital Systems Research Center (1997)Google Scholar
  23. 23.
    Rasmussen, E.: Clustering algorithms. In Frakes, W.B., Baeza-Yates, R., eds.: Information Retrieval: Data Structures and Algorithms, Prentice Hall, Englewood Cliffs, New Jersey (1992) 419–442Google Scholar
  24. 24.
    Willet, P.: Recent trends in hierarchic document clustering: a critical review. Inform. Proc. & Management (1988) 577–597Google Scholar
  25. 25.
    Boley, D., Gini, M., Gross, R., Han, E.H., Hastings, K., Karypis, G., Kumar, V., Mobasher, B., Moore, J.: Document categorization and query generation on the World Wide Web using WebACE. AI Review (1998)Google Scholar
  26. 26.
    Cutting, D.R., Karger, D.R., Pedersen, J.O., Tukey, J.W.: Scatter/gather: A cluster-based approach to browsing large document collections. In: ACM SIGIR. (1992)Google Scholar
  27. 27.
    Sahami, M., Yusufali, S., Baldonado, M.: SONIA: A service for organizing net-worked information autonomously. In: ACM Digital Libraries. (1999)Google Scholar
  28. 28.
    Silverstein, C., Pedersen, J.O.: Almost-constant-time clustering of arbitrary corpus subsets. In: ACM SIGIR. (1997)Google Scholar
  29. 29.
    Zamir, O., Etzioni, O.: Web document clustering: A feasibility demonstration. In: ACM SIGIR. (1998)Google Scholar
  30. 30.
    Dhillon, I.S., Modha, D.S.: Concept decompositions for large sparse text data using clustering. Technical Report RJ 10147 (95022), IBM Almaden Research Center (July 8, 1999)Google Scholar
  31. 31.
    Duda, R.O., Hart, P.E.: Pattern Classification and Scene Analysis. Wiley (1973)Google Scholar
  32. 32.
    Gropp, W., Lusk, E., Skjellum, A.: Using MPI: Portable Parallel Programming with the Message Passing Interface. The MIT Press, Cambridge, MA (1996)Google Scholar
  33. 33.
    Snir, M., Otto, S.W., Huss-Lederman, S., Walker, D.W., Dongarra, J.: MPI: The Complete Reference. The MIT Press, Cambridge, MA (1997)Google Scholar
  34. 34.
    Garey, M.R., Johnson, D.S., Witsenhausen, H.S.: The complexity of the generalized Lloyd-Max problem. IEEE Trans. Inform. Theory 28 (1982) 255–256zbMATHCrossRefMathSciNetGoogle Scholar
  35. 35.
    SAS Institute Cary, NC, USA: SAS Manual. (1997)Google Scholar
  36. 36.
    Zhang, T., Ramakrishnan, R., Livny, M.: Birch: An efficient data clustering method for very large databases. In: Proceedings of the ACM SIGMOD Conference on Management of Data, Montreal, Canada. (1996)Google Scholar
  37. 37.
    Bottou, L., Bengio, Y.: Convergence properties of the k-means algorithms. In Tesauro, G., Touretzky, D., eds.: Advances in Neural Information Processing Systems 7, The MIT Press, Cambridge, MA (1995) 585–592Google Scholar
  38. 38.
    Culler, D.E., Karp, R.M., Patterson, D., Sahay, A., Santos, E.E., Schauser, K.E., Subramonian, R., von Eicken, T.: LogP: A practical model of parallel computation. Communications of the ACM 39 (1996) 78–85CrossRefGoogle Scholar
  39. 39.
    Snir, M., Hochschild, P., Frye, D.D., Gildea, K.J.: The communication software and parallel environment of the IBM SP2. IBM Systems Journal 34 (1995) 205–221CrossRefGoogle Scholar
  40. 40.
    Milligan, G.: An algorithm for creating artificial test clusters. Psychometrika 50 (1985) 123–127CrossRefGoogle Scholar
  41. 41.
    Dubin, D.: clusgen.c. (1996)
  42. 42.
    Northrup, C.J.: Programming with UNIX Threads. John Wiley & Sons (1996)Google Scholar
  43. 43.
    McLachlan, G.J., Krishnan, T.: The EM Algorithm and Extentions. Wiley (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Inderjit S. Dhillon
    • 1
  • Dharmendra S. Modha
    • 2
  1. 1.Department of Computer ScienceUniversity of TexasAustinUSA
  2. 2.IBM Almaden Research CenterSan JoseUSA

Personalised recommendations