An Efficient MapReduce-Based Apriori-Like Algorithm for Mining Frequent Itemsets from Big Data
Data mining can discover valuable information from large amounts of data so as to utilize this information to enhance personal or organizational competitiveness. Apriori is a classic algorithm for mining frequent itemsets. Recently, with rapid growth of the Internet as well as fast development of information and communications technology, the amount of data is augmented in an explosive fashion at a speed of tens of petabytes per day. These rapidly expensive data are characterized by huge amount, high speed, continuous arrival, real-time, and unpredictability. Traditional data mining algorithms are not applicable. Therefore, big data mining has become an important research issue.
Clouding computing is a key technique for big data. In this paper, we study the issue of applying cloud computing to mining frequent itemsets from big data. We propose a MapReduce-based Apriori-like frequent itemset mining algorithm called Apriori-MapReduce (abbreviated as AMR). The salient feature of AMR is that it deletes the items of itemsets lower than the minimum support from the transaction database. In such a way, it can greatly reduce the generation of candidate itemsets to avoid a memory shortage and an overload to I/O and CPU, so that a better mining efficiency can be achieved. Empirical studies show that the processing efficiency of the AMR algorithm is superior to that of another efficient MapReduce-based Apriori algorithm under various minimum supports and numbers of transactions.
KeywordsData mining Frequent itemsets Big data MapReduce Apriori
- 1.Agarwal R., Srikant, R.: Fast algorithms for mining association rules in large database. In: Proceedings of the 20th International Conference on Very Large Data Bases, pp. 487–499, Santiago de Chile (1994)Google Scholar
- 2.Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In: Proceedings of the 21st ACM SIGMOD-SIGACT-AIGART Symposium on Principles of Database Systems, pp. 1–16, Madison, WI, June 2002Google Scholar
- 3.Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., Byers, A.H.: Big data: the next frontier for innovation, competition, and productivity. McKinsey Global Institute, New York (2011)Google Scholar
- 4.Turner, V., Gantz, J.F., Reinsel, D., Minton, S.: The digital universe of opportunities: rich data and the increasing value of the internet of things. In: International Data Corporation, White Paper, IDC_1672, May 2014Google Scholar
- 5.Gaber, M.M., Zaslavsky, A., Krishnaswamy, S.: Towards an adaptive approach for mining data streams in resource constrained environments. In: Proceedings of the 2004 International Conference on Data Warehousing and Knowledge Discovery, pp. 189–198, Zaragoza, Spain, September 2004Google Scholar
- 8.Wang, F., Ercegovac, V., Syeda-Mahmood, T., et al.: Large-scale multimodal mining for healthcare with MapReduce. In: Proceedings of the 1st ACM International Health Informatics Symposium, pp. 479–483, New York, November 2010Google Scholar
- 9.Lin, R.C.H., Liao, H.J., Tung, K.Y., Lin, Y.C., Wu, S.L.: Network traffic analysis with cloud platform. J. Internet Technol. 13(6), 953–961 (2012)Google Scholar