Parallel Mining of Top-k High Utility Itemsets in Spark In-Memory Computing Architecture
Top-k high utility itemset (abbr. Top-k HUI) mining aims at efficiently mining k itemsets having the highest utility without setting the minimum utility thresholds. Although some studies have been conducted on top-k HUI mining recently, they mainly focus on centralized databases and are not scalable for big data environments. To address the above issues, this paper proposes a novel framework for parallel mining of top-k high utility itemsets in big data. Besides, a new algorithm called PKU (Parallel Top-K High Utility Itemset Mining) is proposed for parallel mining of top-k HUIs on Spark in-memory platform. It adopts MapReduce architecture to divide the whole mining task into several independent subtasks, and takes good use of Spark in-memory computing technology for efficiently processing data in parallel. Moreover, several novel strategies are also proposed for pruning the redundant candidates such that the execution time and memory usage in the mining process are reduced greatly. The proposed PKU algorithm inherits several advantages of Spark, including low communication cost, fault tolerance, and high scalability. Experimental results on both real and synthetic datasets show that PKU has good scalability and performance on large datasets with outperforming several benchmarking algorithms.
KeywordsTop-k high utility itemset In-memory computing Big data MapReduce Spark platform
This work is supported in part by Ministry of Science and Technology, Taiwan, ROC under grant no. 104-2221-E-009-128-MY3, 107-2218-E-009-050 and 107-2218-E-197-002.
- 3.Liu, Y., Liao, W., Choudhary, A.: A fast high utility itemsets mining algorithm. In: Proceedings of the 1st International Workshop on Utility-Based Data Mining, pp. 90–99 (2005)Google Scholar
- 4.Liu, M., Qu, J.: Mining high utility itemsets without candidate generation. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, pp. 55–64 (2012)Google Scholar
- 5.Lin, Y., Wu, C., Tseng, V.S.: Mining high utility itemsets in big data. In: Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 649–661 (2015)Google Scholar
- 7.Subramanian, K., Kandhasamy, P., Subramanian, S.: A novel approach to extract high utility itemsets from distributed databases. Comput. Inform. 31, 1597–1615 (2012)Google Scholar
- 10.Vo, B., Nguyen, H., Ho, T.B., Le, B.: Parallel method for mining high utility itemsets from vertically partitioned distributed databases. In: Proceedings of the 13th International Conference on Knowledge-Based and Intelligent Information and Engineering Systems, pp. 251–260 (2009)Google Scholar
- 11.Wu, C., Shie, B., Tseng, V.S., Yu, P.S.: Mining top-k high utility itemsets. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 78–86 (2012)Google Scholar
- 12.Apache Software Foundation. http://www.apache.org/
- 13.Hadoop. http://hadoop.apache.org/
- 14.IBM Quest Data Mining Project, Quest Synthetic Data Generation Code. (https://sourceforge.net/projects/ibmquestdatagen/)
- 15.Spark. http://spark.apache.org/