Advertisement

Parallel Mining of Top-k High Utility Itemsets in Spark In-Memory Computing Architecture

  • Chun-Han Lin
  • Cheng-Wei Wu
  • JianTao Huang
  • Vincent S. TsengEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11440)

Abstract

Top-k high utility itemset (abbr. Top-k HUI) mining aims at efficiently mining k itemsets having the highest utility without setting the minimum utility thresholds. Although some studies have been conducted on top-k HUI mining recently, they mainly focus on centralized databases and are not scalable for big data environments. To address the above issues, this paper proposes a novel framework for parallel mining of top-k high utility itemsets in big data. Besides, a new algorithm called PKU (Parallel Top-K High Utility Itemset Mining) is proposed for parallel mining of top-k HUIs on Spark in-memory platform. It adopts MapReduce architecture to divide the whole mining task into several independent subtasks, and takes good use of Spark in-memory computing technology for efficiently processing data in parallel. Moreover, several novel strategies are also proposed for pruning the redundant candidates such that the execution time and memory usage in the mining process are reduced greatly. The proposed PKU algorithm inherits several advantages of Spark, including low communication cost, fault tolerance, and high scalability. Experimental results on both real and synthetic datasets show that PKU has good scalability and performance on large datasets with outperforming several benchmarking algorithms.

Keywords

Top-k high utility itemset In-memory computing Big data MapReduce Spark platform 

Notes

Acknowledgement

This work is supported in part by Ministry of Science and Technology, Taiwan, ROC under grant no. 104-2221-E-009-128-MY3, 107-2218-E-009-050 and 107-2218-E-197-002.

References

  1. 1.
    Ahmed, C.F., Tanbeer, S.K., Jeong, B., Lee, Y.: Efficient tree structures for high utility pattern mining in incremental databases. IEEE Trans. Knowl. Data Eng. 21, 1708–1721 (2009)CrossRefGoogle Scholar
  2. 2.
    Fournier-Viger, P., Gomariz, A., Gueniche, T., Soltani, A., Wu, C., Tseng, V.S.: SPMF: a java open-source pattern mining library. J. Mach. Learn. Res. 15, 3389–3393 (2014)zbMATHGoogle Scholar
  3. 3.
    Liu, Y., Liao, W., Choudhary, A.: A fast high utility itemsets mining algorithm. In: Proceedings of the 1st International Workshop on Utility-Based Data Mining, pp. 90–99 (2005)Google Scholar
  4. 4.
    Liu, M., Qu, J.: Mining high utility itemsets without candidate generation. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, pp. 55–64 (2012)Google Scholar
  5. 5.
    Lin, Y., Wu, C., Tseng, V.S.: Mining high utility itemsets in big data. In: Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 649–661 (2015)Google Scholar
  6. 6.
    Ryang, H., Yun, U.: Top-k high utility pattern mining with effective threshold raising strategies. Knowl.-Based Syst. 76, 109–126 (2015)CrossRefGoogle Scholar
  7. 7.
    Subramanian, K., Kandhasamy, P., Subramanian, S.: A novel approach to extract high utility itemsets from distributed databases. Comput. Inform. 31, 1597–1615 (2012)Google Scholar
  8. 8.
    Tseng, V.S., Shie, B., Wu, C., Yu, P.S.: Efficient algorithms for mining high utility itemsets from transactional databases. IEEE Trans. Knowl. Data Eng. 25, 1772–1786 (2013)CrossRefGoogle Scholar
  9. 9.
    Tseng, V.S., Wu, C., Fournier-Viger, P., Yu, P.S.: Efficient algorithms for mining top-k high utility itemsets. IEEE Trans. Knowl. Data Eng. 28, 54–67 (2016)CrossRefGoogle Scholar
  10. 10.
    Vo, B., Nguyen, H., Ho, T.B., Le, B.: Parallel method for mining high utility itemsets from vertically partitioned distributed databases. In: Proceedings of the 13th International Conference on Knowledge-Based and Intelligent Information and Engineering Systems, pp. 251–260 (2009)Google Scholar
  11. 11.
    Wu, C., Shie, B., Tseng, V.S., Yu, P.S.: Mining top-k high utility itemsets. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 78–86 (2012)Google Scholar
  12. 12.
    Apache Software Foundation. http://www.apache.org/
  13. 13.
  14. 14.
    IBM Quest Data Mining Project, Quest Synthetic Data Generation Code. (https://sourceforge.net/projects/ibmquestdatagen/)
  15. 15.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Chun-Han Lin
    • 1
  • Cheng-Wei Wu
    • 2
  • JianTao Huang
    • 2
  • Vincent S. Tseng
    • 3
    Email author
  1. 1.Department of Computer Science and Information EngineeringNational Cheng Kung UniversityTainanTaiwan, ROC
  2. 2.Department of Computer Science and Information EngineeringNational Ilan UniversityYilanTaiwan, ROC
  3. 3.Department of Computer ScienceNational Chiao Tung UniversityHsinchuTaiwan, ROC

Personalised recommendations