Mining Associations Between Two Categories Using Unstructured Text Data in Cloud
Finding associations between itemsets within two categories (e.g., drugs and adverse effects, genes and diseases) are very important in many domains. However, these association mining tasks often involve computation-intensive algorithms and a large amount of data. This paper investigates how to leverage MapReduce to effectively mine the associations between itemsets within two categories using a large set of unstructured data. While existing MapReduce-based association mining algorithms focus on frequent itemset mining (i.e., finding itemsets whose frequencies are higher than a threshold), we proposed a MapReduce algorithm that could be used to compute all the interestingness measures defined on the basis of a 2 × 2 contingency table. The algorithm was applied to mine the associations between drugs and diseases using 33,959 full-text biomedical articles on the Amazon Elastic MapReduce (EMR) platform. Experiment results indicate that the proposed algorithm exhibits linear scalability.
KeywordsAssociation mining MapReduce Cloud computing
- 1.L. Geng, H.J. Hamilton, Interestingness measures for data mining: a survey, ACM. Comput. Surv. 38, (2006)Google Scholar
- 2.P.-N. Tan, M. Steinbach, V. Kumar, Introduction to Data Mining, 1st edn. (Addison Wesley, Boston, 2005)Google Scholar
- 3.R. Agrawal, R. Srikant, Fast algorithms for mining association rules. Presented at the proceedings of the 20th international conference on very large databases, Santiago, Chile, 1994Google Scholar
- 5.N.C.f.B. Information, PubMed, 2017. http://www.ncbi.nlm.nih.gov/pubmed.
- 7.F. Kovács, J. Illés, Frequent itemset mining on hadoop, in 2013 IEEE 9th International Conference on Computational Cybernetics (ICCC), 2013, pp. 241–245Google Scholar
- 8.X.Y. Yang, Z. Liu, Y. Fu, MapReduce as a programming model for association rules algorithm on Hadoop, in The 3rd International Conference on Information Sciences and Interaction Sciences, 2010, pp. 99–102Google Scholar
- 9.K. Chavan, P. Kulkarni, P. Ghodekar, S.N. Patil, Frequent itemset mining for Big data, in 2015 International Conference On Green Computing and Internet of Things (ICGCIoT), 2015, pp. 1365–1368Google Scholar
- 10.N. Li, L. Zeng, Q. He, Z. Shi, Parallel implementation of apriori algorithm based on MapReduce, in 2012 13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, 2012, pp. 236–241Google Scholar
- 13.Y. Ji, Y. Tian, F. Shen, J. Tran, High-performance biomedical association mining with MapReduce, in 2015 12th International Conference on Information Technology—New Generations, 2015, pp. 465–470Google Scholar
- 14.T.R. Conference, TREC 2006 Genomics Track. http://skynet.ohsu.edu/trec-gen/.