Abstract
Large-scale knowledge graph is finding widely practical applications in many fields such as information retrieval, question answering, health care, and knowledge management and so on. To carry out computations on such large-scale knowledge graphs with millions of entities and facts, partitioning of the graphs is necessary. However, the existing partitioning algorithms are difficult to meet the requirements on both partition efficiency and partition quality at the same time. In this paper, we utilize the community-based characteristic that real-world graphs are mostly power-law distribution, and propose a new graph-partitioning algorithm (called MCS) based on message cluster and streaming partitioning. Compared with the traditional algorithms, MCS is closer to or even surpasses Metis package in the partition quality. In the partition efficiency, we use the PageRank algorithm in the spark cluster system to compute the Twitter graph data, and the total time of MCS is lower than that of Hash partitioning. With an increasing number of iterations, the effect is more obvious, which proves the effectiveness of MCS.
This work gets supports from the National Key Research and Development Program of China under Grant 2017YFB1402401, Technology Innovation and Application Demonstration project of Chongqing under Grant cstc2018jszx-cyzdX0086, Fundamental Research Funds for the Central University under Grant 2018CDYJSY0055 and CERNET Next Generation Internet Technology Innovation Project under Grant NGII20150706.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Wei, Y., Luo, J., Xie, H.: KGRL: an OWL2 RL reasoning system for large scale knowledge graph. In: 12th International Conference on Semantics, Knowledge and Grids, pp. 83–89. IEEE, Piscataway (2017)
Chen, J., Chen, Y., Du, X., et al.: SEED: a system for entity exploration and debugging in large-scale knowledge graphs. In: 32nd IEEE, International Conference on Data Engineering, pp. 1350–1353. IEEE, Piscataway (2016)
Passant, A.: dbrec—Music Recommendations Using DBpedia. The Semantic Web. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-17749-1_14
Tan, Z., Zhao, X., Fang, Y., et al.: GTrans: generic knowledge graph embedding via multi-state entities and dynamic relation spaces. IEEE Access 6(99), 8232–8244 (2018)
Dong, X., Gabrilovich, E., Heitz, G., et al.: Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 601–610. ACM, New York (2014)
Auer, S., Bizer, C., Kobilarov, G., et al.: Dbpedia: A Nucleus for a Web of Open Data. The Semantic Web. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-76298-0_52
Hoffart, J., Suchanek, F.M., Berberich, K., et al.: YAGO2: A spatially and temporally enhanced knowledge base from Wikipedia. Artif. Intell. 1(194), 28–61 (2013)
Zaharia, M., Chowdhury, M., Franklin, M.J., et al.: Spark: cluster computing with working sets. HotCloud 10(95), 10 (2010)
Malewicz, G., Austern, M.H., Bik, A.J., et al.: Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pp. 135–146. ACM, New York (2010)
Low, Y., Gonzalez, J.E., Kyrola, A., et al.: Graphlab: a new framework for parallel machine learning. arXiv preprint arXiv,1408-2041 (2014)
Karypis, G., Kumar, V.: A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J. Sci. Comput. 1(20), 359–392 (1998)
Dutt, S.: New faster kernighan-lin-type graph-partitioning algorithms. In: Proceedings of 1993 International Conference on Computer Aided Design (ICCAD), pp. 370–377. IEEE, Piscataway (1993)
Battaglino, C., Pienta, P., Vuduc, R.: GraSP: distributed streaming graph partitioning. In: Proceeding of first High performance Graph Mining Workshop, Sydney (2015)
Nishimura, J., Ugander, J.: Restreaming graph partitioning: simple versatile algorithms for advanced balancing. In: Proceedings of 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1106–1114. ACM, New York (2013)
Stanton, I., Kliot, G.: Streaming graph partitioning for large distributed graphs. In: Proceedings of 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1222–1230. ACM, New York (2012)
Chen, L., Li, X., Sheng, Q.Z., et al.: Mining health examination records—a graph-based approach. IEEE Trans. Knowl. Data Eng. 9(28), 2423–2437 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhong, J., Wang, C., Li, Q., Li, Q. (2018). A New Graph-Partitioning Algorithm for Large-Scale Knowledge Graph. In: Gan, G., Li, B., Li, X., Wang, S. (eds) Advanced Data Mining and Applications. ADMA 2018. Lecture Notes in Computer Science(), vol 11323. Springer, Cham. https://doi.org/10.1007/978-3-030-05090-0_37
Download citation
DOI: https://doi.org/10.1007/978-3-030-05090-0_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-05089-4
Online ISBN: 978-3-030-05090-0
eBook Packages: Computer ScienceComputer Science (R0)