A New Graph-Partitioning Algorithm for Large-Scale Knowledge Graph

  • Jiang ZhongEmail author
  • Chen Wang
  • Qi Li
  • Qing Li
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11323)


Large-scale knowledge graph is finding widely practical applications in many fields such as information retrieval, question answering, health care, and knowledge management and so on. To carry out computations on such large-scale knowledge graphs with millions of entities and facts, partitioning of the graphs is necessary. However, the existing partitioning algorithms are difficult to meet the requirements on both partition efficiency and partition quality at the same time. In this paper, we utilize the community-based characteristic that real-world graphs are mostly power-law distribution, and propose a new graph-partitioning algorithm (called MCS) based on message cluster and streaming partitioning. Compared with the traditional algorithms, MCS is closer to or even surpasses Metis package in the partition quality. In the partition efficiency, we use the PageRank algorithm in the spark cluster system to compute the Twitter graph data, and the total time of MCS is lower than that of Hash partitioning. With an increasing number of iterations, the effect is more obvious, which proves the effectiveness of MCS.


Large-scale knowledge graph Graph partitioning Streaming partitioning Community detection Parallel computing 


  1. 1.
    Wei, Y., Luo, J., Xie, H.: KGRL: an OWL2 RL reasoning system for large scale knowledge graph. In: 12th International Conference on Semantics, Knowledge and Grids, pp. 83–89. IEEE, Piscataway (2017)Google Scholar
  2. 2.
    Chen, J., Chen, Y., Du, X., et al.: SEED: a system for entity exploration and debugging in large-scale knowledge graphs. In: 32nd IEEE, International Conference on Data Engineering, pp. 1350–1353. IEEE, Piscataway (2016)Google Scholar
  3. 3.
    Passant, A.: dbrec—Music Recommendations Using DBpedia. The Semantic Web. Springer, Heidelberg (2010). Scholar
  4. 4.
    Tan, Z., Zhao, X., Fang, Y., et al.: GTrans: generic knowledge graph embedding via multi-state entities and dynamic relation spaces. IEEE Access 6(99), 8232–8244 (2018)CrossRefGoogle Scholar
  5. 5.
    Dong, X., Gabrilovich, E., Heitz, G., et al.: Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 601–610. ACM, New York (2014)Google Scholar
  6. 6.
    Auer, S., Bizer, C., Kobilarov, G., et al.: Dbpedia: A Nucleus for a Web of Open Data. The Semantic Web. Springer, Heidelberg (2007). Scholar
  7. 7.
    Hoffart, J., Suchanek, F.M., Berberich, K., et al.: YAGO2: A spatially and temporally enhanced knowledge base from Wikipedia. Artif. Intell. 1(194), 28–61 (2013)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Zaharia, M., Chowdhury, M., Franklin, M.J., et al.: Spark: cluster computing with working sets. HotCloud 10(95), 10 (2010)Google Scholar
  9. 9.
    Malewicz, G., Austern, M.H., Bik, A.J., et al.: Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pp. 135–146. ACM, New York (2010)Google Scholar
  10. 10.
    Low, Y., Gonzalez, J.E., Kyrola, A., et al.: Graphlab: a new framework for parallel machine learning. arXiv preprint arXiv,1408-2041 (2014)Google Scholar
  11. 11.
    Karypis, G., Kumar, V.: A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J. Sci. Comput. 1(20), 359–392 (1998)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Dutt, S.: New faster kernighan-lin-type graph-partitioning algorithms. In: Proceedings of 1993 International Conference on Computer Aided Design (ICCAD), pp. 370–377. IEEE, Piscataway (1993)Google Scholar
  13. 13.
    Battaglino, C., Pienta, P., Vuduc, R.: GraSP: distributed streaming graph partitioning. In: Proceeding of first High performance Graph Mining Workshop, Sydney (2015)Google Scholar
  14. 14.
    Nishimura, J., Ugander, J.: Restreaming graph partitioning: simple versatile algorithms for advanced balancing. In: Proceedings of 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1106–1114. ACM, New York (2013)Google Scholar
  15. 15.
    Stanton, I., Kliot, G.: Streaming graph partitioning for large distributed graphs. In: Proceedings of 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1222–1230. ACM, New York (2012)Google Scholar
  16. 16.
    Chen, L., Li, X., Sheng, Q.Z., et al.: Mining health examination records—a graph-based approach. IEEE Trans. Knowl. Data Eng. 9(28), 2423–2437 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.College of Computer ScienceChongqing UniversityChongqingChina

Personalised recommendations