A low-latency computing framework for time-evolving graphs

  • Shuo Ji
  • Yinliang ZhaoEmail author
  • Xiaomei Zhao


The demand to deliver fast responses in processing time-evolving graphs is higher than ever before in a large number of big data applications. This problem promotes extensive uses of an incremental computing model, which executes the underlying graph algorithm on the newly updated graph structure by taking the results of the computation on the outdated graph structure as initial values, in distributed time-evolving graph computing systems. In this paper, we experimentally study how the initial values of the computation on a newly updated graph structure influence the convergence of the iterative graph analysis, and we develop an optimization framework on the basis of the incremental computing model to accelerate the convergence of processing time-evolving graphs thus achieving high performance for time-evolving graph analysis. In contrast to the traditional incremental computing model, which uses the results of the computation on the outdated graph structure directly, the proposed framework predicts the optimal initial values of the computation on the new graph structure and thereby reduces the number of iterations. Two different prediction approaches are designed to optimize the initial values based on a combination of the results of the computation on the previous graph data and the newly incoming graph data. We have evaluated our optimization framework using the graph algorithms PageRank and KMeans on Amazon EC2 clusters. The experiments demonstrate that the incremental computing implementation with the initial value prediction have reduced the number of iterations by 30% for the PageRank algorithm and 13.7% for the KMeans algorithm, and reduced the response time by 12.7% and 10.6% accordingly compared to the traditional incremental computing model.


Distributed computing Time-evolving graph computing Incremental computing 



We thank our colleagues for their collaboration. We also thank all the reviewers for their specific comments and suggestions. This work is supported by National Natural Science Foundation of China through Grants No. 61640219.


  1. 1.
    Amazon (2018) Amazon ec2.
  2. 2.
    Apache (2012) Apache giraph.
  3. 3.
    Broder AZ, Lempel R, Maghoul F, Pedersen JO (2004) Efficient pagerank approximation via graph aggregation. In: International World Wide Web Conferences, pp 484–485Google Scholar
  4. 4.
    Cai Z, Logothetis D, Siganos G (2012) Facilitating real-time graph mining. In: International Workshop on Cloud Data Management, ACM, pp 1–8Google Scholar
  5. 5.
    Cheng R, Hong J, Kyrola A, Miao Y, Weng X, Wu M, Yang F, Zhou L, Zhao F, Chen E (2012) Kineograph: taking the pulse of a fast-changing and connected world. In: European Conference on Computer Systems, ACM, pp 85–98Google Scholar
  6. 6.
    Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113CrossRefGoogle Scholar
  7. 7.
    Gaito S, Zignani M, Rossi GP, Sala A, Zhao X, Zheng H, Zhao BY (2012) On the bursty evolution of online social networks. In: Proceedings of the First ACM International Workshop on Hot Topics on Interdisciplinary Social Networks Research, pp 1–8Google Scholar
  8. 8.
    Gonzalez JE, Low Y, Gu H, Bickson D, Guestrin C (2012) Powergraph: distributed graph-parallel computation on natural graphs. In: Operating Systems Design and Implementation, pp 17–30Google Scholar
  9. 9.
    Gonzalez JE, Xin RS, Dave A, Crankshaw D, Franklin MJ, Stoica I (2014) Graphx: graph processing in a distributed dataflow framework. In: Operating Systems Design and Implementation, pp 599–613Google Scholar
  10. 10.
    Han W, Miao Y, Li K, Wu M, Yang F, Zhou L, Prabhakaran V, Chen W, Chen E (2014) Chronos: a graph engine for temporal graph analysis. In: European Conference on Computer Systems, pp 1–14Google Scholar
  11. 11.
    Iyer AP, Li LE, Das T, Stoica I (2016) Time-evolving graph processing at scale. In: International Workshop on Graph Data Management Experiences and Systems. ACM, pp 1–6Google Scholar
  12. 12.
    Ji S, Zhao Y (2018) A local approximation approach for processing time-evolving graphs. Symmetry 10(7):247CrossRefGoogle Scholar
  13. 13.
    Ju W, Li J, Yu W, Zhang R (2016) iGraph: an incremental data processing system for dynamic graph. Front Comput Sci 10(3):462–476CrossRefGoogle Scholar
  14. 14.
    Kan M, Thi HON (2005) Fast webpage classification using URL features. In: Conference on Information and Knowledge Management, pp 325–326Google Scholar
  15. 15.
    Konect (2017) Konect network dataset.
  16. 16.
    Malewicz G, Austern MH, Bik AJ, Dehnert JC, Horn I, Leiser N, Czajkowski G (2010) Pregel: a system for large-scale graph processing. In: ACM SIGMOD International Conference on Management of Data. ACM, pp 135–146Google Scholar
  17. 17.
    McCune RR, Weninger T, Madey G (2015) Thinking like a vertex: a survey of vertex-centric frameworks for large-scale distributed graph processing. ACM Comput Surv 48(2):25CrossRefGoogle Scholar
  18. 18.
    Morshed SJ, Rana J, Milrad M (2016) Real-time data analytics: an algorithmic perspective. In: International Conference on Data Mining, pp 311–320Google Scholar
  19. 19.
    Murray DG, Mcsherry F, Isaacs R, Isard M, Barham P, Abadi M (2013) Naiad: a timely dataflow system. In: Symposium on Operating Systems Principles, pp 439–455Google Scholar
  20. 20.
    Sha M, Li Y, He B, Tan KL (2017) Accelerating dynamic graph analytics on gpus. Very Large Data Bases 11(1):107–120Google Scholar
  21. 21.
    Shi X, Cui B, Shao Y, Tong Y (2016) Tornado: a system for real-time iterative analysis over evolving data. In: International Conference on Management of Data, pp 417–430Google Scholar
  22. 22.
  23. 23.
    Valiant LG (1990) A bridging model for parallel computation. Commun ACM 33(8):103–111CrossRefGoogle Scholar
  24. 24.
    Vaquero LM, Cuadrado F, Logothetis D, Martella C (2013) xDGP: a dynamic graph processing system with adaptive partitioning. arXiv preprint arXiv:1309.1049
  25. 25.
    Vaquero LM, Cuadrado F, Ripeanu M (2014) Systems for near real-time analysis of large-scale dynamic graphs. arXiv preprint arXiv:1410.1903
  26. 26.
    Vazirgiannis M, Drosos D, Senellart P, Vlachou A (2008) Web page rank prediction with Markov models. In: International World Wide Web Conferences, pp 1075–1076Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer Science and TechnologyXi’an Jiaotong UniversityXi’anPeople’s Republic of China
  2. 2.School of Software EngineeringXi’an Jiaotong UniversityXi’anPeople’s Republic of China

Personalised recommendations