Abstract
Pregel is a popular distributed computing model for dealing with large-scale graphs. However, it can be tricky to implement graph algorithms correctly and efficiently in Pregel’s vertex-centric model, especially when the algorithm has multiple computation stages, complicated data dependencies, or even communication over dynamic internal data structures. Some domain-specific languages (DSLs) have been proposed to provide more intuitive ways to implement graph algorithms, but due to the lack of support for remote access — reading or writing attributes of other vertices through references — they cannot handle the above mentioned dynamic communication, causing a class of Pregel algorithms with fast convergence impossible to implement.
To address this problem, we design and implement Palgol, a more declarative and powerful DSL which supports remote access. In particular, programmers can use a more declarative syntax called chain access to naturally specify dynamic communication as if directly reading data on arbitrary remote vertices. By analyzing the logic patterns of chain access, we provide a novel algorithm for compiling Palgol programs to efficient Pregel code. We demonstrate the power of Palgol by using it to implement several practical Pregel algorithms, and the evaluation result shows that the efficiency of Palgol is comparable with that of hand-written code.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Palgol stands for Pregel algorithmic language. The system with all implementation code and test examples is available at https://bitbucket.org/zyz915/palgol.
- 2.
- 3.
BCC is a complicated algorithm, whose efficient implementation requires constructing an intermediate graph, which is currently beyond Palgol’s capabilities. Palgol is powerful enough to express the rest of the algorithm, however.
- 4.
- 5.
- 6.
Palgol does not target a specific Pregel-like system. Instead, by properly implementing different backends of the compiler, Palgol can be transformed into any Pregel-like system, as long as the system supports the basic Pregel interfaces including message passing between arbitrary pairs of vertices and aggregators.
- 7.
- 8.
- 9.
References
Chung, S., Condon, A.: Parallel implementation of Borůvka’s minimum spanning tree algorithm. In: IPPS, pp. 302–308. IEEE (1996)
Emoto, K., Matsuzaki, K., Morihata, A., Hu, Z.: Think like a vertex, behave like a function! A functional DSL for vertex-centric big graph processing. In: ICFP, pp. 200–213. ACM (2016)
Fard, A., Nisar, M.U., Ramaswamy, L., Miller, J.A., Saltz, M.: A distributed vertex-centric approach for pattern matching in massive graphs. In: BigData, pp. 403–411. IEEE (2013)
Gabow, H.N., Tarjan, R.E.: A linear-time algorithm for a special case of disjoint set union. J. Comput. System Sci. 30(2), 209–221 (1985)
Halpern, J.Y., Moses, Y.: Knowledge and common knowledge in a distributed environment. J. ACM 37(3), 549–587 (1990)
Hong, S., Chafi, H., Sedlar, E., Olukotun, K.: Green-Marl: a DSL for easy and efficient graph analysis. In: ASPLOS, pp. 349–362. ACM (2012)
Hong, S., Salihoglu, S., Widom, J., Olukotun, K.: Simplifying scalable graph processing with a domain-specific language. In: CGO, p. 208. ACM (2014)
Lesniak, M.: Palovca: describing and executing graph algorithms in Haskell. In: Russo, C., Zhou, N.-F. (eds.) PADL 2012. LNCS, vol. 7149, pp. 153–167. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27694-1_12
Malewicz, G., Austern, M.H., Bik, A.J., Dehnert, J.C., Horn, I., Leiser, N., Czajkowski, G.: Pregel: a system for large-scale graph processing. In: SIGMOD, pp. 135–146. ACM (2010)
Quick, L., Wilkinson, P., Hardcastle, D.: Using Pregel-like large scale graph processing frameworks for social network analysis. In: ASONAM, pp. 457–463. IEEE (2012)
Ruiz, O.C., Matsuzaki, K., Sato, S.: s6raph: vertex-centric graph processing framework with functional interface. In: FHPC, pp. 58–64. ACM (2016)
Salihoglu, S., Widom, J.: Optimizing graph algorithms on Pregel-like systems. PVLDB 7(7), 577–588 (2014)
Tian, Y., Balmin, A., Corsten, S.A., Tatikonda, S., McPherson, J.: From think like a vertex to think like a graph. PVLDB 7(3), 193–204 (2013)
Valiant, L.G.: A bridging model for parallel computation. Commun. ACM 33(8), 103–111 (1990)
Xie, M., Yang, Q., Zhai, J., Wang, Q.: A vertex centric parallel algorithm for linear temporal logic model checking in Pregel. J. Parallel Distrib. Com. 74(11), 3161–3174 (2014)
Yan, D., Cheng, J., Lu, Y., Ng, W.: Blogel: a block-centric framework for distributed computation on real-world graphs. PVLDB 7(14), 1981–1992 (2014)
Yan, D., Cheng, J., Xing, K., Lu, Y., Ng, W., Bu, Y.: Pregel algorithms for graph connectivity problems with performance guarantees. PVLDB 7(14), 1821–1832 (2014)
Acknowledgements
We thank Dr. Kento Emoto for his advice in the design of Palgol, Mr. Smith Dhumbumroong for his help in setting up the experiments, and the reviewers for their insightful comments to improve this paper. This work was supported by JSPS KAKENHI Grant Numbers 26280020 and 17H06099.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Zhang, Y., Ko, HS., Hu, Z. (2017). Palgol: A High-Level DSL for Vertex-Centric Graph Processing with Remote Data Access. In: Chang, BY. (eds) Programming Languages and Systems. APLAS 2017. Lecture Notes in Computer Science(), vol 10695. Springer, Cham. https://doi.org/10.1007/978-3-319-71237-6_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-71237-6_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-71236-9
Online ISBN: 978-3-319-71237-6
eBook Packages: Computer ScienceComputer Science (R0)