Skip to main content
SpringerLink
Log in

Parallel Shortest Path Graph Computations of United States Road Network Data on Apache Spark

  • Conference paper
  • First Online:
Smart Societies, Infrastructure, Technologies and Applications (SCITA 2017)

Abstract

Big data is being generated from various sources such as Internet of Things (IoT) and social media. Big data cannot be processed by traditional tools and technologies due to their properties, volume, velocity, veracity, and variety. Graphs are becoming increasingly popular to model real-world problems; the problems are typically large and, hence, give rise to large graphs, which could be analysed and solved using big data technologies. This paper explores the performance of single source shortest path graph computations using the Apache Spark big data platform. We use the United States road network data, modelled as graphs, and calculate shortest paths between vertices. The experiments are performed on the Aziz supercomputer (a Top500 machine). We solve problems of varying graph sizes, i.e. various states of the US, and analyse Spark’s parallelization behavior. As expected, the speedup is dependent on both the size of the data and the number of parallel nodes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Lu, Y., Cheng, J., Yan, D., Wu, H.: Large-scale distributed graph computing systems. Proc. VLDB Endow. 8, 281–292 (2014)

    Article  Google Scholar 

  2. Sanfeliu, A., Alquézar, R., Andrade, J., Climent, J., Serratosa, F., Vergés, J.: Graph-based representations and techniques for image processing and image analysis. Pattern Recognit. 35, 639–650 (2002)

    Article  Google Scholar 

  3. Ding, Y., Yan, S., Zhang, Y.B., Dai, W., Dong, L.: Predicting the attributes of social network users using a graph-based machine learning method. Comput. Commun. 73, 3–11 (2016)

    Article  Google Scholar 

  4. Khan, A., Uddin, S., Srinivasan, U.: Adapting graph theory and social network measures on healthcare data. In: Proceedings of the Australasian Computer Science Week Multiconference on - ACSW 2016, pp. 1–7. ACM Press, New York (2016)

    Google Scholar 

  5. Mehmood, R., Graham, G.: Big data logistics: a health-care transport capacity sharing model. Procedia Comput. Sci. 64, 1107–1114 (2015)

    Article  Google Scholar 

  6. Mehmood, R., Meriton, R., Graham, G., Hennelly, P., Kumar, M.: Exploring the influence of big data on city transport operations: a Markovian approach. Int. J. Oper. Prod. Manag. (2016). Forthcoming

    Google Scholar 

  7. Hendrickson, B., Kolda, T.G.: Graph partitioning models for parallel computing. Parallel Comput. 26, 1519–1534 (2000)

    Article  MathSciNet  Google Scholar 

  8. Mehmood, R., Crowcroft, J.: Parallel iterative solution method for large sparse linear equation systems. Comput. Lab. Univ. (2005)

    Google Scholar 

  9. Kwiatkowska, M., Parker, D., Zhang, Y., Mehmood, R.: Dual-processor parallelisation of symbolic probabilistic model checking. In: DeGroot, D., Harrison, P. (eds.) Proceedings - IEEE Computer Society’s Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS, pp. 123–130. IEEE, Volendam (2004)

    Google Scholar 

  10. Alazawi, Z., Abdljabar, M.B., Altowaijri, S., Vegni, A.M., Mehmood, R.: ICDMS: an intelligent cloud based disaster management system for vehicular networks. In: Vinel, A., Mehmood, R., Berbineau, M., Garcia, C.R., Huang, C.-M., Chilamkurti, N. (eds.) Nets4Cars/Nets4Trains 2012. LNCS, vol. 7266, pp. 40–56. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29667-3_4

    Chapter  Google Scholar 

  11. Junghanns, M., Petermann, A., Neumann, M., Rahm, E.: Management and analysis of big graph data: current systems and open challenges. In: Zomaya, A., Sakr, S. (eds.) Handbook of Big Data Technologies, pp. 457–505. Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3-319-49340-4_14

    Chapter  Google Scholar 

  12. Oh, S., Ha, J., Lee, K., Oh, S.: DegoViz: an interactive visualization tool for a differentially expressed genes heatmap and gene ontology graph. Appl. Sci. 7, 543 (2017)

    Article  Google Scholar 

  13. Mehmood, R., Faisal, M.A., Altowaijri, S.: Future networked healthcare systems: a review and case study. In: Big Data: Concepts, Methodologies, Tools, and Applications, pp. 2429–2457. IGI Global (2016)

    Google Scholar 

  14. Arfat, Y., Aqib, M., Mehmood, R., Albeshri, A., Katib, I., Albogami, N., Alzahrani, A.: Enabling smarter societies through mobile big data fogs and clouds. Procedia Comput. Sci. 109, 1128–1133 (2017)

    Article  Google Scholar 

  15. GraphX | Apache Spark

    Google Scholar 

  16. Apache Spark: Apache SparkTM - Lightning-Fast Cluster Computing

    Google Scholar 

  17. Welcome to ApacheTM Hadoop®! - index.pdf. https://hadoop.apache.org/index.pdf

  18. Kajdanowicz, T., Kazienko, P., Indyk, W.: Parallel processing of large graphs. Futur. Gener. Comput. Syst. 32, 324–337 (2014)

    Article  Google Scholar 

  19. Liu, X., Zhou, Y., Guan, X., Sun, X.: A feasible graph partition framework for random walks implemented by parallel computing in big graph. In: Chinese Control Conference, CCC 2015, pp. 4986–4991, September 2015

    Google Scholar 

  20. Wang, Z., Chen, Q., Hou, B., Suo, B., Li, Z., Pan, W., Ives, Z.G.: Parallelizing maximal clique and k-plex enumeration over graph data. J. Parallel Distrib. Comput. 106, 79–91 (2017)

    Article  Google Scholar 

  21. Braun, P., Cuzzocrea, A., Leung, C.K., Pazdor, A.G.M., Tran, K.: Knowledge Discovery from Social Graph Data. Procedia Comput. Sci. 96, 682–691 (2016)

    Article  Google Scholar 

  22. Laboshin, L.U., Lukashin, A.A., Zaborovsky, V.S.: The big data approach to collecting and analyzing traffic data in large scale networks. Procedia Comput. Sci. 103, 536–542 (2017)

    Article  Google Scholar 

  23. Liu, R., Li, X., Du, L., Zhi, S., Wei, M.: Parallel implementation of density peaks clustering algorithm based on spark. Procedia Comput. Sci. 107, 442–447 (2017)

    Article  Google Scholar 

  24. Aridhi, S., Mephu Nguifo, E.: Big graph mining: frameworks and techniques. Big Data Res. 6, 1–10 (2016)

    Article  Google Scholar 

  25. Drosou, A., Kalamaras, I., Papadopoulos, S., Tzovaras, D.: An enhanced Graph Analytics Platform (GAP) providing insight in Big Network Data. J. Innov. Digit. Ecosyst. 3, 83–97 (2016)

    Article  Google Scholar 

  26. Zhao, Y., Yoshigoe, K., Xie, M., Zhou, S., Seker, R., Bian, J.: Evaluation and analysis of distributed graph-parallel processing frameworks. J. Cyber Secur. Mobil. 3, 289–316 (2014)

    Article  Google Scholar 

  27. Mohan, A., Remya, R.: A review on large scale graph processing using big data based parallel programming models. Int. J. Intell. Syst. Appl. 9, 49–57 (2017)

    Google Scholar 

  28. Pollard, S., Norris, B.: A Comparison of Parallel Graph Processing Benchmarks (2017)

    Google Scholar 

  29. Suma, S., Mehmood, R., Albugami, N., Katib, I., Albeshri, A.: Enabling next generation logistics and planning for smarter societies. Procedia Comput. Sci. 109, 1122–1127 (2017)

    Article  Google Scholar 

  30. Miller, J.A., Ramaswamy, L., Kochut, K.J., Fard, A.: Research directions for big data graph analytics. In: Proceedings of the 2015 IEEE International Congress on Big Data, BigData Congress 2015, pp. 785–794 (2015)

    Google Scholar 

  31. Chakaravarthy, V.T., Checconi, F., Petrini, F., Sabharwal, Y.: Scalable single source shortest path algorithms for massively parallel systems. In: Proceedings of the 28th International Parallel and Distributed Processing Symposium, IPDPS, pp. 889–901 (2014)

    Google Scholar 

  32. Xia, Y., Tanase, I.G., Nai, L., Tan, W., Liu, Y., Crawford, J., Lin, C.: Explore efficient data organization for large scale graph analytics and storage. In: Proceedings of the 2014 IEEE BigData Conference, pp. 942–951 (2014)

    Google Scholar 

  33. Zhang, B., Liu, X., Lang, B.: Fast Graph Similarity Search via Locality Sensitive Hashing. In: Ho, Y.-S., Sang, J., Ro, Y.M., Kim, J., Wu, F. (eds.) PCM 2015. LNCS, vol. 9314, pp. 623–633. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24075-6_60

    Chapter  Google Scholar 

  34. Shao, Y., Cui, B., Ma, L.: PAGE: A partition aware engine for parallel graph computation. IEEE Trans. Knowl. Data Eng. 27, 518–530 (2015)

    Article  Google Scholar 

  35. Chen, R., Yang, M., Weng, X., Choi, B., He, B., Li, X.: Improving large graph processing on partitioned graphs in the cloud. In: Proceedings of the Third ACM Symposium on Cloud Computing, SoCC 2012, pp. 1–13 (2012)

    Google Scholar 

  36. Zeng, Z., Wu, B., Wang, H.: A parallel graph partitioning algorithm to speed up the large-scale distributed graph mining. In: Proceedings of the 1st International Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications - BigMine 2012, pp. 61–68 (2012)

    Google Scholar 

  37. Lee, K., Liu, L.: Efficient data partitioning model for heterogeneous graphs in the cloud. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, pp. 1–12 (2013)

    Google Scholar 

  38. Xu, N., Chen, L., Cui, B.: LogGP: a log-based dynamic graph partitioning method. Proc. VLDB Endow. 7, 1917–1928 (2014)

    Article  Google Scholar 

  39. Yang, S., Yan, X., Zong, B., Khan, A.: Towards effective partition management for large graphs. In: Proceedings of the 2012 International Conference on Management Data - SIGMOD 2012, pp. 517–528 (2012)

    Google Scholar 

  40. Spark Data Locality Documentation. https://spark.apache.org/docs/latest/tuning.html#data-locality

  41. GraphX Partitioning Scheme Documentation. https://spark.apache.org/docs/latest/graphx-programming-guide.html#optimized-representation

  42. DIMACS Implementation Challenge. http://www.dis.uniroma1.it/challenge9/download.shtml

  43. Gephi - The Open Graph Viz Platform. https://gephi.org/

Download references

Acknowledgments

The authors acknowledge with thanks the technical and financial support from the Deanship of Scientific Research (DSR) at the King Abdulaziz University (KAU), Jeddah, Saudi Arabia, under the grant number G-661-611-38. The experiments reported in this paper were performed on the Aziz supercomputer at King Abdulaziz University.

Author information

Authors and Affiliations

  1. Department of Computer Science, FCIT, King Abdulaziz University, Jeddah, 21589, Saudi Arabia

    Yasir Arfat & Aiiad Albeshri

  2. High Performance Computing Center, King Abdulaziz University, Jeddah, 21589, Saudi Arabia

    Rashid Mehmood

Authors
  1. Yasir Arfat
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rashid Mehmood
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aiiad Albeshri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yasir Arfat .

Editor information

Editors and Affiliations

  1. King Abdulaziz University, Jeddah, Saudi Arabia

    Rashid Mehmood

  2. Geographic Information Science and Technology, Oak Ridge National Laboratory, Oak Ridge, Tennessee, USA

    Budhendra Bhaduri

  3. King Abdulaziz University, Jeddah, Saudi Arabia

    Iyad Katib

  4. European Alliance for Innovation, Bratislava, Slovakia

    Imrich Chlamtac

Rights and permissions

Reprints and permissions

Copyright information

© 2018 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Arfat, Y., Mehmood, R., Albeshri, A. (2018). Parallel Shortest Path Graph Computations of United States Road Network Data on Apache Spark. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 224. Springer, Cham. https://doi.org/10.1007/978-3-319-94180-6_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-94180-6_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-94179-0

  • Online ISBN: 978-3-319-94180-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation