Skip to main content
SpringerLink
Log in

Design and Engineering of External Memory Traversal Algorithms for General Graphs

  • Chapter
Algorithmics of Large and Complex Networks

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 5515))

Abstract

Large graphs arise naturally in many real world applications. The actual performance of simple RAM model algorithms for traversing these graphs (stored in external memory) deviates significantly from their linear or near-linear predicted performance because of the large number of I/Os they incur. In order to alleviate the I/O bottleneck, many external memory graph traversal algorithms have been designed with provable worst-case guarantees. We describe the various techniques used in the design and engineering of such algorithms and survey the state-of-the-art in I/O-efficient graph traversal algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abello, J., Buchsbaum, A., Westbrook, J.: A functional approach to external graph algorithms. Algorithmica 32(3), 437–458 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  2. Abello, J., Pardalos, P., Resende, M.: On maximum clique problems in very large graphs. External Memory Algorithms, AMS-DIMACS Series in Discrete Mathematics and Theoretical Computer Science 50, 119–130 (1999)

    MathSciNet  MATH  Google Scholar 

  3. Aggarwal, A., Vitter, J.S.: The input/output complexity of sorting and related problems. Communications of the ACM 31(9), 1116–1127 (1988)

    Article  MathSciNet  Google Scholar 

  4. Ajwani, D., Dementiev, R., Meyer, U.: A computational study of external memory bfs algorithms. In: Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 601–610 (2006)

    Google Scholar 

  5. Ajwani, D., Meyer, U., Osipov, V.: Improved external memory BFS implementation. In: Proceedings of the workshop on Algorithm Engineering and Experiments (ALENEX), pp. 3–12 (2007)

    Google Scholar 

  6. Allulli, L., Lichodzijewski, P., Zeh, N.: A faster cache-oblivious shortest-path algorithm for undirected graphs with bounded edge lengths. In: Proceedings of the 18th annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 910–919 (2007)

    Google Scholar 

  7. Arge, L.: The Buffer Tree: A new technique for optimal I/O-algorithms. In: Sack, J.-R., Akl, S.G., Dehne, F., Santoro, N. (eds.) WADS 1995. LNCS, vol. 955, pp. 334–345. Springer, Heidelberg (1995)

    Chapter  Google Scholar 

  8. Arge, L., Bender, M.A., Demaine, E.D., Holland-Minkley, B., Munro, J.I.: Cache-oblivious priority queue and graph algorithm applications. In: Proceedings of the 34th Annual ACM Symposium on Theory of Computing (STOC), pp. 268–276 (2002)

    Google Scholar 

  9. Arge, L., Brodal, G., Toma, L.: On external-memory MST, SSSP and multi-way planar graph separation. In: Halldórsson, M.M. (ed.) SWAT 2000. LNCS, vol. 1851, pp. 433–447. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  10. Arge, L., Chase, J.S., Halpin, P., Toma, L., Vitter, J.S., Urban, D., Wickremesinghe, R.: Efficient flow computation on massive grid terrain datasets. Geoinformatica 7(4), 283–313 (2003)

    Article  Google Scholar 

  11. Arge, L., Meyer, U., Toma, L.: External memory algorithms for diameter and all-pairs shortest-paths on sparse graphs. In: Díaz, J., Karhumäki, J., Lepistö, A., Sannella, D. (eds.) ICALP 2004. LNCS, vol. 3142, pp. 146–157. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  12. Arge, L., Procopiuc, O., Vitter, J.: Implementing I/O-efficient data structures using TPIE. In: Möhring, R.H., Raman, R. (eds.) ESA 2002. LNCS, vol. 2461, pp. 88–100. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  13. Boruvka, O.: O jistém problému minimálním. In: Práce, Moravské Prirodovedecké Spolecnosti, pp. 1–58 (1926)

    Google Scholar 

  14. Brandes, U., Erlebach, T. (eds.): Network Analysis. LNCS, vol. 3418. Springer, Heidelberg (2005)

    MATH  Google Scholar 

  15. Brodal, G., Fagerberg, R.: Cache oblivious distribution sweeping. In: Widmayer, P., Triguero, F., Morales, R., Hennessy, M., Eidenbenz, S., Conejo, R. (eds.) ICALP 2002. LNCS, vol. 2380, pp. 426–438. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  16. Brodal, G., Fagerberg, R., Meyer, U., Zeh, N.: Cache-oblivious data structures and algorithms for undirected breadth-first search and shortest paths. In: Hagerup, T., Katajainen, J. (eds.) SWAT 2004. LNCS, vol. 3111, pp. 480–492. Springer, Heidelberg (2004)

    Google Scholar 

  17. Brodal, G.S., Fagerberg, R.: Funnel heap - a cache oblivious priority queue. In: Bose, P., Morin, P. (eds.) ISAAC 2002. LNCS, vol. 2518, pp. 219–228. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  18. Brodal, G.S., Fagerberg, R., Vinther, K.: Engineering a cache-oblivious sorting algorithm. In: Proceedings of the sixth workshop on Algorithm Engineering and Experiments (ALENEX), pp. 4–17 (2004)

    Google Scholar 

  19. Buchsbaum, A., Goldwasser, M., Venkatasubramanian, S., Westbrook, J.: On external memory graph traversal. In: 11th Annual Symposium on Discrete Algorithms (SODA), pp. 859–860. ACM-SIAM (2000)

    Google Scholar 

  20. Chan, T.M.: All-pairs shortest paths with real weights in O(n 3/logn) time. Algorithmica 50(2), 236–243 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  21. Chazelle, B.: A minimum spanning tree algorithm with inverse-ackermann type complexity. JACM 47(6), 1028–1047 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  22. Chen, M., Chowdhury, R.A., Ramachandran, V., Roche, D.L., Tong, L.: Priority queues and dijkstra’s algorithm. Technical report TR-07-54, The University of Texas at Austin, Department of Computer Sciences (2007)

    Google Scholar 

  23. Chiang, Y.J., Goodrich, M.T., Grove, E.F., Tamasia, R., Vengroff, D.E., Vitter, J.S.: External memory graph algorithms. In: Proceedings of the 6th annual Symposium on Discrete Algorithms (SODA), pp. 139–149. ACM-SIAM (1995)

    Google Scholar 

  24. Chowdhury, R.A., Ramachandran, V.: External-memory exact and approximate all-pairs shortest-paths in undirected graphs. In: Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 735–744 (2005)

    Google Scholar 

  25. Cormen, T., Leiserson, C., Rivest, R.: Introduction to Algorithms. The MIT Press, Cambridge (1989)

    MATH  Google Scholar 

  26. de Kunder, M.: World wide web size, http://www.worldwidewebsize.com/

  27. Dementiev, R., Kettner, L., Sanders, P.: Stxxl: Standard Template Library for XXL Data Sets. Technical Report 18, Fakultät für Informatik, University of Karlsruhe (2005)

    Google Scholar 

  28. Dementiev, R., Kettner, L., Sanders, P.: STXXL: Standard Template library for XXL data sets. Software: Practice and Experience 38(6), 589–637 (2008)

    Google Scholar 

  29. Dementiev, R., Sanders, P., Kettner, L.: STXXL: Standard Template library for XXL data sets. In: Brodal, G.S., Leonardi, S. (eds.) ESA 2005. LNCS, vol. 3669, pp. 640–651. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  30. Dementiev, R., Sanders, P., Schultes, D., Sibeyn, J.: Engineering an external memory minimum spanning tree algorithm. In: Proceedings of the 3rd International Conference on Theoretical Computer Science (TCS), pp. 195–208. Kluwer, Dordrecht (2004)

    Google Scholar 

  31. Dijkstra, E.W.: A note on two problems in connexion with graphs. In Numerische Mathematik 1, 269–271 (1959)

    Article  MathSciNet  MATH  Google Scholar 

  32. Edelkamp, S., Jabbar, S.: Large-scale directed model checking LTL. In: Valmari, A. (ed.) SPIN 2006. LNCS, vol. 3925, pp. 1–18. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  33. Edelkamp, S., Jabbar, S., Schrödl, S.: External A*. In: Biundo, S., Frühwirth, T., Palm, G. (eds.) KI 2004. LNCS (LNAI), vol. 3238, pp. 226–240. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  34. Erdős, P., Rényi, A.: On random graphs. Publ. Math. Debrecen 6, 290–297 (1959)

    MathSciNet  MATH  Google Scholar 

  35. Erdős, P., Rényi, A.: On the evolution of random graphs. Magyar Tud. Akad. Mat. Kutato Int. Kozl. 5, 17–61 (1960)

    MathSciNet  MATH  Google Scholar 

  36. Fredman, M.L., Tarjan, R.E.: Fibonacci heaps and their uses in improved network optimization algorithms. Journal of the ACM 34(3), 596–615 (1987)

    Article  MathSciNet  Google Scholar 

  37. Frigo, M., Leiserson, C.E., Prokop, H., Ramachandran, S.: Cache-oblivious algorithms. In: 40th Annual Symposium on Foundations of Computer Science, pp. 285–297. IEEE Computer Society Press, Los Alamitos (1999)

    Google Scholar 

  38. Galler, B.A., Fisher, M.J.: An improved equivalence algorithm. Communications of the ACM 7(5), 301–303 (1964)

    Article  MATH  Google Scholar 

  39. Goldberg, A., Werneck, R.: Computing point-to-point shortest paths from external memory. In: Proceedings of the 7th workshop on Algorithms Engineering and Experiments (ALENEX), pp. 26–40 (2005)

    Google Scholar 

  40. Hart, P.E., Nilsson, N.J., Raphael, B.: A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics SSC 4(2), 100–107 (1968)

    Article  Google Scholar 

  41. Jabbar, S., Edelkamp, S.: I/O efficient directed model checking. In: Cousot, R. (ed.) VMCAI 2005. LNCS, vol. 3385, pp. 313–329. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  42. Jabbar, S., Edelkamp, S.: Parallel external directed model checking with linear I/O. In: Emerson, E.A., Namjoshi, K.S. (eds.) VMCAI 2006. LNCS, vol. 3855, pp. 237–251. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  43. Jarník, V.: O jistém problému minimálním [About a certain minimal problem]. Práce Moravské Přírodovědecké Společnosti 6, 57–63 (1930)

    Google Scholar 

  44. Karger, D., Klein, P., Tarjan, R.: A randomized linear time algorithm to find minimum spanning trees. Journal of the ACM 42(2), 321–328 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  45. Kruskal, J.B.: On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical Society 7(1), 48–50 (1956)

    Article  MathSciNet  MATH  Google Scholar 

  46. Kumar, V., Schwabe, E.J.: Improved algorithms and data structures for solving graph problems in external memory. In: Proceedings of the 8th Symposium on Parallel and Distributed Processing, pp. 169–177. IEEE, Los Alamitos (1996)

    Google Scholar 

  47. Laura, L., Leonardi, S., Millozzi, S., Meyer, U., Sibeyn, J.F.: Algorithms and experiments for the webgraph. In: Di Battista, G., Zwick, U. (eds.) ESA 2003. LNCS, vol. 2832, pp. 703–714. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  48. Maon, Y., Scheiber, B., Vishkin, U.: Parallel ear decomposition search (EDS) and st-numbering in graphs. Theoretical Computer Science 47, 277–298 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  49. Mehlhorn, K., Meyer, U.: External-memory Breadth-First Search with sublinear I/O. In: Möhring, R.H., Raman, R. (eds.) ESA 2002. LNCS, vol. 2461, pp. 723–735. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  50. Mehlhorn, K., Naher, S.: The LEDA Platform of Combinatorial and Geometric Computing. Cambridge University Press, Cambridge (1999)

    MATH  Google Scholar 

  51. Meyer, U., Zeh, N.: I/O-efficient undirected shortest paths. In: Di Battista, G., Zwick, U. (eds.) ESA 2003. LNCS, vol. 2832, pp. 434–445. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  52. Meyer, U., Zeh, N.: I/O-efficient undirected shortest paths with unbounded edge lengths. In: Azar, Y., Erlebach, T. (eds.) ESA 2006. LNCS, vol. 4168, pp. 540–551. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  53. Munagala, K., Ranade, A.: I/O-complexity of graph algorithms. In: Proceedings of the 10th Annual Symposium on Discrete Algorithms (SODA), pp. 687–694. ACM-SIAM (1999)

    Google Scholar 

  54. Pettie, S.: A new approach to all-pairs shortest paths on real-weighted graphs. Theoretical Computer Science 312(1), 47–74 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  55. Pettie, S., Ramachandran, V.: An optimal minimum spanning tree algorithm. Journal of the ACM 49(1), 16–34 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  56. Prim, R.C.: Shortest connection networks and some generalisations. Bell System Technical Journal 36, 1389–1401 (1957)

    Article  Google Scholar 

  57. Sach, B., Clifford, R.: An empirical study of cache-oblivious priority queues and their application to the shortest path problem (2008), http://arxiv.org/abs/0802.1026v1

  58. Sanders, P.: Random permutations on distributed, external and hierarchical memory. Information Processing Letters 67, 305–309 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  59. Sanders, P., Schultes, D., Vetter, C.: Mobile route planning. In: Halperin, D., Mehlhorn, K. (eds.) Esa 2008. LNCS, vol. 5193, pp. 732–743. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  60. Sibeyn, J.F.: From parallel to external list ranking, Technical report, Max Planck Institut für Informatik, Saarbrücken, Germany (1997)

    Google Scholar 

  61. Sibeyn, J.F., Abello, J., Meyer, U.: Heuristics for semi-external depth first search on directed graphs. In: Proceedings of the 14th Annual ACM Symposium on Parallel Algorithms and Architectures (SPAA), pp. 282–292 (2002)

    Google Scholar 

  62. Stepanov, A., Lee, M.: The Standard Template Library. Hewlett Packard Laboratories (1995), http://www.sgi.com/tech/stl/

  63. Tarjan, R.E.: Efficiency of a good but not linear set union algorithm. Journal of ACM 22(2), 215–225 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  64. Seagate Technology, http://www.seagate.com/cda/products/discsales/marketing/detail/0,1081,628,00.html

Download references

Author information

Authors and Affiliations

  1. Max-Planck-Institut für Informatik, Saarbrücken, Germany

    Deepak Ajwani

  2. Goethe-Universität, Frankfurt am Main, Germany

    Ulrich Meyer

Authors
  1. Deepak Ajwani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ulrich Meyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer & Information Science, University of Konstanz, 78457, Konstanz, Germany

    Jürgen Lerner

  2. Institute of Theoretical Informatics, Technical University of Karlsruhe, 76128, Karlsruhe, Germany

    Dorothea Wagner

  3. Department of Biological Physics, Eötvös Lorand University, Pazmany P. Stny 1A, 1117, Budapest, Hungary

    Katharina A. Zweig

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Ajwani, D., Meyer, U. (2009). Design and Engineering of External Memory Traversal Algorithms for General Graphs. In: Lerner, J., Wagner, D., Zweig, K.A. (eds) Algorithmics of Large and Complex Networks. Lecture Notes in Computer Science, vol 5515. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02094-0_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-02094-0_1

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-02093-3

  • Online ISBN: 978-3-642-02094-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation