Interoperable Convergence of Storage, Networking, and Computation

  • Micah BeckEmail author
  • Terry Moore
  • Piotr Luszczek
  • Anthony Danalis
Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 70)


In every form of digital store-and-forward communication, intermediate forwarding nodes are computers, with attendant memory and processing resources. This has inevitably stimulated efforts to create a wide-area infrastructure that goes beyond simple store-and-forward to create a platform that makes more general and varied use of the potential of this collection of increasingly powerful nodes. Historically, these efforts predate the advent of globally routed packet networking. The desire for a converged infrastructure of this kind has only intensified over the last 30 years, as memory, storage, and processing resources have increased in both density and speed while simultaneously decreasing in cost. Although there is a general consensus that it should be possible to define and deploy such a dramatically more capable wide-area platform, a great deal of investment in research prototypes has yet to produce a credible candidate architecture. Drawing on technical analysis, historical examples, and case studies, we present an argument for the hypothesis that in order to realize a distributed system with the kind of convergent generality and deployment scalability that might qualify as “future-defining,” we must build it from a small set of simple, generic, and limited abstractions of the low level resources (processing, storage and network) of its intermediate nodes.


Networking Distributed computing 



The ideas in this paper were influenced by many spirited discussions with Martin Swany on the integration of storage and processing with scalable networking, and by recent conversations with Glenn Ricart on the definition and justification of interoperable convergence. The concept of “exposed buffer protocol/processing” was coined during discussions between Swany and Beck, although its best definition and implementation are still subject to debate. The authors are also indebted to David Rogers for his professional rendering of the artwork in this any many other papers and presentations, and to Chris Brumgard for his helpful comments.


  1. 1.
    Mynatt, E., Clark, J., Hager, G., Lopresti, D., Morrisett, G., Nahrstedt, K., Pappas, G., Patel, S., Rexford, J., Wright, H., et al.: A national research agenda for intelligent infrastructure. arXiv preprint arXiv:1705.01920 (2017)
  2. 2.
    Networking, I. T. Research, and D. N. Program.: Smart and Connected Cities Framework. (2015)
  3. 3.
    Nahrstedt, K., Cassandras, C.G., Catlett, C.: City-scale intelligent systems and platforms. arXiv preprint arXiv:1705.01990 (2017)
  4. 4.
    Anderson, T., Peterson, L., Shenker, S., Turner, J.: Overcoming the internet impasse through virtualization. Computer 38(4), 34–41 (2005)CrossRefGoogle Scholar
  5. 5.
    Tennenhouse, D.L., Wetherall, D.J.: Towards an active network architecture. Comput. Commun. Rev. 26, 5–18 (1996)CrossRefGoogle Scholar
  6. 6.
    Foster, I., Kesselman, C.: The Grid: Blueprint for a New Computing Infrastructure, pp. 47–48. Advanced computing. Computer systems design. Morgan Kaufmann Publishers (1999)Google Scholar
  7. 7.
    Chun, B., Culler, D., Roscoe, T., Bavier, A., Peterson, L., Wawrzoniak, M., Bowman, M.: Planetlab: an overlay testbed for broad-coverage services. SIGCOMM Comput. Commun. Rev. 33(3), 3–12 (2003)CrossRefGoogle Scholar
  8. 8.
    McGeer, R., Berman, M., Elliott, C., Ricci, R. (eds.): The GENI Book. Springer, Berlin (2016)Google Scholar
  9. 9.
    Clark, D.D.: Interoperation, open interfaces, and protocol architecture. Unpredictable Certainty White Pap. 2, 133–144 (1995)Google Scholar
  10. 10.
    Uux(1p) posix programmer’s manual (2013)Google Scholar
  11. 11.
    Saltzer, J.H., Reed, D.P., Clark, D.D.: End-to-end arguments in system design. ACM Trans. Comput. Syst. 2(4), 277–288 (1984)CrossRefGoogle Scholar
  12. 12.
    Carpenter, B., Brim, S.: Middleboxes: Taxonomy and issues. RFC 3234, Feb 2002. Network Working GroupGoogle Scholar
  13. 13.
    Dongarra, J., Golub, G.H., Grosse, E., Moler, C., Moore, K.: Netlib and NA-Net: building a scientific computing community. IEEE Ann. Hist. Comput. 30, 30–41 (2008)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Chankhunthod, A., Danzig, P.B., Neerdaels, C., Schwartz, M.F., Worrell, K.J.: A hierarchical Internet object cache. In: Proceedings of the 1996 USENIX Technical Conference, pp. 153–163 (1995)Google Scholar
  15. 15.
    Wessels, D., Claffy, K.: ICP and the Squid web cache. IEEE J. Sel. Areas Commun. 16, 345–357 (1998)CrossRefGoogle Scholar
  16. 16.
    Beck, M., Moore, T.: The Internet2 distributed storage infrastructure project: an architecture for internet content channels. In: Computer Networking and ISDN Systems, pp. 2141–2148 (1998)CrossRefGoogle Scholar
  17. 17.
    Kirkpatrick, D.: IBM’s Olympic Fiasco Department of Groundless Optimism. Fortune Magazine, 9 Sept 1996Google Scholar
  18. 18.
    Buyya, R., Pathan, M., Vakali, A. (eds.): Content Delivery Networks. Springer, Berlin (2008)Google Scholar
  19. 19.
    Nygren, E., Sitaraman, R.K., Sun, J.: The Akamai network: a platform for high-performance internet applications. In: SIGOPS Operating Systems Review (2010)Google Scholar
  20. 20.
    Beck, M., Moore, T., Plank, J.S.: An end-to-end approach to globally scalable network storage. In: ACM SIGCOMM 2002 (2002)Google Scholar
  21. 21.
    Beck, M., Moore, T., Plank, J.S.: An end-to-end approach to globally scalable programmable networking. In: Future Directions in Network Architecture, pp. 328–339. ACM Press (2003)Google Scholar
  22. 22.
    David, P.A.: Path dependence: a foundational concept for historical social science. Cliometrica 1(2), 91–114 (2007)CrossRefGoogle Scholar
  23. 23.
    Messerschmitt, D.G.: The convergence of telecommunications and computing: what are the implications today? Proc. IEEE 84(8), 1167–1186 (1996)CrossRefGoogle Scholar
  24. 24.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  25. 25.
    Peterson, L.L., Davie, B.S.: Computer Networks: A Systems Approach, 5th edn. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (2011)zbMATHGoogle Scholar
  26. 26.
    Beck, M.: On the hourglass model, end-to-end arguments, and deployment scalability. Commun, ACM (to appear 2018)Google Scholar
  27. 27.
    Will the real end-to-end argument please stand up?
  28. 28.
    Ritchie, D.M., Thompson, K.: The Unix time-sharing system. Commun. ACM 17, 365–375 (1974)CrossRefGoogle Scholar
  29. 29.
    Satyanarayanan, M.: The emergence of edge computing. Computer 50(1), 30–39 (2017)CrossRefGoogle Scholar
  30. 30.
    Plank, J.S., Bassi, A., Beck, M., Moore, T., Swany, D.M., Wolski, R.: Managing data storage in the network. IEEE Internet Comput. 5(5), 50–58 (2001)CrossRefGoogle Scholar
  31. 31.
    Beck, M., Arnold, D., Bassi, R., Berman, F., Casanova, H., Moore, T., Obertelli, G., Plank, J., Swany, M., Vadhiyar, S., Wolski, R.: Logistical computing and internetworking: middleware for the use of storage in communication. In: In 3rd Annual International Workshop on Active Middleware Services (AMS) (2001)Google Scholar
  32. 32.
    Plank, J.S., Atchley, S., Ding, Y., Beck, M.: Algorithms for high performance, wide-area, distributed file downloads. Technical report, Letters (2002)Google Scholar
  33. 33.
    Beck, M., Liu, H., Huang, J., Moore, T.: Scalable distributed execution environment for large data visualization. In: IEEE Explorer, Nov 2007Google Scholar
  34. 34.
    Liu, H.: Scalable, data-intensive network computation. Ph.D. thesis, University of Tennessee, Knoxville (2008)Google Scholar
  35. 35.
    Liu, H., Beck, M., Huang, J.: Dynamic co-scheduling of distributed computation and replication. In: IEEE International Symposium on Cluster Computing and the Grid, May 2006Google Scholar
  36. 36.
    Kitchin, R.: The Data Revolution: Big Data, Open Data, Data Infrastructures and their Consequences. Sage, London (2014)Google Scholar
  37. 37.
    Reichman, O.J., Jones, M.B., Schildhauer, M.P.: Challenges and opportunities of open data in ecology. Science 331(6018), 703–705 (2011)CrossRefGoogle Scholar
  38. 38.
    Board, S.S., Council, N.R., et al.: Landsat and Beyond: Sustaining and Enhancing the Nation’s Land Imaging Program. National Academies Press (2014)Google Scholar
  39. 39.
    Bird, I.: Computing for the Large Hadron Collider. Annu. Rev. Nucl. Part. Sci. 61, 99–118 (2011)CrossRefGoogle Scholar
  40. 40.
    Breeding, M.: Building a digital library of television news. Comput. Libr. 23(6), 47–49 (2003)Google Scholar
  41. 41.
    Wikipedia Contributors. Information logistics (2017) (Online). Accessed 15 Sept 2018Google Scholar
  42. 42.
    Banerjee, S., Wu, D.O.: Final report from the NSF Workshop on Future Directions in Wireless Networking. National Science Foundation (2013)Google Scholar
  43. 43.
    Chen, M., Mao, S., Liu, Y.: Big data: a survey. Mobile Netw. Appl. 19(2), 171–209 (2014)CrossRefGoogle Scholar
  44. 44.
    Avron, H., Maymounkov, P., Toledo, S.: Blendenpik: supercharging LAPACK’s least-squares solver. SIAM J. Sci. Comput. 32(3), 1217–1236 (2010)MathSciNetCrossRefGoogle Scholar
  45. 45.
    Drineas, P., Mahoney, M.W.: RandNLA: randomized numerical linear algebra. Commun. ACM 59(6), 80–90 (2016)CrossRefGoogle Scholar
  46. 46.
    Valiant, L.G.: A theory of the learnable. Commun. ACM 27, 1134–1142 (1984)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Micah Beck
    • 1
    Email author
  • Terry Moore
    • 1
  • Piotr Luszczek
    • 1
  • Anthony Danalis
    • 1
  1. 1.Department of Electrical Engineering and Computer ScienceUniversity of TennesseeKnoxvilleUSA

Personalised recommendations