Sloppy Hashing and Self-Organizing Clusters

  • Michael J. Freedman
  • David Maziéres
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2735)


We are building Coral, a peer-to-peer content distribution system. Coral creates self-organizing clusters of nodes that fetch information from each other to avoid communicating with more distant or heavily-loaded servers. Coral indexes data, but does not store it. The actual content resides where it is used, such as in nodes’ local web caches. Thus, replication happens exactly in proportion to demand. We present two novel mechanisms that let Coral achieve scalability and high performance. First, a new abstraction called a distributed sloppy hash table (DSHT) lets nodes locate nearby copies of a file, regardless of its popularity, without causing hot spots in the indexing infrastructure. Second, based on the DSHT interface, we introduce a decentralized clustering algorithm by which nodes can find each other and form clusters of varying network diameters.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Andersen, D.: Resilient overlay networks testbed,
  2. 2.
    Chen, Y., Katz, R.H., Kubiatowicz, J.D.: SCAN: A dynamic, scalable, and efficient content distribution network. In: Proceedings of the International Conference on Pervasive Computing, Zurich, Switzerland (August 2002)Google Scholar
  3. 3.
    Dabek, F., Frans Kaashoek, M., Karger, D., Morris, R., Stoica, I.: Wide-area cooperative storage with CFS. In: Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP 2001), Banff, Canada (October 2001)Google Scholar
  4. 4.
    Iyer, S., Rowstron, A., Druschel, P.: Squirrel: A decentralized, peer-to-peer web cache. In: Principles of Distributed Computing (PODC), Monterey, CA (July 2002)Google Scholar
  5. 5.
    Kubiatowicz, J., Bindel, D., Chen, Y., Czerwinski, S., Eaton, P., Geels, D., Gummadi, R., Rhea, S., Weatherspoon, H., Weimer, W., Wells, C., Zhao, B.: OceanStore: An architecture for global-scale persistent storage. In: Proc. ASPLOS, Cambridge, MA (November 2000)Google Scholar
  6. 6.
    Maymounkov, P., Mazières, D.: Kademlia: A peer-to-peer information system based on the xor metric. In: Druschel, P., Kaashoek, M.F., Rowstron, A. (eds.) IPTPS 2002. LNCS, vol. 2429, p. 53. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  7. 7.
    Ratnasamy, S., Francis, P., Handley, M., Karp, R., Shenker, S.: A scalable content-addressable network. In: Proc. ACM SIGCOMM, San Diego, CA (August 2001)Google Scholar
  8. 8.
    Rowstron, A., Druschel, P.: Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In: Guerraoui, R. (ed.) Middleware 2001. LNCS, vol. 2218, p. 329. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  9. 9.
    Rowstron, A., Druschel, P.: Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility. In: Proc. 18th ACM Symposium on Operating Systems Principles (SOSP 2001), Banff, Canada (October 2001)Google Scholar
  10. 10.
    Sen, S., Wang, J.: Analyzing peer-to-peer traffic across large networks. In: Proc. ACM SIGCOMM Internet Measurement Workshop, Marseille, France ( November 2002)Google Scholar
  11. 11.
    Stading, T., Maniatis, P., Baker, M.: Peer-to-peer caching schemes to address flash crowds. In: Druschel, P., Kaashoek, M.F., Rowstron, A. (eds.) IPTPS 2002. LNCS, vol. 2429, p. 203. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  12. 12.
    Stavrou, A., Rubenstein, D., Sahu, S.: A lightweight, robust p2p system to handle flash crowds. In: IEEE International Conference on Network Protocol (ICNP), Paris, France (November 2002)Google Scholar
  13. 13.
    Stoica, I., Morris, R., Liben-Nowell, D., Karger, D.R., Frans Kaashoek, M., Dabek, F., Balakrishnan, H.: Chord: A scalable peer-to-peer lookup protocol for internet applications. IEEE/ACM Trans. on Networking (2002)Google Scholar
  14. 14.
    Zhao, B., Kubiatowicz, J., Joseph, A.: Tapestry: An infrastructure for fault-tolerant wide-area location and routing. Technical Report UCB/CSD- 01-1141, Computer Science Division, U.C. Berkeley (April 2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Michael J. Freedman
    • 1
  • David Maziéres
    • 1
  1. 1.Dept. of Computer ScienceNew York UniversityNew YorkUSA

Personalised recommendations