An initial evaluation of 6Stor, a dynamically scalable IPv6-centric distributed object storage system

Precise architecture description and first benchmark results
  • Guillaume RutyEmail author
  • Jean-Louis Rougier
  • André Surcouf
  • Aloys Augustin


The exponentially growing demand for storage puts a huge stress on traditional distributed storage systems. Historically, I/Ops (Inputs/Outputs per second) of hard drives have been the main limitation of storage systems. With the rapid deployment of solid state drives (SSDs) and the expected evolutions of their capacities, price and performance, we claim that CPU and network capacities will become bottlenecks in the future. In this context, we introduce 6Stor, an innovative, software-defined distributed storage system fully integrated with the networking layer. This storage system departs from traditional approaches in two manners: it leverages IPv6 new capabilities to increase the efficiency of its data plane—notably by using directly UDP and TCP rather than HTTP—and thus its performance; and it circumvents scalability limitations of other distributed systems by using a fully distributed metadata layer of indirection to offer flexibility. In this paper, we introduce and describe in details the architecture of 6Stor, with an emphasis on dynamic scalability and robustness to failure. We also present a testbed that we use to evaluate our novel approach by using Ceph—another well known distributed storage system—as baseline. Results obtained on an extensive testbed are presented and some initial conclusions are drawn.


Distributed object storage IPv6 centric networking Software defined systems 


  1. 1.
    Chen, P.M., Lee, E.K., Gibson, G.A., Katz, R.H., Patterson, D.A.: Raid: High-performance, reliable secondary storage. ACM Comput. Surv. (CSUR) 26(2), 145–185 (1994)CrossRefGoogle Scholar
  2. 2.
    Cabrera, L.-F., Long, D.D.: Swift: using distributed disk striping to provide high I/O data rates. University of California, Santa Cruz, Computer Research Laboratory, vol. 8523 (1991)Google Scholar
  3. 3.
    Hachman, M.: Notebook hard drives are dead: how SSDS will dominate mobile pc storage by 2018, December 2015.
  4. 4.
    Nanavati, M., Schwarzkopf, M., Wires, J., Warfield, A.: Non-volatile storage. Queue 139, 20:33–20:56.
  5. 5.
    Davis, R.: The network is the new storage bottleneck, November 2016.
  6. 6.
    McKusick, K., Quinlan, S.: Gfs: Evolution on fast-forward. Commun. ACM 533, 42–49 (2010).
  7. 7.
    Shvachko, K.: HDFS scalability: the limits to growth. Login: the magazine of USENIX & SAGE 35, 6–16 (2010)Google Scholar
  8. 8.
    Lee, D.-Y., Jeong, K., Han, S.-H., Kim, J.-S., Hwang, J.-Y., Cho, S.: Understanding write behaviors of storage backends in ceph object store. In: Proceedings of the 2017 IEEE International Conference on Massive Storage Systems and Technology, vol. 10 (2017)Google Scholar
  9. 9.
    Ruty, G., Surcouf, A., Rougier, J.L.: Collapsing the layers: 6stor, a scalable and ipv6-centric distributed storage system. In: 2017 Fourth International Conference on Software Defined Systems (SDS), May 2017, pp. 81–86 (2017)Google Scholar
  10. 10.
    Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D.E., Maltzahn, C.: Ceph: A scalable, high-performance distributed file system. In: Proceedings of the 7th Symposium on Operating Systems Design and Implementation, ser. OSDI ’06. Berkeley, CA, USA: USENIX Association, 2006, pp. 307–320.
  11. 11.
    DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: Amazon’s highly available key-value store. In: Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles, ser. SOSP ’07, pp. 205–220. ACM, New York (2007).
  12. 12.
    Arnold, J.: Openstack Swift: Using, Administering, and Developing for Swift Object Storage. O’Reilly Media, Inc., Sebastopol (2014)Google Scholar
  13. 13.
  14. 14.
    Ghemawat, S., Gobioff, H., Leung, S.-T.: The google file system. SIGOPS Oper. Syst. Rev. 37(5), 29–43 (2003).
  15. 15.
    Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), May 2010, pp. 1–10 (2010)Google Scholar
  16. 16.
    Fizians, S.: Rozofs: a fault tolerant i/o intensive distributed file system based on mojette erasure code. In: Workshop Autonomic Oct, vol. 16, p. 17 (2014)Google Scholar
  17. 17.
    Weil, S.A., Leung, A.W., Brandt, S.A., Maltzahn, C.: Rados: a scalable, reliable storage service for petabyte-scale storage clusters. In: Proceedings of the 2nd International Workshop on Petascale Data Storage: Held in Conjunction with Supercomputing’07. ACM, pp. 35–44 (2007)Google Scholar
  18. 18.
    Weil, S.A., Brandt, S.A., Miller, E.L., Maltzahn, C.: Crush: Controlled, scalable, decentralized placement of replicated data. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, ser. SC ’06. ACM, New York (2006).
  19. 19.
    Metz, C.: Google remakes online empire with ‘colossus’. Wired (2012)
  20. 20.
  21. 21.
    Faibish, S., Peng, T., Bent, J.M., Gupta, U.K., Pedone, Jr, J.M.: Lustre file system, Oct. 3 2017, US Patent 9,779,108Google Scholar
  22. 22.
    Niazi, S., Ismail, M., Haridi, S., Dowling, J., Grohsschmiedt, S., Ronström, M.: Hopsfs: Scaling hierarchical file system metadata using newsql databases. In: 15th USENIX Conference on File and Storage Technologies (FAST 17). Santa Clara, CA: USENIX Association, pp. 89–104 (2017).
  23. 23.
    Noel, R.R., Lama, P.: Taming performance hotspots in cloud storage with dynamic load redistribution. In: 2017 IEEE 10th International Conference on Cloud Computing (CLOUD), June 2017, pp. 42–49 (2017)Google Scholar
  24. 24.
    Krish, K.R., Anwar, A., Butt, A.R.: hats: a heterogeneity-aware tiered storage for hadoop. In: 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, May 2014, pp. 502–511 (2014)Google Scholar
  25. 25.
  26. 26.
    Nvfuse: a nvme user-space filesystem.
  27. 27.
    Marinos, I., Watson, R.N., Handley, M., Stewart, R.R.: Disk, crypt, net: Rethinking the stack for high-performance video streaming. In: Proceedings of the Conference of the ACM Special Interest Group on Data Communication, ser. SIGCOMM ’17, pp. 211–224. ACM, New York (2017).
  28. 28.
    Braun, M.B., Crowcroft, J.: Sna: Sourceless network architecture. University of Cambridge, Computer Laboratory, Tech. Rep. (2014)Google Scholar
  29. 29.
    Deen, G., Naik, G., Brzozowski, J.J., Daigle, L., Rose, B., Townsley, M.: “Using Media Encoding Networks to address MPEG-DASH video ,” Internet Engineering Task Force, Internet-Draft draft-deen-naik-ggie-men-mpeg-dash-00, July 2016, work in Progress.
  30. 30.
    Srinivasan, S., Schulzrinne, H.: Ipv6 addresses as content names in information-centric networkingGoogle Scholar
  31. 31.
    Ahlgren, B., Dannewitz, C., Imbrenda, C., Kutscher, D., Ohlman, B.: A survey of information-centric networking. IEEE Communications Magazine 50(7) (2012)Google Scholar
  32. 32.
    Muscariello, L., Carofiglio, G., Auge, J., Papalini, M.: Hybrid information-centric networking, internet engineering task force, internet-draft draft-muscariello-intarea-hicn-00, Jun. 2018, work in Progress.
  33. 33.
    Carofiglio, G.: Mobile video delivery with hybrid icn. Cisco, Tech. Rep, Tech. Rep (2016)Google Scholar
  34. 34.
    Bonald, T., Oueslati-Boulahia, S., Roberts, J.: Ip traffic and qos control: the need for a flow-aware architecture. In: World Telecommunications Congress. Citeseer (2002)Google Scholar
  35. 35.
    Zhao, W., Olshefski, D., Schulzrinne, H.: Internet quality of service: an overview. Columbia University, New York, Technical Report CUCS-003-00 (2000)Google Scholar
  36. 36.
    Dimakis, A.G., Ramchandran, K., Wu, Y., Suh, C.: A survey on network codes for distributed storage. Proc. IEEE 99(3), 476–489 (2011)CrossRefGoogle Scholar
  37. 37.
    Ford, D., Labelle, F., Popovici, F.I., Stokely, M., Truong, V.-A., Barroso, L., Grimes, C., Quinlan, S.: Availability in globally distributed storage systems. In: Osdi, vol. 10, pp. 1–7 (2010)Google Scholar
  38. 38.
    Luo, T., Lee, R., Mesnier, M., Chen, F., Zhang, X.: hstorage-db: heterogeneity-aware data management to exploit the full capability of hybrid storage systems. Proc. VLDB Endow. 5(10), 1076–1087 (2012).
  39. 39.
    Kaneko, S., Nakamura, T., Kamei, H., Muraoka, H.: A guideline for data placement in heterogeneous distributed storage systems. In: 2016 5th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI), July 2016, pp. 942–945 (2016)Google Scholar
  40. 40.
    Qin, Y., Ai, X., Chen, L., Yang, W.: Data placement strategy in data center distributed storage systems. In: 2016 IEEE International Conference on Communication Systems (ICCS), Dec 2016, pp. 1–6 (2016)Google Scholar
  41. 41.
    Wu, C.F., Hsu, T.C., Yang, H., Chung, Y.C.: File placement mechanisms for improving write throughputs of cloud storage services based on ceph and hdfs. In: 2017 International Conference on Applied System Innovation (ICASI), May 2017, pp. 1725–1728 (2017)Google Scholar
  42. 42.
    Stoica, I., Morris, R., Liben-Nowell, D., Karger, D.R., Kaashoek, M.F., Dabek, F., Balakrishnan, H.: Chord: A scalable peer-to-peer lookup protocol for internet applications. IEEE/ACM Trans. Netw. 11(1), 17–32 (2003).
  43. 43.
    Claise, B., Fullmer, M., Calato, P., Penno, R.: Ipfix protocol specification. Interrnet-draft, work in progress (2005)Google Scholar
  44. 44.
    Wikipedia, “NetFlow—Wikipedia, the free encyclopedia. (2018)
  45. 45.
    Hawari, M.: “TCP fast open cookie for IPv6 prefixes,” internet engineering task force, internet-draft draft-hawari-tcpm-tfo-ipv6-prefixes-00, July 2015, work in Progress.
  46. 46.
    Goedegebure, S.: Big buck bunny.
  47. 47.
    Baumann, T.: Valkaama project.
  48. 48.
    wrk—a http benchmarking tool.
  49. 49.
  50. 50.
    Filsfils, C., Nainar, N.K., Pignataro, C., Cardona, J.C., Francois, P.: The segment routing architecture. In: 2015 IEEE Global Communications Conference (GLOBECOM), Dec 2015, pp. 1–6 (2015)Google Scholar
  51. 51.
  52. 52.
    Desmouceaux, Y., Pfister, P., Tollet, J., Townsley, M., Clausen, T.: 6lb: scalable and application-aware load balancing with segment routing. IEEE/ACM Trans. Netw. 26(2), 819–834 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Cisco Systems PIRLParisFrance
  2. 2.Telecom ParisTechParisFrance

Personalised recommendations