DA Placement: A Dual-Aware Data Placement in a Deduplicated and Erasure-Coded Storage System

  • Mingzhu DengEmail author
  • Ming Zhao
  • Fang Liu
  • Zhiguang Chen
  • Nong Xiao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11334)


Simultaneously incorporating deduplication as well as erasure coding is preferred for modern storage systems for the enhanced storage efficiency and economical data reliability. However, simple incorporation suffers from the “read imbalance problem”, in which parallel data accesses are curbed by throttled storage nodes. This problem is due to the uneven data placement in the system, which is unaware of the employment of both deduplication and erasure coding, each of whom alters the order of data if unattended. This paper proposes a systematic design and implementation of a Dual-Aware(DA) placement in a combined storage system to achieve both deduplication-awareness and erasure-coding-awareness at the same time. DA not only records the node number of each unique data to allow for quick references with ease, but also dynamically tracks used nodes for each writes request. In this way, deduplication awareness is formed to skip inconvenient placement locations. Besides, DA serializes the placement of parity blocks with a stripe and across stripes. Such realization of erasure coding awareness ensures the separation of data and parity, as well as maintains data sequentiality at bordering stripes. Additionally, DA manages to extend with further load-balancing through an innovative use of the deduplication level, which intuitively predicts future accesses of a piece of data. In short, DA manages to boost system performance with little memory or computation cost. Extensive experiments using both real-world traces and synthesized workloads, prove DA achieves a better read performance. For example, DA respectively leads an average latency margin of 30.86% and 29.63%, over the baseline rolling placement(BA) and random placement(RA) under CAFTL traces over a default cluster of 12 nodes with RS(8,4).



We would like to greatly appreciate the anonymous reviewers for their insightful comments. This work is supported by the National Natural Science Foundation of China under Grant Nos. 61433019, U1435217, and the National High Technology Research and Development Program of China under Grant No. 2016YFB1000302.


  1. 1.
    Chen, F., Luo, T., Zhang, X.: CAFTL: a content-aware flash translation layer enhancing the lifespan of flash memory based solid state drives. FAST 11, 77–90 (2011)Google Scholar
  2. 2.
    Hong, B., Plantenberg, D., Long, D.D., Sivan-Zimet, M.: Duplicate data elimination in a SAN file system. In: MSST, pp. 301–314 (2004)Google Scholar
  3. 3.
    Huang, C., et al.: Erasure coding in windows azure storage. In: Usenix Annual Technical Conference, pp. 15–26. , Boston, MA (2012)Google Scholar
  4. 4.
    Jin, K., Miller, E.L.: The effectiveness of deduplication on virtual machine disk images. In: Proceedings of SYSTOR 2009: The Israeli Experimental Systems Conference, p. 7. ACM (2009)Google Scholar
  5. 5.
    Li, W., Jean-Baptise, G., Riveros, J., Narasimhan, G., Zhang, T., Zhao, M.: Cachededup: In-line deduplication for flash caching. In: FAST, pp. 301–314 (2016)Google Scholar
  6. 6.
    Li, X., Dong, B., Xiao, L., Ruan, L., Liu, D.: CEFLS: a cost-effective file lookup service in a distributed metadata file system. In: Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2012), pp. 25–32. IEEE Computer Society (2012)Google Scholar
  7. 7.
    Li, X., Dong, B., Xiao, L., Ruan, L., Liu, D.: HCCache: a hybrid client-side cache management scheme for i/o-intensive workloads in network-based file systems. In: 2012 13th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT), pp. 467–473. IEEE (2012)Google Scholar
  8. 8.
    Li, X., Xiao, L., Ke, X., Dong, B., Li, R., Liu, D.: Towards hybrid client-side cache management in network-based file systems. Comput. Sci. Inf. Syst. 11(1), 271–289 (2014)CrossRefGoogle Scholar
  9. 9.
    Liu, N., et al.: On the role of burst buffers in leadership-class storage systems. In: 2012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–11. IEEE (2012)Google Scholar
  10. 10.
    Liu, Y., Figueiredo, R., Xu, Y., Zhao, M.: On the design and implementation of a simulator for parallel file system research. In: 2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–5. IEEE (2013)Google Scholar
  11. 11.
    Meister, D., Brinkmann, A.: Multi-level comparison of data deduplication in a backup scenario. In: Proceedings of SYSTOR 2009: The Israeli Experimental Systems Conference, p. 8. ACM (2009)Google Scholar
  12. 12.
    Ng, C.H., Lee, P.P.: Revdedup: a reverse deduplication storage system optimized for reads to latest backups. In: Proceedings of the 4th Asia-Pacific Workshop on Systems, p. 15. ACM (2013)Google Scholar
  13. 13.
    Plank, J.S.: Erasure codes for storage systems: a brief primer. Usenix Mag. 38(6), 44–50 (2013)MathSciNetGoogle Scholar
  14. 14.
    Quinlan, S., Dorward, S.: Venti: a new approach to archival storage. FAST 2, 89–101 (2002)Google Scholar
  15. 15.
    Rashmi, K., Chowdhury, M., Kosaian, J., Stoica, I., Ramchandran, K.: EC-Cache: load-balanced, low-latency cluster caching with online erasure coding. In: OSDI, pp. 401–417 (2016)Google Scholar
  16. 16.
    Rashmi, K., Shah, N.B., Gu, D., Kuang, H., Borthakur, D., Ramchandran, K.: A Hitchhiker’s guide to fast and efficient data reconstruction in erasure-coded data centers. ACM SIGCOMM Comput. Commun. Rev. 44(4), 331–342 (2015)CrossRefGoogle Scholar
  17. 17.
    Rivest, R.: The MD5 message-digest algorithm (1992)Google Scholar
  18. 18.
    Secure Hash Standard: Federal information processing standards publication 180-1 (1995)Google Scholar
  19. 19.
    Xu, M., Zhu, Y., Lee, P.P., Xu, Y.: Even data placement for load balance in reliable distributed deduplication storage systems. In: 2015 IEEE 23rd International Symposium on Quality of Service (IWQoS), pp. 349–358. IEEE (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Mingzhu Deng
    • 1
    • 2
    Email author
  • Ming Zhao
    • 2
  • Fang Liu
    • 3
  • Zhiguang Chen
    • 1
    • 3
  • Nong Xiao
    • 1
    • 3
  1. 1.College of Computer, National University of Defense TechnologyChangshaChina
  2. 2.Arizona State UniversityTempeUSA
  3. 3.School of Data and Computer ScienceSUN YAT-SEN UniversityGuangzhouChina

Personalised recommendations