DL-Dedupe: Dual-Level Deduplication Scheme for Flash-Based SSDs

  • Wanhui He
  • Nong Xiao
  • Fang Liu
  • Zhiguang Chen
  • Yinjin Fu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7901)


Although flash memory-based Solid State Drive (SSD) was born as the replacement of Hard Disk Drive (HDD) due to its high performance and low power consumption, the limited write endurance of flash and the reliability problem hold back the footsteps of SSD popularization. We propose DL-Dedupe, a dual-level deduplication scheme for flash-based SSDs to reduce the amount of data written to flash memory and enhance the storage capacity at the same time by eliminating unnecessary replicated write operations, further reducing garbage collection to improve the lifespan of SSD. It combines chunk and page level deduplication to save memory utilization, reduce access latency and ensure deduplication effectiveness. It also introduces hybrid address mapping and adaptive cache replacement scheme to accelerate index lookup and reduce memory overhead for desirable access speed. The experimental results show that DL-Dedupe can significantly reduce redundant writes, thus enhance the reliability and lifespan of SSD.


SSD deduplication lifespan dual-level 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Solid state 101 – an introduction to solid state storage (EB/OI) (April 17, 2009),
  2. 2.
    Yinjin, F., Hong, J., Nong, X., Lei, T., Fang, L.: AA-Dedupe: An Application-Aware Source Deduplication Approach for Cloud Backup Services in the Personal Computing Environment. In: CLUSTER 2011 (2011)Google Scholar
  3. 3.
    Richard, V., Carl, O., Matthew, E.: Big Data: What It Is and Why You Should Care. Framingham: IDC (White Paper), 1–14 (2011)Google Scholar
  4. 4.
    Quinlan, S., Dorward, S.: Venti: a new approach to archival storage. In: FAST 2002. USENIX Association (2002)Google Scholar
  5. 5.
    Kulkarni, P., Douglis, F.: Redundancy elimination within large collections of files. In: USENIX 2004. USENIX Association (2004)Google Scholar
  6. 6.
    Guo, F., Efstathopoulos, P.: Building a high-performance deduplication system. In: USENIX ATC 2011. USENIX Association (2011)Google Scholar
  7. 7.
  8. 8.
    Lee, S., Park, D., Chung, T., Lee, D., Park, S., Song, H.: FAST: An FTL Scheme with Fully Associative Sector Translations. In: UKC (August 2005)Google Scholar
  9. 9.
    Lee, S., Shin, D.: LAST: locality-aware sector translation for NAND flash memory-based storage systems. SIGOPS 42(6) (2008)Google Scholar
  10. 10.
    Gupta, A., Kim, Y., Urgaonkar, B.: DFTL: A Flash Translation Layer Employing Demand-based Selective Caching of Page-level Address Mappings. In: ASPLOS 2009 (2009)Google Scholar
  11. 11.
    SNIA. IOTTA repository (January 2009),
  12. 12.
    Ricardo, K., Raju, R.: I/O Deduplication: Utilizing Content Similarity to Improve I/O Performance. In: FAST 2010. USENIX Association (2010)Google Scholar
  13. 13.
    Gupta, A., Pisolkar, R., Urgaonkar, B.: Leveraging Value Locality in Optimizing NAND Flash-Based SSDs. In: FAST 2011. USENIX Association (2011)Google Scholar
  14. 14.
    Debnath, B., Sengupta, S.: ChunkStash: Speeding up Inline Storage Deduplication using Flash Memory. In: USENIX ATC 2010 (2010)Google Scholar
  15. 15.
    Meister, D.: dedupv1: Improving Deduplication Throughput using Solid State Drives (SSD). In: MSST 2010 (2010)Google Scholar
  16. 16.
    Wu, G., He, X.: ΔFTL: Improving SSD Lifetime via Exploiting Content Locality. In: EuroSys 2012, Bern, Switzerland, April 10-13 (2012)Google Scholar
  17. 17.
    Chen, F., Luo, T., Zhang, X.: CAFTL: A content-aware flash translation layer enhancing the lifespan of flash memory based solid state drives. In: FAST 2011, San Jose, CA, February 15-17 (2011)Google Scholar
  18. 18.
    Kim, J., Lee, C., Lee, S.: Deduplication in SSDs: Model and Quantitative Analysis. In: MSST 2012 (2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Wanhui He
    • 1
  • Nong Xiao
    • 1
  • Fang Liu
    • 1
  • Zhiguang Chen
    • 1
  • Yinjin Fu
    • 1
  1. 1.State Key Laboratory of High Performance ComputingNational University of Defense TechnologyChangshaChina

Personalised recommendations