Improving Restore Performance of Deduplication Systems by Leveraging the Chunk Sequence in Backup Stream

  • Ru Yang
  • Yuhui DengEmail author
  • Cheng Hu
  • Lei Si
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11334)


Traditional deduplication based backup systems normally employ containers to reduce the chunk fragmentation, thus improving the restore performance. However, the shared chunks belonging to a single backup grows with the increase of the number of backups. Those shared chunks are normally distributed across multiple containers. This feature increases chunk fragmentation and significantly degrades the restore performance. In order to improve the restore performance, some schemes are proposed to optimize the replacement strategy of the restore cache, such as the ones using LRU and OPT. However, LRU is inefficient and OPT consumes additional computational overhead. By analyzing the backup and restore process, we observe that the sequence of the chunks in the backup stream is consistent to that in the restore stream. Based on this observation, this paper proposes an off-line optimal replacement strategy—OFL for the restore cache. The OFL records the chunk sequence of backup process, and then uses this sequence to calculate the exact information of the required chunks in advance for the restore process. Finally, accurate prefetch will be employed by leveraging the above information to reduce the impact of chunk fragmentation. Real data sets are employed to evaluate the proposed OFL. The experimental results demonstrate that OFL improves the restore performance over 8% in contrast to the traditional LRU and OPT.



This work is supported by the NSFC (No. 61572232), in part by the Science and Technology Planning Project of Guangzhou (No. 201802010028, and No. 201802010060), in part by the Science and Technology Planning Project of Nansha (No. 2017CX006), and in part by the Open Research Fund of Key Laboratory of Computer System and Architecture, Institute of Computing Technology, Chinese Academy of Sciences under Grant CARCH201705.


  1. 1.
    Dubois, L., Amaldas, M., Sheppard, E.: Key considerations as deduplication evolves into primary storage. White Paper (2011)Google Scholar
  2. 2.
    Deng, Y.: What is the future of disk drives, death or rebirth? ACM Comput. Surv. 43(3), 23:1–23:27 (2011)CrossRefGoogle Scholar
  3. 3.
    Zhou, K., Hu, S., Huang, P., Zhao, Y.: LX-SSD: enhancing the lifespan of NAND flash-based memory via recycling invalid pages. In: Proceedings of the 33rd International Conference on Massive Storage Systems and Technology, MSST 2017 (2017)Google Scholar
  4. 4.
    Wei, J., Jiang, H., Zhou, K., Feng, D.: Efficiently representing membership for variable large data sets. IEEE Trans. Parallel Distrib. Syst. 25(4), 960–970 (2014)CrossRefGoogle Scholar
  5. 5.
    Benjamin, Z., Kai, L., Patterson, R.H.: Avoiding the disk bottleneck in the data domain deduplication file system. In: Proceedings of the 6th USENIX Conference on File and Storage Technologies, FAST 2008, vol. 8, pp. 269–282 (2008)Google Scholar
  6. 6.
    Bhagwat, D., Eshghi, K., Long, D.D.E., Lillibridge, M.: Extreme binning: scalable, parallel deduplication for chunk-based file backup. In: Proceedings of the 2009 IEEE International Symposium on Modeling, Analysis Simulation of Computer and Telecommunication Systems, pp. 1–9 (2009)Google Scholar
  7. 7.
    Mark, L., Kave, E., Deepavali, B., Vinay, D., Greg, T., Peter, C.: Sparse indexing: large scale, inline deduplication using sampling and locality. In: Proceedings of the 7th USENIX Conference on File and Storage Technologies, Fast 2009, vol. 9, pp. 111–123 (2009)Google Scholar
  8. 8.
    Wen, X., Hong, J., Dan, F., Yu, H.: SiLo: a similarity-locality based near-exact deduplication scheme with low ram overhead and high throughput. In: Proceedings of the 2011 USENIX Conference on USENIX Annual Technical Conference, USENIXATC 2011, pp. 26–28 (2011)Google Scholar
  9. 9.
    Zhou, Y., Deng, Y., Yang, L.T., Yang, R., Si, L.: LDFS: a low latency in-line data deduplication file system. IEEE Access 6, 15 743–15 753 (2018)CrossRefGoogle Scholar
  10. 10.
    Erik, K., Cristian, U., Cezary, D.: Bimodal content defined chunking for backup streams. In: Proceedings of the 8th USENIX Conference on File and Storage Technologies, FAST 2010, pp. 239–252 (2010)Google Scholar
  11. 11.
    Quinlan, S., Dorward, S.: Venti: a new approach to archival storage. In: Proceedings of the Conference on File Storage Technologies, FAST 2002, vol. 2, pp. 89–101 (2002)Google Scholar
  12. 12.
    Athicha, M., Benjie, C., David, M.: A low-bandwidth network file system. In: Proceedings of the 18th ACM Symposium on Operating Systems Principles, vol. 35, no. 5, pp. 174–187. ACM (2001)Google Scholar
  13. 13.
    Nam, Y.J., Park, D., Du, D.H.: Assuring demanded read performance of data deduplication storage with backup datasets. In: Proceedings of the 2012 IEEE 20th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2012, pp. 201–208. IEEE (2012)Google Scholar
  14. 14.
    Deng, Y., Huang, X., Song, L., Zhou, Y., Wang, F.: Memory deduplication: an effective approach to improve the memory system. J. Inf. Sci. Eng. 33(5), 1103–1120 (2017)Google Scholar
  15. 15.
    Deng, Y., Hu, Y., Meng, X., Zhu, Y., Zhang, Z., Han, J.: Predictively booting nodes to minimize performance degradation of a power-aware web cluster. Cluster Comput. 17(4), 1309–1322 (2014)CrossRefGoogle Scholar
  16. 16.
    Qu, Z., Chen, Y.: Efficient data restoration for a disk-based network backup system. In: Proceedings of the IEEE International Conference, vol. 1, pp. 584–590 (2004)Google Scholar
  17. 17.
    Schulman, R.R.: Disaster recovery issues and solutions. Hitachi Data Systems White Paper, p. 23 (2004)Google Scholar
  18. 18.
    Xie, J., Deng, Y., Min, G., Zhou, Y.: An incrementally scalable and cost-efficient interconnection structure for datacenters. IEEE Trans. Parallel Distrib. Syst. 28(6), 1578–1592 (2017)CrossRefGoogle Scholar
  19. 19.
    Kaczmarczyk, M., Barczynski, M., Kilian, W., Dubnicki, C.: Reducing impact of data fragmentation caused by in-line deduplication. In: Proceedings of the 5th Annual International Systems and Storage Conference, SYSTOR 2012, pp. 15:1–15:12 (2012)Google Scholar
  20. 20.
    Lillibridge, M., Eshghi, K., Bhagwat, D.: Improving restore speed for backup systems that use inline chunk-based deduplication. In: Proceedings of the 11th USENIX Conference on File and Storage Technologies, FAST 2013, pp. 183–198 (2013)Google Scholar
  21. 21.
    Fu, M., et al.: Accelerating restore and garbage collection in deduplication-based backup systems via exploiting historical information. In: Proceedings of the 2014 USENIX Conference on USENIX Annual Technical Conference, USENIX ATC 2014, pp. 181–192 (2014)Google Scholar
  22. 22.
    Srinivasan, K., Bisson, T., Goodson, G.R., Voruganti, K.: iDedup: latency-aware, inline data deduplication for primary storage. In: Proceedings of the 10th USENIX Conference on File and Storage Technologies, FAST 2012, vol. 12, pp. 1–14 (2012)Google Scholar
  23. 23.
    EMC: Achieving storage efficiency through EMC celerra data deduplication. White Paper (2010)Google Scholar
  24. 24.
    Adlercohen, C., Czarnowicki, T., Dreiher, J., Ruzicka, T., Ingber, A., Harari, M.: NetApp deduplication for FAS and V-series deployment and implementation guide. Technical report, vol. 2009, no. 1, pp. 141 753–141 753 (2011)Google Scholar
  25. 25.
    Min, F., et al.: Design tradeoffs for data deduplication performance in backup workloads. In: Proceedings of the 13th USENIX Conference on File and Storage Technologies, FAST 2015, pp. 331–344 (2015)Google Scholar
  26. 26.
    Belady, L.A.: A study of replacement algorithms for a virtual-storage computer. IBM Syst. J. 5(2), 78–101 (1966)CrossRefGoogle Scholar
  27. 27.
    Meister, D., Brinkmann, A., Süß, T.: File recipe compression in data deduplication systems. In: Proceedings of the 11th USENIX Conference on File and Storage Technologies, FAST 2013, pp. 175–182 (2013)Google Scholar
  28. 28.
    Agrawal, N., Bolosky, W.J., Douceur, J.R., Lorch, J.R.: A five-year study of file-system metadata. Trans. Storage 3(3), 9 (2007)CrossRefGoogle Scholar
  29. 29.
    Meyer, D.T., Bolosky, W.J.: A study of practical deduplication. Trans. Storage 7(4), 14:1–14:20 (2012)CrossRefGoogle Scholar
  30. 30.
    Rabin, M.: Fingerprinting by random polynomials (1981)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceJinan UniversityGuangzhouPeople’s Republic of China
  2. 2.State Key Laboratory of Computer Architecture, Institute of ComputingChinese Academy of SciencesBeijingChina
  3. 3.School of Information Science and TechnologyGuangdong University of Foreign StudiesGuangzhouChina

Personalised recommendations