SAFE: Structure-Aware File and Email Deduplication for Cloud-Based Storage Systems

Abstract

In this chapter, we introduce Structure-Aware File and Email Deduplication for Cloud-based Storage Systems (SAFE), a client-based deduplication technique that is fast and provides the same space savings as variable-size block deduplication using structure-based granularity rather than physical chunk granularity for cloud-based storage. Cloud-based storage, including Dropbox (http://www.dropbox.com), JustCloud (http://www.justcloud.com/), and Mozy (http://mozy.com/), has become popular because people can access data at any time, anywhere and using various types of devices such as laptops, tablets and smartphones. Cloud-based storage services use deduplication techniques to avoid sending and storing duplicate files (or blocks), reducing network bandwidth and storage space, which gives the subsequent benefit of data upload speed. Existing deduplication techniques (file-level and fixed-size block deduplication) that cloud-based storage uses are fast and have low index overhead but find fewer redundancies than variable-size block deduplication. However, owing to excessive CPU and memory overhead from chunking, indexing and fragmentation, variable-size block deduplication cannot be used for cloud-based storage. Thus, we developed SAFE, Structure-Aware File and Email Deduplication, that achieves both fast speeds and shows good space savings in clients by using structure-based granularity for cloud-based storage systems. Evaluation results show that SAFE has as good storage space savings as existing variable-size block deduplication while being as fast as file-level or a large fixed-size block deduplication technique.