Abstract
In this chapter, we show why data deduplication is important by stressing data explosion and large amounts of redundancies. We elaborate on current solutions (including storage data deduplication, redundancy elimination, information-centric networking) for data deduplication and the limitations of current solutions. We introduce a deduplication framework that optimizes data from clients to servers through networks. The framework consists of three components based on the level of deduplication. The client component removes local redundancies that occur in a client, the network component removes redundant transfers coming from different clients using redundancy elimination (RE) devices, and the server component eliminates redundancies coming from different networks. Then we show the evolution of data storage. Data storage has evolved from storage devices attached to a single computer (direct-attached storage) into storage devices attached to computer networks (storage area network and network-attached storage). We discuss the different kinds of storage devices and how they differ from one another. A redundant array of inexpensive disks (RAID), which improves storage access performance, is explained, and direct-attached storage (DAS), where storage is incorporated into a computer, is illustrated. We elaborate on storage area networks (SANs) and network-attached storage (NAS), where data from computers are transferred to storage devices through a dedicated network (SAN) or a general local area network used for sending and receiving application data (NAS). SAN and NAS consolidate and efficiently provide storage without wasting storage space compared to a DAS device. We describe a storage virtualization technique known as software-defined storage.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Carson, M., Yoder, A., Schoeb, L., Deel, D., Pratt, C.: Software defined storage. http://www.snia.org/sites/default/files/SNIA%20Software%20Defined%20Storage%20White%20Paper-%20v1.0k-DRAFT.pdf (2014)
Chadalapaka, M., Satran, J., Meth, K., Black, D.: Internet Small Computer System Interface (iSCSI) Protocol (Consolidated). http://www.rfc-editor.org/info/rfc7143 (2014)
Haynes, T., Noveck, D., Primary Data: Network File System (NFS) Version 4, External Data Representation Standard (XDR) Description. https://tools.ietf.org/html/rfc7531 (2015)
Hazelcast.org: Hazelcast. http://hazelcast.org/ (2016)
IBM: eXtremeScale. http://www-03.ibm.com/software/products/en/websphere-extreme-scale (2016)
IDC: The digital universe in 2020. https://www.emc.com/collateral/analyst-reports/idc-the-digital-universe-in-2020.pdf (2012)
Jin, W., Li, M., Khasnabish, B.: Content De-duplication for CDNi Optimization. https://tools.ietf.org/html/draft-jin-cdni-content-deduplication-optimization-04 (2013)
lustre.org: Lustre. http://lustre.org/ (2016)
Meyer, D.T., Bolosky, W.J.: A study of practical deduplication. In: Proceeding of the USENIX Conference on File and Storage Technologies (FAST) (2011)
Monia, C., Mullendore, R., Travostino, F., Jeong, W., Edwards, M.: iFCP - A Protocol for Internet Fibre Channel Storage Networking. http://www.rfc-editor.org/info/rfc4172 (2005)
openstack.org: OpenStack Swift. http://www.openstack.org/software/releases/liberty/components/swift (2016)
Oracle: Coherence. http://www.oracle.com/technetwork/middleware/coherence/overview/index.html (2016)
Patterson, D.A., Gibson, G., Katz, R.H.: A case for redundant arrays of inexpensive disks (raid). In: Proceedings of the 1988 ACM SIGMOD International Conference on Management of Data, SIGMOD ’88 (1988)
Serial_ATA: Fast Just Got Faster: SATA 6Gb/s. https://www.sata-io.org/system/files/member-downloads/SATA-6Gbs-Fast-Just-Got-Faster_2.pdf (2009)
Serial_ATA: SATA revision 3.2 specification. https://www.sata-io.org/sites/default/files/documents/SATA_v3%202_PR__Final_BusinessWire_8.20.13.pdf (2013)
Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., Noveck, D.: Network File System (NFS) Version 4 Protocol. http://www.rfc-editor.org/info/rfc7530 (2015)
Spring, N.T., Wetherall, D.: A protocol-independent technique for eliminating redundant network traffic. In: Proceedings of the ACM SIGCOMM 2000 conference on Data communication (2000)
Srinivasan, R.: RPC: Remote Procedure Call Protocol Specification Version 2. https://tools.ietf.org/html/rfc1831 (1995)
T10, I.T.C.: SCSI Architecture Model-2 (SAM-2). ANSI INCITS 366-2003, ISO/IEC 14776-412 (2003)
VMWare: Gemfire. https://www.vmware.com/support/pubs/vfabric-gemfire.html (2016)
Weber, R., Rajagopal, M., Travostino, F., O’Donnell, M., Monia, C., Merhar, M.: Fibre Channel (FC) Frame Encapsulation. http://www.rfc-editor.org/info/rfc3643 (2003)
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Kim, D., Song, S., Choi, BY. (2017). Introduction. In: Data Deduplication for Data Optimization for Storage and Network Systems. Springer, Cham. https://doi.org/10.1007/978-3-319-42280-0_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-42280-0_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-42278-7
Online ISBN: 978-3-319-42280-0
eBook Packages: EngineeringEngineering (R0)