Redundant Independent Files (RIF): A Technique for Reducing Storage and Resources in Big Data Replication
Most of cloud computing storage systems widely use a distributed file system (DFS) to store big data, such as Hadoop Distributed File System (HDFS) and Google File System (GFS). Therefore, the DFS depends on replicate data and stores it as multiple copies, to achieve high reliability and availability. On the other hand, that technique increases storage and resources consumption.
This paper addresses these issues by presenting a decentralized hybrid model. That model; called CPRIF, is a combination of a cloud provider (CP) and a suggested service that we call Redundant Independent Files (RIF). The CP provides HDFS without replica, and the RIF acts as a service layer that splits data into three parts and uses the XOR operation to generate a fourth part as parity. These four parts are to be stored in HDFS files as independent files on CP. The generated parity file not only guarantees the security and reliability of data but also reduces storage space, resources consumption and operational costs. It also improved the writing and reading performance.
The suggested model was implemented on a cloud computing storage that we built using three physical servers (Dell T320) running a total 12 virtual nodes. The TeraGen benchmark tool and Java Code were used to test the model. Implemented results show the suggested model decreased the storage space by 35% compared to other models and improved the data writing and reading by about 34%.
KeywordsBig data Cloud storage Cloud computing Cloud provider Google File System (GFS) Hadoop distributed file system (HDFS)
- 1.Patel, Y.S., Mehrotra, N., Soner, S.: Green cloud computing: a review on green IT areas for cloud computing environment. In: IEEE 1st International Conference on Futuristic trend in Computational Analysis and Knowledge Management, pp. 327–332 (2015)Google Scholar
- 2.Nair, M.K., Gopalakrishna, D.V.: Generic web services: a step towards green computing. Int. J. Comput. Sci. Eng. 1, 248–253 (2009)Google Scholar
- 3.Asadianfam, S., Shamsi, M., Kashany, S.: A review distributed file system. Int. J. Comput. Netw. Commun. Secur. 3(5), 229–234 (2015)Google Scholar
- 4.Krishna, T.L.S.R., Ragunathan, T., Battula, S.K.: Customized web user interface for hadoop distributed file system. In: Proceedings of the Second International Conference on Computer and Communication Technologies, 04 September 2015Google Scholar
- 5.Ghemawat, S., Gobioff, H., Leung, S.: The Google file system. In: Proceedings of ACM Symposium on Operating Systems Principles, Lake George, NY, pp. 29–43, October 2003Google Scholar
- 6.The Apache Hadoop Project. https://hadoop.apache.org/. Accessed 17 Nov 2017
- 7.Shvachko, K., Kuang, H., Radia, S.: The Hadoop distributed file system. In: Proceedings of the 10th IEEE Symposium on Mass Storage Systems and Technologies, MSST 2010, pp. 1–10 (2010)Google Scholar
- 8.Carns, P.H., Ligon III, W.B., Ross, R.B., Thakur, R.: PVFS: a parallel file system for Linux clusters. In: Proceedings of 4th Annual Linux Showcase and Conference, pp. 317–327 (2000)Google Scholar
- 9.Braam, P.J.: The Lustre storage architecture. Cluster File Systems, Inc., August 2004. http://www.lustre.org/documentation.html
- 10.Wu, S., Zhu, W., Mao, B., Li, K.-C.: PP: popularity-based proactive data recovery for HDFS RAID systems. Future Generation Computer Systems (2017)Google Scholar
- 11.Abead, E.S., Khafagy, M.H., Omara, F.A.: An efficient replication technique for hadoop distributed file system. Int. J. Sci. Eng. Res. 7(1), 254–261 (2016)Google Scholar
- 12.Patel Neha, M., Patel Narendra, M., Hasan, M.I., Shah Parth, D., Patel Mayur, M.: Improving HDFS write performance using efficient replica placement. In: 2014 5th International Conferences - Confluence The Next Generation Information Technology Summit (Confluence), pp. 36–39 (2014)Google Scholar