Redundant Independent Files (RIF): A Technique for Reducing Storage and Resources in Big Data Replication

  • Mostafa R. Kaseb
  • Mohamed H. Khafagy
  • Ihab A. Ali
  • ElSayed M. Saad
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 745)

Abstract

Most of cloud computing storage systems widely use a distributed file system (DFS) to store big data, such as Hadoop Distributed File System (HDFS) and Google File System (GFS). Therefore, the DFS depends on replicate data and stores it as multiple copies, to achieve high reliability and availability. On the other hand, that technique increases storage and resources consumption.

This paper addresses these issues by presenting a decentralized hybrid model. That model; called CPRIF, is a combination of a cloud provider (CP) and a suggested service that we call Redundant Independent Files (RIF). The CP provides HDFS without replica, and the RIF acts as a service layer that splits data into three parts and uses the XOR operation to generate a fourth part as parity. These four parts are to be stored in HDFS files as independent files on CP. The generated parity file not only guarantees the security and reliability of data but also reduces storage space, resources consumption and operational costs. It also improved the writing and reading performance.

The suggested model was implemented on a cloud computing storage that we built using three physical servers (Dell T320) running a total 12 virtual nodes. The TeraGen benchmark tool and Java Code were used to test the model. Implemented results show the suggested model decreased the storage space by 35% compared to other models and improved the data writing and reading by about 34%.

Keywords

Big data Cloud storage Cloud computing Cloud provider Google File System (GFS) Hadoop distributed file system (HDFS) 

References

  1. 1.
    Patel, Y.S., Mehrotra, N., Soner, S.: Green cloud computing: a review on green IT areas for cloud computing environment. In: IEEE 1st International Conference on Futuristic trend in Computational Analysis and Knowledge Management, pp. 327–332 (2015)Google Scholar
  2. 2.
    Nair, M.K., Gopalakrishna, D.V.: Generic web services: a step towards green computing. Int. J. Comput. Sci. Eng. 1, 248–253 (2009)Google Scholar
  3. 3.
    Asadianfam, S., Shamsi, M., Kashany, S.: A review distributed file system. Int. J. Comput. Netw. Commun. Secur. 3(5), 229–234 (2015)Google Scholar
  4. 4.
    Krishna, T.L.S.R., Ragunathan, T., Battula, S.K.: Customized web user interface for hadoop distributed file system. In: Proceedings of the Second International Conference on Computer and Communication Technologies, 04 September 2015Google Scholar
  5. 5.
    Ghemawat, S., Gobioff, H., Leung, S.: The Google file system. In: Proceedings of ACM Symposium on Operating Systems Principles, Lake George, NY, pp. 29–43, October 2003Google Scholar
  6. 6.
    The Apache Hadoop Project. https://hadoop.apache.org/. Accessed 17 Nov 2017
  7. 7.
    Shvachko, K., Kuang, H., Radia, S.: The Hadoop distributed file system. In: Proceedings of the 10th IEEE Symposium on Mass Storage Systems and Technologies, MSST 2010, pp. 1–10 (2010)Google Scholar
  8. 8.
    Carns, P.H., Ligon III, W.B., Ross, R.B., Thakur, R.: PVFS: a parallel file system for Linux clusters. In: Proceedings of 4th Annual Linux Showcase and Conference, pp. 317–327 (2000)Google Scholar
  9. 9.
    Braam, P.J.: The Lustre storage architecture. Cluster File Systems, Inc., August 2004. http://www.lustre.org/documentation.html
  10. 10.
    Wu, S., Zhu, W., Mao, B., Li, K.-C.: PP: popularity-based proactive data recovery for HDFS RAID systems. Future Generation Computer Systems (2017)Google Scholar
  11. 11.
    Abead, E.S., Khafagy, M.H., Omara, F.A.: An efficient replication technique for hadoop distributed file system. Int. J. Sci. Eng. Res. 7(1), 254–261 (2016)Google Scholar
  12. 12.
    Patel Neha, M., Patel Narendra, M., Hasan, M.I., Shah Parth, D., Patel Mayur, M.: Improving HDFS write performance using efficient replica placement. In: 2014 5th International Conferences - Confluence The Next Generation Information Technology Summit (Confluence), pp. 36–39 (2014)Google Scholar
  13. 13.
    Li, J., Zhang, P., Li, Y., Chen, W., Liu, Y., Wang, L.: A data-check based distributed storage model for storing hot temporary data. Future Generation Comp. Syst. 73, 13–21 (2017)CrossRefGoogle Scholar
  14. 14.
    Thomasian, A.: Multi-level raid for very large disk arrays. ACM SIGMETRICS Perform. Eval. Rev. 33(4), 17–22 (2006)CrossRefGoogle Scholar
  15. 15.

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Mostafa R. Kaseb
    • 1
  • Mohamed H. Khafagy
    • 1
  • Ihab A. Ali
    • 2
  • ElSayed M. Saad
    • 2
  1. 1.Faculty of Computers and InformationFayoum UniversityFayoumEgypt
  2. 2.Faculty of EngineeringHelwan UniversityHelwanEgypt

Personalised recommendations