SP-TSRM: A Data Grouping Strategy in Distributed Storage System
With the development of smart devices and social media, massive unstructured data is uploaded to distributed storage systems. Since the characteristics of multi-users and high concurrency the unstructured data accesses have, it brings new challenges to traditional distributed storage systems designed for large files. We propose a grouping strategy to analyze relevant data in access according to disk access logs in the real distributed storage systems environment. When any data in the group is accessed, the whole group is prefetched from disk to the cache. Firstly, we conduct statistical analysis on the access logs and propose a preliminary classification method to classify files in spatiotemporal locality. Secondly, a strength-priority tree structure relation model (SP-TSRM) is proposed to mine file group efficiently. Finally, experiments show that the proposed model can improve the cache hit rate significantly, thereby improving the read efficiency of distributed storage systems.
KeywordsPrefetching model Storage optimization Unstructured data Distributed storage systems
Thanks to the students of HIT at Weihai. This work is supported by project under Grant no. 520613170002, SGSDWH00YXJS1700522, SGSDWH00YXJS1700270, the Fundamental Research Funds for the Central Universities (Grant No. HIT.NSRIF.201714), Weihai Science and Technology Development Program (2016DXGJMS15) and Key Research and Development Program in Shandong Provincial (2017GGX90103).
- 2.Zhu, Y., Zhang, X., Zhao, R., Dong, X.: Data De-duplication on similar file detection. In: Innovative Mobile and Internet Services in Ubiquitous Computing, pp. 66–73. IEEE Press (2014)Google Scholar
- 4.Dong, B., Qiu, J., Zheng, Q., Zhong, X., Li, J., Li, Y.: A novel approach to improving the efficiency of storing and accessing small files on hadoop: a case study by powerpoint files. In: Services Computing (SCC), pp. 65–72. IEEE Press (2010)Google Scholar
- 5.Bok, K., Lim, J., Oh, H., Yoo, J.: An efficient cache management scheme for accessing small files in distributed file systems. In: Big Data and Smart Computing (BigComp), pp. 151–155. IEEE Press (2017)Google Scholar
- 6.Lin, L., Li, X., Jiang, H., Zhu, Y., Tian, L., AMP: an affinity-based metadata prefetching scheme in large-scale distributed storage systems. In: Cluster Computing and the Grid, pp. 459–466. IEEE Press (2008)Google Scholar
- 8.Cherubini, G., Kim, Y., Lantz, M., Venkatesan, V.: Data prefetching for large tiered storage systems. In: Data Mining (ICDM), pp. 823–828. IEEE Press (2017)Google Scholar
- 9.Kroeger, T.M., Long, D.D., Mogul, J.C.: Exploring the bounds of web latency reduction from caching and prefetching. In: USENIX Symposium on Internet Technologies and Systems, pp. 13–22 (1997)Google Scholar
- 10.Wildani, A., Miller, E.L.: Can we group storage? Statistical techniques to identify predictive groupings in storage system accesses. ACM Trans. Storage (TOS) 12(2), 7–40 (2016)Google Scholar