I/O Optimizations Based on Workload Characteristics for Parallel File Systems

  • Bing Wei
  • Limin XiaoEmail author
  • Bingyu Zhou
  • Guangjun QinEmail author
  • Baicheng Yan
  • Zhisheng Huo
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11783)


Parallel file systems usually provide a unified storage solution, which fails to meet specific application needs. In this paper, we propose an extended file handle scheme to address this problem. It allows the file systems to specify optimizations for individual file or directory based on workload characteristics. One case study shows that our proposed approach improves the aggregate throughput of large files and small files by up to 5% and 30%, respectively. To further improve the access performance of small files in parallel file systems, we also propose a new metadata-based small file optimization method. The experimental results show that the aggregate throughput of small files can be effectively improved through our method.


Parallel file systems Workload characteristics Extended file handle Small file optimizations 



This work was supported by the National key R&D Program of China under Grant NO. 2017YFB1010000, the National Natural Science Foundation of China under Grant No. 61772053, the Science Challenge Project, No. TZ2016002, and the fund of the State Key Laboratory of Software Development Environment under Grant No. SKLSDE-2017ZX-10.


  1. 1.
    Ross, R.B., Thakur, R.: PVFS: a parallel file system for Linux clusters. In: Proceedings of the 4th Annual Linux Showcase and Conference, pp. 391–430 (2000)Google Scholar
  2. 2.
    Isaila, F.: Collective I/O tuning using analytical and machine learning models. In: 2015 IEEE International Conference on Cluster Computing. pp. 128–137. IEEE (2015)Google Scholar
  3. 3.
    Zhang, S., Catanese, H.: The composite-file file system: decoupling the one-to-one mapping of files and metadata for better performance. In: 14th USENIX Conference on File and Storage Technologies. pp. 15–22 (2016)Google Scholar
  4. 4.
    Byna, S., Chen, Y.: Parallel I/O prefetching using MPI file caching and I/O signatures. In: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing. pp. 44. IEEE (2008)Google Scholar
  5. 5.
    Carns, P., Lang, S.: Small-le access in parallel le systems. IEEE IPDPS 2009, 1–11 (2009)Google Scholar
  6. 6.
    Uysal, M., Acharya, A.: Requirements of I/O systems for parallel machines: An application-driven study (1998)Google Scholar
  7. 7.
    Kuhn, M., Kunkel, J.M.: Dynamic le system semantics to enable metadata optimizations in PVFS. Concurr. Comput. Pract. Exper. 21(14), 1775–1788 (2009)CrossRefGoogle Scholar
  8. 8.
    LNCS Homepage. Accessed 16 May 2019

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  1. 1.State Key Laboratory of Software Development EnvironmentBeihang UniversityBeijingChina
  2. 2.School of Computer Science and EngineeringBeihang UniversityBeijingChina

Personalised recommendations