A Quantitative Approach to Architecting All-Flash Lustre File Systems

  • Glenn K. LockwoodEmail author
  • Kirill Lozinskiy
  • Lisa Gerhardt
  • Ravi Cheema
  • Damian Hazen
  • Nicholas J. Wright
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11887)


New experimental and AI-driven workloads are moving into the realm of extreme-scale HPC systems at the same time that high-performance flash is becoming cost-effective to deploy at scale. This confluence poses a number of new technical and economic challenges and opportunities in designing the next generation of HPC storage and I/O subsystems to achieve the right balance of bandwidth, latency, endurance, and cost. In this work, we present quantitative models that use workload data from existing, disk-based file systems to project the architectural requirements of all-flash Lustre file systems. Using data from NERSC’s Cori I/O subsystem, we then demonstrate the minimum required capacity for data, capacity for metadata and data-on-MDT, and SSD endurance for a future all-flash Lustre file system.


Architecture Lustre Flash 



The authors would like to thank John Bent, Andreas Dilger, and the anonymous reviewers for their valuable feedback on this work. This material is based upon work supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-05CH11231. This research used resources and data generated from resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.


  1. 1.
    APEX Workflows Whitepaper. Tech. Rep., Los Alamos National Laboratory, Lawrence Berkeley National Laboratory, and Sandia National Laboratories (2016).
  2. 2.
  3. 3.
    Alewijnse, B., et al.: Best practices for managing large CryoEM facilities. J. Struct. Biol. 199(3), 225–236 (2017). Scholar
  4. 4.
    Austin, B., et al.: A metric for evaluating supercomputer performance in the era of extreme Heterogeneity. In: 2018 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pp. 63–71. IEEE (November 2018).,
  5. 5.
    Bhimji, W., et al.: Extreme I/O on HPC for HEP using the burst buffer at NERSC. J. Phys. Conf. Ser. 898, 082015 (2017). Scholar
  6. 6.
    Bhimji, W., et al.: Accelerating science with the NERSC burst buffer early user program. In: Proceedings of the 2016 Cray User Group, London (2016).
  7. 7.
    Daley, C.S., Ghoshal, D., Lockwood, G.K., Dosanjh, S., Ramakrishnan, L., Wright, N.J.: Performance characterization of scientific workflows for the optimal use of burst buffers. In: Future Generation Computer Systems (December 2017).,
  8. 8.
    Declerck, T.M.: Using Robinhood to purge data from Lustre file systems. In: Proceedings of the 2014 Cray User Group, Lugano, CH (2014).
  9. 9.
    Fontana, R.E., Decad, G.M.: Moore’s law realities for recording systems and memory storage components: HDD, tape, NAND, and optical. AIP Adv. 8(5), 056506 (2018). Scholar
  10. 10.
    Gunasekaran, R., Oral, S., Hill, J., Miller, R., Wang, F., Leverman, D.: Comparative I/O workload characterization of two leadership class storage clusters. In: Proceedings of the 10th Parallel Data Storage Workshop (PDSW 2015), pp. 31–36. ACM Press, New York (2015).,
  11. 11.
    Hemmert, K.S., et al.: Trinity: architecture and early experience. In: Proceedings of the 2017 Cray User Group (2017)Google Scholar
  12. 12.
    Bent, J., Settlemeyer, B., Grider, G.: Serving data to the lunatic fringe: the evolution of HPC storage. Login 41(2), 34–39 (2016). Scholar
  13. 13.
    Joubert, W., et al.: Attacking the opioid epidemic: determining the epistatic and pleiotropic genetic architectures for chronic pain and opioid addiction. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, pp. 57:1–57:14, SC 2018. IEEE Press, Piscataway (2018).
  14. 14.
    Kurth, T., et al.: Exascale deep learning for climate analytics. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, pp. 51:1–51:12, SC 2018. IEEE Press, Piscataway (2018)., arXiv:1810.01993
  15. 15.
    Lockwood, G.K., et al.: Storage 2020: a vision for the future of HPC storage. Tech. rep., Lawrence Berkeley National Laboratory, Berkeley (2017).
  16. 16.
    Lockwood, G.K., Wagner, R., Tatineni, M.: Storage utilization in the long tail of science. In: Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure (2015).,
  17. 17.
    Regier, J., et al.: Cataloging the visible universe through Bayesian inference at petascale. In: 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 44–53 (May 2018).
  18. 18.
    Standish, K.A., et al.: Group-based variant calling leveraging next-generation supercomputing for large-scale whole-genome sequencing studies. BMC Bioinf. 16(1), 304 (2015). Scholar
  19. 19.
    Strande, S.M., et al.: Gordon: design, performance, and experiences deploying and supporting a data intensive supercomputer. In: Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment: Bridging from the eXtreme to the Campus and Beyond, pp. 3:1–3:8, XSEDE 2012. ACM, New York (2012).
  20. 20.
    Thayer, J., et al.: Data systems for the linac coherent light source. J. Appl. Crystallogr. 49(4), 1363–1369 (2016). Scholar
  21. 21.
    Uselton, A.: Deploying server-side file system monitoring at NERSC. In: Proceedings of the 2009 Cray User Group (2009)Google Scholar
  22. 22.
    Vazhkudai, S.S., et al.: GUIDE: a scalable information directory service to collect, federate, and analyze logs for operational insights into a leadership HPC facility. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC 2017, pp. 1–12 (2017).,
  23. 23.
    Wang, F., Sim, H., Harr, C., Oral, S.: Diving into petascale production file systems through large scale profiling and analysis. In: Proceedings of the 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems - PDSW-DISCS 2017, pp. 37–42. ACM Press, New York (2017).,
  24. 24.
    Weeks, N.T., Luecke, G.R.: Optimization of SAMtools sorting using OpenMP tasks. Cluster Comput. 20(3), 1869–1880 (2017). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Glenn K. Lockwood
    • 1
    Email author
  • Kirill Lozinskiy
    • 1
  • Lisa Gerhardt
    • 1
  • Ravi Cheema
    • 1
  • Damian Hazen
    • 1
  • Nicholas J. Wright
    • 1
  1. 1.Lawrence Berkeley National LaboratoryBerkeleyUSA

Personalised recommendations