Abstract
New experimental and AI-driven workloads are moving into the realm of extreme-scale HPC systems at the same time that high-performance flash is becoming cost-effective to deploy at scale. This confluence poses a number of new technical and economic challenges and opportunities in designing the next generation of HPC storage and I/O subsystems to achieve the right balance of bandwidth, latency, endurance, and cost. In this work, we present quantitative models that use workload data from existing, disk-based file systems to project the architectural requirements of all-flash Lustre file systems. Using data from NERSC’s Cori I/O subsystem, we then demonstrate the minimum required capacity for data, capacity for metadata and data-on-MDT, and SSD endurance for a future all-flash Lustre file system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We do not consider write amplification caused by garbage collection internal to the SSDs since drive endurance is warranted on the basis of host-initiated write load, not total write load to NAND.
- 2.
Strictly speaking, we define \(C^\text {inode}\) to include the MDT block allocated for inodes and additional data blocks that may be required to store, for example, large numbers of directory entries.
References
APEX Workflows Whitepaper. Tech. Rep., Los Alamos National Laboratory, Lawrence Berkeley National Laboratory, and Sandia National Laboratories (2016). https://www.nersc.gov/assets/apex-workflows-v2.pdf
Intel SSD Data Center Tool (2017). https://www.intel.com/content/www/us/en/support/articles/000006289
Alewijnse, B., et al.: Best practices for managing large CryoEM facilities. J. Struct. Biol. 199(3), 225–236 (2017). https://doi.org/10.1016/j.jsb.2017.07.011. https://linkinghub.elsevier.com/retrieve/pii/S1047847717301314
Austin, B., et al.: A metric for evaluating supercomputer performance in the era of extreme Heterogeneity. In: 2018 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pp. 63–71. IEEE (November 2018). https://doi.org/10.1109/PMBS.2018.8641549, https://ieeexplore.ieee.org/document/8641549/
Bhimji, W., et al.: Extreme I/O on HPC for HEP using the burst buffer at NERSC. J. Phys. Conf. Ser. 898, 082015 (2017). https://doi.org/10.1088/1742-6596/898/8/082015. https://iopscience.iop.org/article/10.1088/1742-6596/898/8/082015
Bhimji, W., et al.: Accelerating science with the NERSC burst buffer early user program. In: Proceedings of the 2016 Cray User Group, London (2016). https://cug.org/proceedings/cug2016_proceedings/includes/files/pap162.pdf
Daley, C.S., Ghoshal, D., Lockwood, G.K., Dosanjh, S., Ramakrishnan, L., Wright, N.J.: Performance characterization of scientific workflows for the optimal use of burst buffers. In: Future Generation Computer Systems (December 2017). https://doi.org/10.1016/j.future.2017.12.022, http://linkinghub.elsevier.com/retrieve/pii/S0167739X16308287
Declerck, T.M.: Using Robinhood to purge data from Lustre file systems. In: Proceedings of the 2014 Cray User Group, Lugano, CH (2014). https://cug.org/proceedings/cug2014_proceedings/includes/files/pap157.pdf
Fontana, R.E., Decad, G.M.: Moore’s law realities for recording systems and memory storage components: HDD, tape, NAND, and optical. AIP Adv. 8(5), 056506 (2018). https://doi.org/10.1063/1.5007621. http://aip.scitation.org/doi/10.1063/1.5007621
Gunasekaran, R., Oral, S., Hill, J., Miller, R., Wang, F., Leverman, D.: Comparative I/O workload characterization of two leadership class storage clusters. In: Proceedings of the 10th Parallel Data Storage Workshop (PDSW 2015), pp. 31–36. ACM Press, New York (2015). https://doi.org/10.1145/2834976.2834985, http://dl.acm.org/citation.cfm?doid=2834976.2834985
Hemmert, K.S., et al.: Trinity: architecture and early experience. In: Proceedings of the 2017 Cray User Group (2017)
Bent, J., Settlemeyer, B., Grider, G.: Serving data to the lunatic fringe: the evolution of HPC storage. Login 41(2), 34–39 (2016). https://www.usenix.org/publications/login/summer2016/bent
Joubert, W., et al.: Attacking the opioid epidemic: determining the epistatic and pleiotropic genetic architectures for chronic pain and opioid addiction. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, pp. 57:1–57:14, SC 2018. IEEE Press, Piscataway (2018). http://dl.acm.org/citation.cfm?id=3291656.3291732
Kurth, T., et al.: Exascale deep learning for climate analytics. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, pp. 51:1–51:12, SC 2018. IEEE Press, Piscataway (2018). http://dl.acm.org/citation.cfm?id=3291656.3291724, arXiv:1810.01993
Lockwood, G.K., et al.: Storage 2020: a vision for the future of HPC storage. Tech. rep., Lawrence Berkeley National Laboratory, Berkeley (2017). https://escholarship.org/uc/item/744479dp
Lockwood, G.K., Wagner, R., Tatineni, M.: Storage utilization in the long tail of science. In: Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure (2015). https://doi.org/10.1145/2792745.2792777, http://dl.acm.org/citation.cfm?id=2792777
Regier, J., et al.: Cataloging the visible universe through Bayesian inference at petascale. In: 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 44–53 (May 2018). https://doi.org/10.1109/IPDPS.2018.00015
Standish, K.A., et al.: Group-based variant calling leveraging next-generation supercomputing for large-scale whole-genome sequencing studies. BMC Bioinf. 16(1), 304 (2015). https://doi.org/10.1186/s12859-015-0736-4. http://www.biomedcentral.com/1471-2105/16/304
Strande, S.M., et al.: Gordon: design, performance, and experiences deploying and supporting a data intensive supercomputer. In: Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment: Bridging from the eXtreme to the Campus and Beyond, pp. 3:1–3:8, XSEDE 2012. ACM, New York (2012). https://doi.org/10.1145/2335755.2335789
Thayer, J., et al.: Data systems for the linac coherent light source. J. Appl. Crystallogr. 49(4), 1363–1369 (2016). https://doi.org/10.1107/S1600576716011055. http://scripts.iucr.org/cgi-bin/paper?S1600576716011055
Uselton, A.: Deploying server-side file system monitoring at NERSC. In: Proceedings of the 2009 Cray User Group (2009)
Vazhkudai, S.S., et al.: GUIDE: a scalable information directory service to collect, federate, and analyze logs for operational insights into a leadership HPC facility. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC 2017, pp. 1–12 (2017). https://doi.org/10.1145/3126908.3126946, http://dl.acm.org/citation.cfm?doid=3126908.3126946
Wang, F., Sim, H., Harr, C., Oral, S.: Diving into petascale production file systems through large scale profiling and analysis. In: Proceedings of the 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems - PDSW-DISCS 2017, pp. 37–42. ACM Press, New York (2017). https://doi.org/10.1145/3149393.3149399, http://dl.acm.org/citation.cfm?doid=3149393.3149399
Weeks, N.T., Luecke, G.R.: Optimization of SAMtools sorting using OpenMP tasks. Cluster Comput. 20(3), 1869–1880 (2017). https://doi.org/10.1007/s10586-017-0874-8
Acknowledgments
The authors would like to thank John Bent, Andreas Dilger, and the anonymous reviewers for their valuable feedback on this work. This material is based upon work supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-05CH11231. This research used resources and data generated from resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Lockwood, G.K., Lozinskiy, K., Gerhardt, L., Cheema, R., Hazen, D., Wright, N.J. (2019). A Quantitative Approach to Architecting All-Flash Lustre File Systems. In: Weiland, M., Juckeland, G., Alam, S., Jagode, H. (eds) High Performance Computing. ISC High Performance 2019. Lecture Notes in Computer Science(), vol 11887. Springer, Cham. https://doi.org/10.1007/978-3-030-34356-9_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-34356-9_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-34355-2
Online ISBN: 978-3-030-34356-9
eBook Packages: Computer ScienceComputer Science (R0)