Skip to main content
SpringerLink
Log in

Analyzing IO Usage Patterns of User Jobs to Improve Overall HPC System Efficiency

  • Conference paper
  • First Online:
Book cover Software Challenges to Exascale Computing (SCEC 2018)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 964))

Included in the following conference series:

  • 360 Accesses

Abstract

This work looks at analyzing I/O traffic of users’ jobs on a HPC machine for a period of time. Monitoring tools are collecting the data in a continuous basis on the HPC system. We looked at aggregate I/O data usage patterns of users’ jobs on the system both on the parallel shared Lustre file system and the node-local SSDs. Data mining tools are then applied to analyze the I/O usage pattern data in an attempt to tie the data to particular codes that produced those I/O behaviors from users’ jobs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Comet User Guide. http://www.sdsc.edu/support/user_guides/comet.html

  2. Hammond, J.: TACC stats: I/O performance monitoring for the instransigent. In: Invited Keynote for the 3rd IASDS Workshop, pp. 1–29 (2011)

    Google Scholar 

  3. http://www.nersc.gov/users/computational-systems/cori/nersc-8-procurement/trinity-nersc-8-rfp/nersc-8-trinity-benchmarks/ior/

  4. Ester, M., Kriegel, H.-P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Simoudis, E., Han, J., Fayyad, U.M. (eds.) Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD-1996), pp. 226–231. AAAI Press (1996)

    Google Scholar 

  5. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. JMLR 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  6. Kriegel, H.-P., Kroeger, P., Sander, J., Zimek, A.: Density-based clustering. WIREs Data Min. Knowl. Discov. 1(3), 231–240 (2011)

    Article  Google Scholar 

Download references

Acknowledgement

Authors acknowledge funding support/sponsorship from Engility Corporation’s High Performance Computing Center of Excellence (HPC CoE) that was used to support the student research. Authors thank Dr. Rajiv Bendale, Engility Corporation, for many valuable suggestions for this project.

Author information

Authors and Affiliations

  1. San Diego Supercomputer Center, University of California San Diego, La Jolla, CA, 92093, USA

    Syed Sadat Nazrul, Cherie Huang, Mahidhar Tatineni, Nicole Wolter, Dmitry Mishin, Trevor Cooper & Amit Majumdar

Authors
  1. Syed Sadat Nazrul
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cherie Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mahidhar Tatineni
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nicole Wolter
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dmitry Mishin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Trevor Cooper
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Amit Majumdar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amit Majumdar .

Editor information

Editors and Affiliations

  1. San Diego Supercomputer Center, University of California, San Diego, La Jolla, CA, USA

    Amit Majumdar

  2. Texas Advanced Computing Center, The University of Texas at Austin, Austin, TX, USA

    Ritu Arora

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nazrul, S.S. et al. (2019). Analyzing IO Usage Patterns of User Jobs to Improve Overall HPC System Efficiency. In: Majumdar, A., Arora, R. (eds) Software Challenges to Exascale Computing. SCEC 2018. Communications in Computer and Information Science, vol 964. Springer, Singapore. https://doi.org/10.1007/978-981-13-7729-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-7729-7_7

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-7728-0

  • Online ISBN: 978-981-13-7729-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation