Skip to main content
SpringerLink
Log in

Cost and Performance Modeling for Earth System Data Management and Beyond

  • Conference paper
  • First Online:
High Performance Computing (ISC High Performance 2018)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11203))

Included in the following conference series:

Abstract

Current and anticipated storage environments confront domain scientist and data center operators with usability, performance and cost challenges. The amount of data upcoming system will be required to handle is expected to grow exponentially, mainly due to increasing resolution and affordable compute power. Unfortunately, the relationship between cost and performance is not always well understood requiring considerable effort for educated procurement. Within the Centre of Excellence in Simulation of Weather and Climate in Europe (ESiWACE) models to better understand cost and performance of current and future systems are being explored. This paper presents models and methodology focusing on, but not limited to, data centers used in the context of climate and numerical weather prediction. The paper concludes with a case study of alternative deployment strategies and outlines the challenges anticipating their impact on cost and performance. By publishing these early results, we would like to make the case to work towards standard models and methodologies collaboratively as a community to create sufficient incentives for vendors to provide specifications in formats which are compatible to these modeling tools. In addition to that, we see application for such formalized models and information in I/O related middleware, which are expected to make automated but reasonable decisions in increasingly heterogeneous data centers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.vi4io.org.

References

  1. Performance Evaluation of the PVFS2 Architecture. Napoli, Italy

    Google Scholar 

  2. SST Simulator - The Structural Simulation Toolkit. http://sst-simulator.org/

  3. Arjona, J.O.: Using UML state diagrams for modelling the performance of parallel programs. Computación y Sistemas 11(3), 199–210 (2008)

    Google Scholar 

  4. Carothers, C.: ROSS: rensselaer’s optimistic simulation system, November 2017. https://github.com/carothersc/ROSS

  5. DEEP Projects. http://www.deep-projects.eu/

  6. ESiWACE: Centre of excellence in simulation of weather and climate in Europe. https://www.esiwace.eu/

  7. ExtremeEarth. http://www.extremeearth.eu/

  8. Fontana, R.E., Decad, G.M., Hetzler, S.R.: The impact of areal density and millions of square inches (MSI) of produced memory on petabyte shipments of TAPE, NAND flash, and HDD storage class memories. In: 2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–8. IEEE (2013). http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6558421

  9. HPSS collaboration: list of sites (2018). http://www.hpss-collaboration.org/customersT.shtml

  10. Intel, T.: HDF Group, EMC, Cray: Fast Forward Storage and I/O, June 2014

    Google Scholar 

  11. Luettgau, J., Kunkel, J., Jensen, J., Lawrence, B.: ESIWACE D4.1 business model with alternative scenarios. Technical report. https://www.esiwace.eu/results/deliverables/d4-1-business-model-with-alternative-scenarios

  12. Kunkel, J.M., Ludwig, T.: IOPm - modeling the I/O path with a functional representation of parallel file system and hardware architecture. In: 20th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (2012). https://doi.org/10.1109/PDP.2012.13

  13. Luettgau, J., Kunkel, J.: Simulation of hierarchical storage systems for TCO and QoS. In: Kunkel, J.M., Yokota, R., Taufer, M., Shalf, J. (eds.) High Performance Computing, pp. 132–144. Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3-319-67630-2_12

    Chapter  Google Scholar 

  14. Mubarak, M., Carothers, C.D., Ross, R., Carns, P.: Modeling a million-node dragonfly network using massively parallel discrete-event simulation. In: 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, pp. 366–376. November 2012. https://doi.org/10.1109/SC.Companion.2012.56

  15. NEXTGenIO: next generation I/O for the exascale. http://www.nextgenio.eu/

  16. Overpeck, J.T., Meehl, G.A., Bony, S., Easterling, D.R.: Climate data challenges in the 21st century. Science 331(6018), 700–702 (2011)

    Article  Google Scholar 

  17. Llopis, P., Dolz, M.F., Blas, J.G., Isaila, F., Heidari, M.R., Kuhn, M.: Analyzing the energy consumption of the storage data path. J. Supercomput. 72(11), 4089–4106 (2016). https://doi.org/10.1007/s11227-016-1729-4

    Article  Google Scholar 

  18. Pentzaropoulos, G.: Computer performance modelling: an overview. Appl. Math. Model. 6(2), 74–80 (1982)

    Article  Google Scholar 

  19. Pereverzeva, I., Laibinis, L., Troubitsyna, E., Holmberg, M., Pöri, M.: Formal modelling of resilient data storage in cloud. In: Groves, L., Sun, J. (eds.) ICFEM 2013. LNCS, vol. 8144, pp. 363–379. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-41202-8_24

    Chapter  Google Scholar 

  20. Tribastone, M., Gilmore, S.: Automatic extraction of PEPA performance models from UML activity diagrams annotated with the MARTE profile. In: Proceedings of the 7th International Workshop on Software and Performance, WOSP 2008, pp. 67–78. ACM (2008). https://doi.org/10.1145/1383559.1383569

  21. Zhang, Y., Myers, D.S., Arpaci-Dusseau, A.C., Arpaci-Dusseau, R.H.: Zettabyte reliability with flexible end-to-end data integrity, pp. 1–14. IEEE May 2013. https://doi.org/10.1109/MSST.2013.6558423, https://ieeexplore.ieee.org/document/6558423/

  22. Zhao, T., March, V., Dong, S., See, S.: Evaluation of a performance model of Lustre file system. In: 2010 Fifth Annual ChinaGrid Conference (ChinaGrid), pp. 191–196. IEEE (2010)

    Google Scholar 

Download references

Acknowledgment

The ESiWACE project received funding from the EU Horizon 2020 research and innovation programme under grant agreement No 675191.

Author information

Authors and Affiliations

  1. Deutsches Klimarechenzentrum, 20146, Hamburg, Germany

    Jakob Lüttgau & Julian Kunkel

  2. University of Reading, Reading, UK

    Julian Kunkel

Authors
  1. Jakob Lüttgau
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Julian Kunkel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jakob Lüttgau .

Editor information

Editors and Affiliations

  1. Tokyo Institute of Technology, Tokyo, Japan

    Rio Yokota

  2. University of Edinburgh, Edinburgh, UK

    Michèle Weiland

  3. Lawrence Berkeley National Laboratory, Berkeley, CA, USA

    John Shalf

  4. Swiss National Supercomputing Centre, Lugano, Switzerland

    Sadaf Alam

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lüttgau, J., Kunkel, J. (2018). Cost and Performance Modeling for Earth System Data Management and Beyond. In: Yokota, R., Weiland, M., Shalf, J., Alam, S. (eds) High Performance Computing. ISC High Performance 2018. Lecture Notes in Computer Science(), vol 11203. Springer, Cham. https://doi.org/10.1007/978-3-030-02465-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-02465-9_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-02464-2

  • Online ISBN: 978-3-030-02465-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation