Skip to main content
SpringerLink
Log in

Dynamically Adaptable I/O Semantics for High Performance Computing

  • Conference paper
  • First Online:
High Performance Computing (ISC High Performance 2015)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 9137))

Included in the following conference series:

Abstract

While an input/output (I/O) interface’s syntax describes the available operations, its semantics determines how these operations behave and which assumptions developers can make about them. There are several different interface standards in existence, some of them dating back decades and having been designed for local file systems; one such representative is POSIX. Many parallel distributed file systems implement a POSIX-compliant interface to improve portability. All currently available interfaces follow a fixed approach regarding semantics, making them only suitable for a subset of use cases and workloads. While the interfaces do not allow application developers to influence the I/O semantics, applications could benefit greatly from the possibility of being able to adapt them to their requirements.

The work presented in this paper includes the design of a novel I/O interface and a file system called JULEA. They offer support for dynamically adaptable semantics and are suited specifically for HPC applications. The introduced concept allows applications to adapt the file system behavior to their exact I/O requirements instead of the other way around. The general goal is an interface that allows developers to specify what operations should do and how they should behave – leaving the actual realization and possible optimizations to the underlying file system.

JULEA has been evaluated using both synthetic benchmarks and real-world applications. Overall, JULEA provides data and metadata performance comparable to that of other established parallel distributed file systems. However, in contrast to the existing solutions, its flexible semantics allows it to cover a wider range of use cases in an efficient way. The results demonstrate that there is need for I/O interfaces that can adapt to the requirements of applications. Even though POSIX facilitates portability, it does not seem to be suited for contemporary HPC demands.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    POSIX’s synchronization functions fsync and fdatasync only allow synchronizing whole files even if this is not necessary.

  2. 2.

    This number is only intended to provide a rough estimate. In practice, the number might be lower due to the high discrepancy between insert and remove performance.

References

  1. Al-Kiswany, S., Gharaibeh, A., Ripeanu, M.: The case for a versatile storage system. SIGOPS Oper. Syst. Rev. 44(1), 10–14 (2010)

    Article  Google Scholar 

  2. Bartz, C.: An in-depth analysis of parallel high level I/O interfaces using HDF5 and NetCDF-4. Master’s thesis, University of Hamburg, April 2014

    Google Scholar 

  3. Brinkmann, A., Cortes, T., Falter, H., Kunkel, J., Narasimhamurthy, S.: E10 – Exascale IO, May 2014. http://www.eiow.org/home/E10-Architecture.pdf?attredirects=0&d=1. Accessed: April 2015

  4. Carns, P.H., Ligon III, W.B., Ross, R.B., Thakur, R.: PVFS: a parallel file system for linux clusters. In: Proceedings of the 4th Annual Linux Showcase and Conference, pp. 317–327. USENIX Association

    Google Scholar 

  5. Cluster File Systems Inc.: Lustre: a scalable, high-performance file system, November 2002. http://www.cse.buffalo.edu/faculty/tkosar/cse710/papers/lustre-whitepaper.pdf. Accessed: November 2014

  6. Frings, W., Wolf, F., Petkov, V.: Scalable massively parallel I/O to task-local files. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, SC 2009. ACM, New York (2009). http://doi.acm.org/10.1145/1654059.1654077

  7. Gharachorloo, K., Gupta, A., Hennessy, J.: Performance evaluation of memory consistency models for shared-memory multiprocessors. In: Proceedings of the Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS IV, pp. 245–257. ACM, New York (1991). http://doi.acm.org/10.1145/106972.106997

  8. Gharachorloo, K., Lenoski, D., Laudon, J., Gibbons, P., Gupta, A., Hennessy, J.: Memory consistency and event ordering in scalable shared-memory multiprocessors. In: Proceedings of the 17th Annual International Symposium on Computer Architecture, ISCA 1990, pp. 15–26. ACM, New York (1990). http://doi.acm.org/10.1145/325164.325102

  9. The IEEE and The Open Group: Standard for Information Technology - Portable Operating System Interface (POSIX) Base Specifications, Issue 7. IEEE Std 1003.1, 2013 Edition (incorporates IEEE Std 1003.1-2008, and IEEE Std 1003.1-2008/Cor 1–2013) pp. 1–3906, April 2013

    Google Scholar 

  10. ISO/IEC JTC 1/SC 22 - Programming languages, their environments and system software interfaces: ISO/IEC 9899:2011 - Information technology - Programming languages - C, December 2011

    Google Scholar 

  11. Klasky, S., Liu, Q., Lofstead, J., Podhorszki, N., Abbasi, H., Chang, C., Cummings, J., Dinakar, D., Docan, C., Ethier, S., Grout, R., Kordenbrock, T., Lin, Z., Ma, X., Oldfield, R., Parashar, M., Romosan, A., Samatova, N., Schwan, K., Shoshani, A., Tian, Y., Wolf, M., Yu, W., Zhang, F., Zheng, F.: ADIOS: powering I/O to extreme scale computing. In: SciDAC 2010 Conference Proceedings, pp. 342–347 (2010). http://computing.ornl.gov/workshops/scidac2010/papers/data_q_liu.pdf

  12. Kuhn, M.: A semantics-aware I/O interface for high performance computing. In: Kunkel, J.M., Ludwig, T., Meuer, H.W. (eds.) ISC 2013. LNCS, vol. 7905, pp. 408–421. Springer, Heidelberg (2013)

    Chapter  Google Scholar 

  13. Kuhn, M.: Dynamically adaptable I/O semantics for high performance computing. Ph.D. thesis, University of Hamburg, Germany, November 2014 (to be published)

    Google Scholar 

  14. Kunkel, J., Minartz, T., Kuhn, M., Ludwig, T.: Towards an energy-aware scientific I/O interface - stretching the ADIOS interface to foster performance analysis and energy awareness. Comput. Sci. Res. Dev. 27(4), 337–345 (2011)

    Article  Google Scholar 

  15. Lofstead, J.F., Klasky, S., Schwan, K., Podhorszki, N., Jin, C.: Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS). In: Proceedings of the 6th International Workshop on Challenges of Large Applications in Distributed Environments, CLADE 2008, pp. 15–24. ACM, New York (2008)

    Google Scholar 

  16. Message Passing Interface Forum: MPI: a message-passing interface standard. Version 3.0, September 2012. http://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf. Accessed: November 2014

  17. Patil, S., Gibson, G.A., Ganger, G.R., Lopez, J., Polte, M., Tantisiroj, W., Xiao, L.: In search of an API for scalable file systems: under the table or above it? In: Proceedings of the 2009 Conference on Hot Topics in Cloud Computing, HotCloud 2009. USENIX Association, Berkeley (2009)

    Google Scholar 

  18. Rew, R., Davis, G.: Data management: NetCDF: an interface for scientific data access. IEEE Comput. Graph. Appl. 10(4), 76–82 (1990). http://dx.doi.org/10.1109/38.56302

    Article  Google Scholar 

  19. The HDF Group: Hierarchical data format version 5, July 2014. http://www.hdfgroup.org/HDF5. Accessed: November 2014

  20. The TOP500 Editors: TOP500, June 2014. http://www.top500.org/. Accessed: November 2014

  21. Vilayannur, M., Lang, S., Ross, R., Klundt, R., Ward, L.: Extending the POSIX I/O interface: a parallel file system perspective. Technical report ANL/MCS-TM-302, October 2008. http://www.mcs.anl.gov/uploads/cels/papers/TM-302-FINAL.pdf

  22. Vilayannur, M., Nath, P., Sivasubramaniam, A.: Providing tunable consistency for a parallel file store. In: Proceedings of the 4th Conference on USENIX Conference on File and Storage Technologies, FAST 2005, vol. 4. USENIX Association, Berkeley (2005)

    Google Scholar 

  23. Wikipedia: Festplattenlaufwerk - Geschwindigkeit, November 2014. http://de.wikipedia.org/wiki/Festplattenlaufwerk#Geschwindigkeit. Accessed: November 2014

  24. Wikipedia: Mark Kryder - Kryder’s Law, November 2014. http://en.wikipedia.org/wiki/Mark_Kryder#Kryder.27s_Law. Accessed: November 2014

Download references

Author information

Authors and Affiliations

  1. University of Hamburg, Hamburg, Germany

    Michael Kuhn

Authors
  1. Michael Kuhn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Kuhn .

Editor information

Editors and Affiliations

  1. Deutsches Klimarechenzentrum (DKRZ), Hamburg, Germany

    Julian M. Kunkel

  2. Deutsches Klimarechenzentrum (DKRZ), Hamburg, Germany

    Thomas Ludwig

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Kuhn, M. (2015). Dynamically Adaptable I/O Semantics for High Performance Computing. In: Kunkel, J., Ludwig, T. (eds) High Performance Computing. ISC High Performance 2015. Lecture Notes in Computer Science(), vol 9137. Springer, Cham. https://doi.org/10.1007/978-3-319-20119-1_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-20119-1_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-20118-4

  • Online ISBN: 978-3-319-20119-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation