Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets
This paper presents an in situ framework focused on time-varying simulations, and uses a novel temporal buffer for storing simulation results sampled at user-defined intervals. This framework has been designed to provide flexible data processing and visualization capabilities in modern HPC operational environments composed of powerful front-end systems, for pre-and post-processing purposes, along with traditional back-end HPC systems. The temporal buffer is implemented using the functionalities provided by Open Address Space (OpAS) library, which enables asynchronous one-sided communication from outside processes to any exposed memory region on the simulator side. This buffer can store time-varying simulation results, and can be processed via in situ approaches with different proximities. We present a prototype of our framework, and code integration process with a target simulation code. The proposed in situ framework utilizes separate files to describe the initialization and execution codes, which are in the form of Python scripts. This framework also enables the runtime modification of these Python-based files, thus providing greater flexibility to the users, not only for data processing, such as visualization and analysis, but also for the simulation steering.
KeywordsTemporal buffer Time-varying data Heterogeneous architecture
This research has used the computational resources of the K computer at RIKEN Center for Computational Science (R-CCS) in Kobe, Japan. This work is partially supported by the “Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures” in Japan (Project ID: jh180060-NAH), and also by the Japanese Ministry of Education, Culture, Sports, Science and Technology (MEXT) as a social and scientific priority issue (Development of Innovative Design and Production Processes that Lead the Way for the Manufacturing Industry in the Near Future) to be tackled by using the post-K supercomputer.
- 1.Ayachit, U., et al.: The SENSEI generic in situ interface. In: Proceedings of the 2nd Workshop on In Situ Infrastructures for Enabling Extreme-scale Analysis and Visualization, ISAV 2016, pp. 40–44. IEEE Press, Piscataway (2016). https://doi.org/10.1109/ISAV.2016.13
- 2.Bauer, A.C., et al.: In situ methods, infrastructures, and applications on high performance computing platforms. Comput. Graph Forum 35(3), 577–597 (2016). https://doi.org/10.1111/cgf.12930. https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.12930CrossRefGoogle Scholar
- 3.Fareed, H., Singler, J.R.: Error Analysis of an Incremental POD Algorithm for PDE Simulation Data. ArXiv e-prints, March 2018Google Scholar
- 5.Research Institute for Information Technology, K.U.: Supercomputer System ITO. https://www.cc.kyushu-u.ac.jp/scp/eng/system/01_into.html. Accessed 15 May 2018
- 6.Kress, J., Klasky, S., Podhorszki, N., Choi, J., Childs, H., Pugmire, D.: Loosely coupled in situ visualization: a perspective on why it’s here to stay. In: Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, pp. 1-6. ISAV 2015. ACM, New York (2015). https://doi.org/10.1145/2828612.2828623, http://doi.acm.org/10.1145/2828612.2828623
- 7.Nanri, T.: Proposal of interface for runtime memory manipulation of applications via PGAS-based communication library. In: Workshop on PGAS programming models: Experiences and Implementations, HPC Asia 2018, 31 January 2018Google Scholar
- 8.Ono, K., Kawashima, Y., Kawanabe, T.: Data centric framework for large-scale high-performance parallel computation. Procedia Comput. Sci. 29, 2336–2350 (2014). https://doi.org/10.1016/j.procs.2014.05.218, http://www.sciencedirect.com/science/article/pii/S1877050914003950. 2014 International Conference on Computational ScienceCrossRefGoogle Scholar