Abstract
Nek5000 is a computational fluid dynamics code based on the spectral element method used for the simulation of incompressible flows. We follow up on an earlier study which ported the simplified version of Nek5000 to a GPU-accelerated system by presenting the hybrid CPU/GPU implementation of the full Nek5000 code using OpenACC. The matrix-matrix multiplication, the Nek5000 gather-scatter operator and a preconditioned Conjugate Gradient solver have implemented using OpenACC for multi-GPU systems. We report an speed-up of 1.3 on single node of a Cray XK6 when using OpenACC directives in Nek5000. On 512 nodes of the Titan supercomputer, the speed-up can be approached to 1.4. A performance analysis of the Nek5000 code using Score-P and Vampir performance monitoring tools shows that overlapping of GPU kernels with host-accelerator memory transfers would considerably increase the performance of the OpenACC version of Nek5000 code.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Fischer, P.F., Lottes J.W., Kerkemeier S.G.: Nek5000 web page. http://nek5000.mcs.anl.gov
The second nek5000 Users and Development Meeting, Zurich, Switzerland. http://nek5000.mcs.anl.gov/index.php/Usermeeting2012
Patera, A.T.: A spectral element method for fluid dynamics: laminar flow in a channel expansion. J. Comput. Phys. 54(3), 468–488 (1984)
OpenACC standard, June 2013. http://www.openacc-standard.org
Ansaloni R., Hart A.: Cray’s approach to heterogeneous computing. In: PARCO (2011)
Markidis, S., Gong, J., Schliephake, M., Laure, E., Hart, A., Henty, D., Heisey, K., Fischer, P.F.: OpenACC acceleration of Nek5000, spectral element code. Int. J. High Perform. Comput. Appl. (IJHPCA)
Deville, M.O., Fischer, P.F., Mund, E.H.: High-Order Method for Incompressible Fluid Flow. Cambridge University Press, Cambridge (2002)
Knüpfer, A., Rössel, C., Mey, D., Biersdorff, S., Diethelm, K., Eschweiler, D., Geimer, M., Gerndt, M., Lorenz, D., Malony, A., Nagel, W.E., Oleynik, Y., Philippen, P., Saviankou, P., Schmidl, D., Shende, S., Tschüter, R., Wagner, M., Wesarg, B., Wolf, F.: Tools for High Performance Computing 2011, pp. 79–91. Springer, Heidelberg (2012). doi:10.1007/978-3-642-31476-6_7
Knüpfer, A., Brunst, H., Doleschal, J., Jurenz, M., Lieber, M., Mickler, H., Müller, M.S., Nagel, W.E.: The vampir performance analysis tool set. In: Resch, M., Keller, R., Himmler, V., Krammer, B., Schulz, A. (eds.) Tools for High Performance Computing, pp. 139–155. Springer, Heidelberg (2008)
Schlatter, P., Khoury, G.K.E.: Turbulent flow in pipes. PDC Newsletter, no. 1 (2012)
Acknowledgment
This research has received funding from the Swedish e-Science Research Centre (SeRC) and the European Community’s Seventh Framework Programme (ICT-2011.9.13) under Grant Agreement no. 287703, cresta.eu. We are grateful for the computing time that was made available to us on the Raven system at Cray and on the Titan supercomputer at Oak Ridge National Laboratory within the INCITE program. We would also like to thank Dr. George K. El Khoury for the benchmark used in the paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Gong, J. et al. (2015). Nek5000 with OpenACC. In: Markidis, S., Laure, E. (eds) Solving Software Challenges for Exascale. EASC 2014. Lecture Notes in Computer Science(), vol 8759. Springer, Cham. https://doi.org/10.1007/978-3-319-15976-8_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-15976-8_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-15975-1
Online ISBN: 978-3-319-15976-8
eBook Packages: Computer ScienceComputer Science (R0)