Accelerating High-Order CFD Simulations for Multi-block Structured Grids on the TianHe-1A Supercomputer

Abstract

In this paper, we present a MPI-CUDA implementation for our in-house CFD software HOSTA to accelerate large-scale high-order CFD simulations on the TianHe-1A supercomputer. HOSTA employs a fifth order weighted compact nonlinear scheme (WCNS-E5) for flux calculation and a Runge-Kutta method for time integration. In our GPU parallelization scheme, we use CUDA thrad blocks to efficiently exploit fine-grained parallelism within a 3D grid block, and CUDA multiple streams to exploit coarse-grained parallelism among multiple grid blocks. At the CUDA-device level, we decompose complex flux kernels to optimize the GPU performance . At the cluster level, we present a Scatter-Gather optimization to reduce the PEI-E data transfer times for 3D block boundary/singularity data, and we overlap MPI communication and GPU execution. We achieve a speedup of about 10 when comparing our GPU code on a Tesla M2050 with the serial code on a Xeon X5670, and our implementation scales well to 128 GPUs on TianHe-1A.