A 64-GB Sort at 28 GB/s on a 4-GPU POWER9 Node for Uniformly-Distributed 16-Byte Records with 8-Byte Keys

  • Gordon C. Fossum
  • Ting Wang
  • H. Peter HofsteeEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11203)


Govinderaju et al. [1] have shown that a hybrid CPU-GPU system is cost-performance effective at sorting large datasets on a single node, but thus far large clusters used on sorting benchmarks have been limited by network and storage performance, and such clusters have remained CPU-only. With network and storage bandwidths improving more rapidly than CPU throughput performance, the cost effectiveness of CPU-GPU clusters for large sorts should be re-examined. As a first step, we evaluate sort performance on a single GPU-accelerated node with initial and final data residing in system memory. Access to main memory is limited to two reads and two writes, while executing the partitioning and sort in GPU memory. On a dual-socket IBM POWER9 system with four NVlink-attached NVIDIA V100 GPUs a single-node sort of 64 GB 8-byte key, 8-byte value records completes in under 2.3 s corresponding to a sort rate of over 28 GB/s. On a small (4-node) cluster with the same amount of data per node, the cluster sort completes in under 4.5 s. Sort performance is enabled by high system memory bandwidth, managing system-memory NUMA affinities, high CPU-GPU bandwidth, an efficient GPU-based partitioner, and an optimized GPU sort implementation. A cluster version of the algorithm benefits from minimizing copy operations by using RDMA. Matching the throughput of an optimized partitioner for our system would require a 50-100 GB/s network, which is feasible with a dual-socket POWER9 system.


Sorting Hybrid GPGPU 



The authors would like to thank Mark Nutter for discussions early-on in the project. We thank Bruce D’Amora for his support and feedback, and we want to thank the anonymous reviewers of an earlier version of this paper for their extensive feedback that contributed to significant improvements in this version of the paper.


  1. 1.
    Govindaraju, N., Gray, J., Kumar, R., Manocha, D.: Gputerasort: high performance graphics co-processor sorting for large database management. In: SIGMOD (2006)Google Scholar
  2. 2.
  3. 3.
    Parallel Integer Sort/BigSort.
  4. 4.
    Kruger, F.: CPU Bandwidth, the Worrisome 2020 Trend, March 2016. Accessed Feb 2018
  5. 5.
    Jiang, J., et al.: Tencent Sort. Accessed Dec 2017
  6. 6.
    Satish, N., Harris, M., Garland, M.: Designing efficient sorting algorithms for manycore gpus. In: IEEE International Symposium on Parallel and Distributed Processing, IPDPS 2009. IEEE, pp. 1–10 (2009)Google Scholar
  7. 7.
    Arkhipov, D.I., Wu, D., Li, K., Regan, A.C.: Sorting with GPUs: a survey. 1709.02520, September 2017. (
  8. 8.
    Stehle, E., Jacobsen, H.-A.: A memory bandwidth-efficient hybrid radix sort on GPUs. In: Proceedings of the 2017 ACM International Conference on Management of Data, pp. 417–432, ACM, New York (2017)Google Scholar
  9. 9.
    Hoberock, J., Bell, N.: Thrust: A Parallel Template 
Library (2016).
  10. 10.
    Cho, M., Brand, D., Bordawekar, R., Finkler,
U., Kulandaisamy, V., Puri, R.: Paradis: an efficient parallel algorithm for in-place radix sort. In: PVLDB (2015) Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Gordon C. Fossum
    • 1
  • Ting Wang
    • 2
  • H. Peter Hofstee
    • 1
    • 3
    Email author
  1. 1.IBM ResearchAustinUSA
  2. 2.IBM SystemsShanghaiChina
  3. 3.TU DelftDelftNetherlands

Personalised recommendations