NUMA-Aware Data-Transfer Measurements for Power/NVLink Multi-GPU Systems

  • Carl PearsonEmail author
  • I-Hsin ChungEmail author
  • Zehra SuraEmail author
  • Wen-Mei HwuEmail author
  • Jinjun XiongEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11203)


High-performance computing increasingly relies on heterogeneous systems with specialized hardware accelerators to improve application performance. For example, NVIDIA’s CUDA programming system and general-purpose GPUs have emerged as a widespread accelerator in HPC systems. This trend has exacerbated challenges of data placement as accelerators often have fast local memories to fuel their computational demands, but slower interconnects to feed those memories. Crucially, real-world data-transfer performance is strongly influenced not just by the underlying hardware, but by the capabilities of the programming systems. Understanding how application performance is affected by the logical communication exposed through abstractions, as well as the underlying system topology, is crucial for developing high-performance applications and architectures. This report presents initial data-transfer microbenchmark results from two POWER-based systems obtained during work towards developing an automated system performance characterization tool.


CUDA NVLink Unified Memory GPGPU Benchmark 


  1. 1.
    NUMA(3) Linux Programmer’s Manual (August 2007)Google Scholar
  2. 2.
    Cuda c programming guide (Nov 2017)Google Scholar
  3. 3.
    Caldeira, A.B.: Ibm power system ac922 introduction and technical overview. IBM Redbooks (2018)Google Scholar
  4. 4.
    Caldeira, A.B., Haug, V., Vetter, S.: Ibm power system 822lc for high performance computing introduction and technical overview. IBM Redbooks (2016)Google Scholar
  5. 5.
    Google: Benchmark. (2018)
  6. 6.
  7. 7.
    Pearson, C., Dakkak, A., Li, C.: microbench. (2018)
  8. 8.
    Wickman, C., Lameter, C., Schermerhorn, L.: numactl v2.0.11. (2015)

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.University of Illinois Urbana-ChampaignUrbanaUSA
  2. 2.IBM Thomas J. Watson Research CenterYorktown HeightsUSA

Personalised recommendations