Evaluating On-Node GPU Interconnects for Deep Learning Workloads

  • Nathan R. TallentEmail author
  • Nitin A. Gawande
  • Charles Siegel
  • Abhinav Vishnu
  • Adolfy Hoisie
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10724)


Scaling deep learning workloads across multiple GPUs on a single node has become increasingly important in data analytics. A key question is how well a PCIe-based GPU interconnect can perform relative to a custom high-performance interconnect such as NVIDIA’s NVLink. This paper evaluates two such on-node interconnects for eight NVIDIA Pascal P100 GPUs: (a) the NVIDIA DGX-1’s NVLink 1.0 ‘hybrid cube mesh’; and (b) the Cirrascale GX8’s two-level PCIe tree using dual SR3615 switch risers. To show the effects of a range of neural network workloads, we define a parameterized version of the popular ResNet architecture. We define a workload intensity metric that characterizes the expected computation/communication ratio; we also locate AlexNet and GoogLeNet within that space. As expected, the DGX-1 typically has superior performance. However, the GX8 is very competitive on all ResNet workloads. With 8 GPUs, the GX8 can outperform the DGX-1 on all-to-all reductions by 10% for medium-sized payloads; and in rare cases, the GX8 slightly outperforms on ResNet.


GPU interconnects NVIDIA DGX-1 NVIDIA NVLink Cirrascale SR3615 switch riser Convolutional neural networks 



The authors thank Matthew Macduff (PNNL) for evaluation assistance. We are grateful for funding support from the U.S. Department of Energy’s (DOE) Office of Advanced Scientific Computing Research as part of the “Center for Advanced Technology Evaluation” (CENATE) and “Convergence of Deep Learning and Machine Learning for HPC Simulation and Modeling.” Pacific Northwest National Laboratory is operated by Battelle for the DOE under Contract DE-AC05-76RL01830.


  1. 1.
    Gawande, N.A., Landwehr, J.B., Daily, J.A., Tallent, N.R., Vishnu, A., Kerbyson, D.J.: Scaling deep learning workloads: NVIDIA DGX-1/Pascal and Intel Knights Landing. In: 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 399–408, May 2017Google Scholar
  2. 2.
    Foley, D., Danskin, J.: Ultra-performance Pascal GPU and NVLink interconnect. IEEE Micro 37(2), 7–17 (2017)CrossRefGoogle Scholar
  3. 3.
    Sodani, A., Gramunt, R., Corbal, J., Kim, H.S., Vinod, K., Chinthamani, S., Hutsell, S., Agarwal, R., Liu, Y.C.: Knights landing: second-generation Intel Xeon Phi product. IEEE Micro 36(2), 34–46 (2016)CrossRefGoogle Scholar
  4. 4.
    Cirrascale: The GX8 series multi-device peering platform, June 2016.
  5. 5.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., Burges, C., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc., Red Hook (2012)Google Scholar
  6. 6.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)Google Scholar
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)Google Scholar
  8. 8.
    Cirrascale: Cirrascale SR3615 PCIe switch riser, July 2015Google Scholar
  9. 9.
  10. 10.
    Luehr, N.: Fast multi-GPU collectives with NCCL, April 2016.
  11. 11.
    NVIDIA: NCCL: NVIDIA collective communications library, August 2017.
  12. 12.
    Awan, A.A., Hamidouche, K., Venkatesh, A., Panda, D.K.: Efficient large message broadcast using NCCL and CUDA-aware MPI for deep learning. In: Proceedings of the 23rd European MPI Users’ Group Meeting, EuroMPI 2016, New York, NY, USA, pp. 15–22. ACM (2016)Google Scholar
  13. 13.
    Jeaugey, S.: Optimized inter-GPU collective operations with NCCL, May 2017.
  14. 14.
    Patarasuk, P., Yuan, X.: Bandwidth optimal all-reduce algorithms for clusters of workstations. J. Parallel Distrib. Comput. 69(2), 117–124 (2009)CrossRefGoogle Scholar
  15. 15.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Berkeley Vision and Learning Center. Berkeley vision and learning center: Caffe (2016).
  17. 17.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014)
  18. 18.
    Berkeley Vision and Learning Center: Convolutional architecture for fast feature embedding (Caffe) (2016).
  19. 19.
    NVIDIA: Convolutional architecture for fast feature embedding (Caffe) (2017).
  20. 20.
    Ben-Nun, T.: MGBench: multi-GPU computing benchmark suite, February 2016.
  21. 21.
    Cirrascale: Cirrascale SR3514: Unexpected performance inequality. Technical Brief M901A–092014Google Scholar
  22. 22.
    Adolf, R., Rama, S., Reagen, B., Wei, G.Y., Brooks, D.: Fathom: reference workloads for modern deep learning methods. In: 2016 IEEE International Symposium on Workload Characterization (IISWC), pp. 1–10, September 2016Google Scholar
  23. 23.
    Christensen, C., Fogal, T., Luehr, N., Woolley, C.: Topology-aware image compositing using NVLink. In: 2016 IEEE 6th Symposium on Large Data Analysis and Visualization (LDAV), pp. 93–94, October 2016Google Scholar
  24. 24.
    Shams, S., Platania, R., Lee, K., Park, S.J.: Evaluation of deep learning frameworks over different HPC architectures. In: Proceedings of the IEEE 37th International Conference on Distributed Computing Systems, pp. 1389–1396, June 2017Google Scholar
  25. 25.
    Nomura, S., Mitsuishi, T., Suzuki, J., Hayashi, Y., Kan, M., Amano, H.: Performance analysis of the multi-GPU system with expether. SIGARCH Comput. Archit. News 42(4), 9–14 (2014)CrossRefGoogle Scholar
  26. 26.
    Ben-Nun, T., Sutton, M., Pai, S., Pingali, K.: Groute: an asynchronous multi-GPU programming model for irregular computations. In: Proceedings of the 22nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, New York, NY, USA, pp. 235–248. ACM (2017)Google Scholar
  27. 27.
    Awan, A.A., Chu, C.H., Subramoni, H., Panda, D.K.: Optimized broadcast for deep learning workloads on dense-GPU infiniband clusters: MPI or NCCL? arXiv preprint arXiv:1707.09414 (2017)

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Nathan R. Tallent
    • 1
    Email author
  • Nitin A. Gawande
    • 1
  • Charles Siegel
    • 1
  • Abhinav Vishnu
    • 1
  • Adolfy Hoisie
    • 2
  1. 1.Pacific Northwest National LaboratoryRichlandUSA
  2. 2.Brookhaven National LaboratoryUptonUSA

Personalised recommendations