Comparative Study of Distributed Deep Learning Tools on Supercomputers

  • Xin Du
  • Di Kuang
  • Yan Ye
  • Xinxin Li
  • Mengqiang Chen
  • Yunfei Du
  • Weigang WuEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11334)


With the growth of the scale of data set and neural networks, the training time is increasing rapidly. Distributed parallel training has been proposed to accelerate deep neural network training, and most efforts are made on top of GPU clusters. This paper focuses on the performance of distributed parallel training in CPU clusters of supercomputer systems. Using resources at the supercomputer system of “Tianhe-2”, we conduct extensive evaluation of the performance of popular deep learning tools, including Caffe, TensorFlow, and BigDL, and several deep neural network models are tested, including AutoEncoder, LeNet, AlexNet and ResNet. The experiment results show that Caffe performs the best in communication efficiency and scalability. BigDL is the fastest in computing speed benefiting from its optimization for CPU, but it suffers from long communication delay due to the dependency on MapReduce framework. The insights and conclusions from our evaluation provides significant reference for improving resource utility of supercomputer resources in distributed deep learning.


Distributed deep learning Tianhe-2 Speedup Performance evaluation Parallel processing 


  1. 1.
    Cong, G., Kingsbury, B., Gosh, S., et al.: Accelerating deep neural network learning for speech recognition on a cluster of GPUs. In: Proceedings of the Machine Learning on HPC Environments, pp. 1–8. ACM, New York (2017)Google Scholar
  2. 2.
    Cui, H., Zhang, H., et al.: GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server. In: Proceedings of the Eleventh European Conference on Computer Systems, pp. 1–16. ACM, New York (2016)Google Scholar
  3. 3.
    Shi, S., Wang, Q., Xu, P., et al.: Benchmarking state-of-the-art deep learning software tools. In: 7th International Conference on Cloud Computing and Big Data, pp. 99–104. IEEE, Macau (2016)Google Scholar
  4. 4.
    Bahrampour, S., Ramakrishnan, N., Schott, L., et al.: Comparative study of deep learning software frameworks Computer Science (2015)Google Scholar
  5. 5.
  6. 6.
    Jia, Y., Shelhamer, E., Donahue, J., et al.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675–678. ACM, New York (2014)Google Scholar
  7. 7.
    Tang, Y.: TF.Learn: TensorFlow’s high-level module for distributed machine learning. CoRR 1612(04251) (2017)Google Scholar
  8. 8.
    Abadi, M., Barham, P., Chen, J., et al.: TensorFlow: a system for large-scale machine learning. In: OSDI 2016 Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, pp. 265–283. USENIX Association, Savannah (2016)Google Scholar
  9. 9.
    Yu, D., Eversole, A., Seltzer, M., et al.: An Introduction to computational Networks and the Computational Network Toolkit. Microsoft Research, Bangalore (2014)Google Scholar
  10. 10.
    Chen, T., Li, M., Li, Y., et al.: MXNet: a flexible and efficient machine learning library for heterogeneous distributed systems, CoRR 1512(01274) (2015)Google Scholar
  11. 11.
    Collobert, R., Kavukcuoglu, K., Farabet, C.: Torch7: a matlab-like environment for machine learning. Biglearn, Nips Workshop, pp. 1–6 (2012)Google Scholar
  12. 12.
  13. 13.
    Liao, X., Xiao, L., Yang, C., et al.: MilkyWay-2 supercomputer: system and application. Front. Comput. Sci. 8(3), 345–356 (2015)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Baldi, P.: Autoencoders, unsupervised learning, and deep architectures. In: Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning Workshop, Washington pp. 37–50 (2011)Google Scholar
  15. 15.
    Lecun, Y., Bottou, L., Bengio, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  16. 16.
    He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE, Las Vegas (2016)Google Scholar
  17. 17.
    Deng, J., Dong, W., Socher, R., et al.: ImageNet: a large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, pp. 248–255. IEEE, Miami (2009)Google Scholar
  18. 18.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolution neural networks. In: Advances in Neural Information Processing Systems, Lake Tahoe, Nevada, pp. 1097-1105 (2012)Google Scholar
  19. 19.
    Orozco, C.I., Iglesias, F., Buemi M.E., Berlles, J.J., et al.: Real-time gender recognition from face images using deep convolutional neural network. In: 7th Latin American Conference on Networked and Electronic Media (LACNEM) IET, Valparaiso, pp. 7–11 (2017)Google Scholar
  20. 20.
    Zbontar, J., LeCun, Y., et al.: Stereo matching by training a convolutional neural network to compare image patches. J. Mach. Learn. Res. 17(1), 2287–2318 (2016)zbMATHGoogle Scholar
  21. 21.
    Hecht-Nielsen, R.: Theory of the backpropagation neural network. In: International 1989 Joint Conference on Neural Networks, pp. 593–605. IEEE, Washington (1989)Google Scholar
  22. 22.
    Szegedy, C., Liu, W., Jia, Y., et al.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9. IEEE, Boston (2015)Google Scholar
  23. 23.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Comput. Sci. 1409(1556) (2014)Google Scholar
  24. 24.
    Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 640–651 (2014)CrossRefGoogle Scholar
  25. 25.
    D’Informatique, D.E., Ese, N., Esent, P., et al.: Long short-term memory in recurrent neural networks. EPFL 9(8), 1735–1780 (2001)Google Scholar
  26. 26.
    November 2017 top500 list.

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Xin Du
    • 1
    • 2
  • Di Kuang
    • 1
    • 2
  • Yan Ye
    • 1
    • 2
  • Xinxin Li
    • 1
    • 3
  • Mengqiang Chen
    • 1
    • 3
  • Yunfei Du
    • 1
    • 3
  • Weigang Wu
    • 1
    • 2
    Email author
  1. 1.School of Data and Computer ScienceSun Yat-sen UniversityGuangzhouChina
  2. 2.Guangdong Province Key Laboratory of Big Data Analysis and ProcessingGuangzhouChina
  3. 3.Key Laboratory of Machine Intelligence and Advanced ComputingMinistry of EducationGuangzhouChina

Personalised recommendations