A Quantitative Analysis on Required Network Bandwidth for Large-Scale Parallel Machine Learning

  • Mingxi Li
  • Yusuke Tanimura
  • Hidemoto NakadaEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10710)


Parallelization is essential for machine learning systems that deals with large-scale dataset. Data parallel machine leaning systems that are composed of multiple machine learning modules, exchange the parameter to synchronize the models in the modules through network. We investigate the network bandwidth requirements for various parameter exchange method using a cluster simulator called SimGrid. We have confirmed that (1) direct exchange methods are substantially more efficient than parameter server based methods, and (2) with proper exchange methods, the bisection-bandwidth of network does not affect the efficiency, which implies smaller investment on network facility will be sufficient.



This paper is based on results obtained from a project commissioned by the New Energy and Industrial Technology Development Organization (NEDO). This work was supported by JSPS KAKENHI Grant Number JP16K00116.


  1. 1.
    Parameter server: Accessed 20 June 2015
  2. 2.
    Simgrid: Versatile simulation of distributed systems. Accessed 11 July 2016
  3. 3.
    Agarwal, A., Chapelle, O., Dudík, M., Langford, J.: A reliable effective terascale linear learning system. J. Mach. Learn. Res. 15(1), 1111–1133 (2014). Scholar
  4. 4.
    Casanova, H., Giersch, A., Legrand, A., Quinson, M., Suter, F.: Versatile, scalable, and accurate simulation of distributed applications and platforms. J. Parallel Distrib. Comput. 74(10), 2899–2917 (2014). Scholar
  5. 5.
    Dean, J., Corrado, G.S., Monga, R., Chen, K., Devin, M., Le, Q.V., Mao, M.Z., Ranzato, M., Senior, A., Tucker, P., Yang, K., Ng, A.Y.: Large scale distributed deep networks. In: NIPS 2012: Neural Information Processing Systems (2012)Google Scholar
  6. 6.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  7. 7.
    Hennessy, J.L., Patterson, D.A.: Computer Architecture: A Quantitative Approach, 3rd edn. Morgan Kaufmann Publishers, Inc., San Francisco (2003)zbMATHGoogle Scholar
  8. 8.
    Ho, Q., Cipar, J., Cui, H., Kim, J.K., Lee, S., Gibbons, P.B., Gibson, G.A., Ganger, G.R., Xing, E.P.: More effective distributed ml via a stale synchronous parallel parameter server. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, NIPS 2013, pp. 1223–1231. Curran Associates Inc., USA (2013).
  9. 9.
    Jin, P., Yuan, Q., Jin, P., Keutzer, K.: How to scale distributed deep learning? In: ML Systems Workshop, NIPS 2016 (2016)Google Scholar
  10. 10.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates Inc. (2012)Google Scholar
  11. 11.
    Le, Q., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G., Dean, J., Ng, A.: Building high-level features using large scale unsupervised learning. In: International Conference in Machine Learning (2012)Google Scholar
  12. 12.
    Leiserson, C.E.: Fat-trees: universal networks for hardware-efficient supercomputing. IEEE Trans. Comput. 34(10), 892–901 (1985)CrossRefGoogle Scholar
  13. 13.
    Li, M., Andersen, D.G., Park, J.W., Smola, A.J., Ahmed, A., Josifovski, V., Long, J., Shekita, E.J., Su, B.Y.: Scaling distributed machine learning with the parameter server. In: 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2014), pp. 583–598. USENIX Association, Broomfield, October 2014.
  14. 14.
    Seide, F., Fu, H., Droppo, J., Li, G., Yu, D.: 1-bit stochastic gradient descent and application to data-parallel distributed training of speech dnns. In: Interspeech 2014, September 2014Google Scholar
  15. 15.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Computer Vision and Pattern Recognition (CVPR) (2015).
  16. 16.
    Thakur, R., Gropp, W.D.: Improving the performance of MPI collective communication on switched networks. Technical report, ANL/MCS-P1007-1102, Argonne National Laboratory, November 2002Google Scholar
  17. 17.
    Valiant, L.G.: A bridging model for parallel computation. Commun. ACM 33(8), 103–111 (1990). Scholar
  18. 18.
    Zhao, H., Canny, J.: Butterfly mixing: accelerating incremental-update algorithms on clusters. In: Proceedings of the 2013 SIAM International Conference on Data Mining (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.University of TsukubaTsukubaJapan
  2. 2.National Institute of Advanced Industrial Science and TechnologyTokyoJapan

Personalised recommendations