Abstract
Parallelization is essential for machine learning systems that deals with large-scale dataset. Data parallel machine leaning systems that are composed of multiple machine learning modules, exchange the parameter to synchronize the models in the modules through network. We investigate the network bandwidth requirements for various parameter exchange method using a cluster simulator called SimGrid. We have confirmed that (1) direct exchange methods are substantially more efficient than parameter server based methods, and (2) with proper exchange methods, the bisection-bandwidth of network does not affect the efficiency, which implies smaller investment on network facility will be sufficient.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This means that the number of worker nodes is different between butterfly network based method and parameter server based method. (Number of worker nodes of parameter server based method is always n nodes fewer, where n is the number of nodes per sub-clusters.) However, even if the butterfly network based method reduces n nodes, the execution time is expected to be the same.
References
Parameter server: http://parameterserver.org/. Accessed 20 June 2015
Simgrid: Versatile simulation of distributed systems. http://simgrid.gforge.inria.fr/index.php. Accessed 11 July 2016
Agarwal, A., Chapelle, O., Dudík, M., Langford, J.: A reliable effective terascale linear learning system. J. Mach. Learn. Res. 15(1), 1111–1133 (2014). http://dl.acm.org/citation.cfm?id=2627435.2638571
Casanova, H., Giersch, A., Legrand, A., Quinson, M., Suter, F.: Versatile, scalable, and accurate simulation of distributed applications and platforms. J. Parallel Distrib. Comput. 74(10), 2899–2917 (2014). http://hal.inria.fr/hal-01017319
Dean, J., Corrado, G.S., Monga, R., Chen, K., Devin, M., Le, Q.V., Mao, M.Z., Ranzato, M., Senior, A., Tucker, P., Yang, K., Ng, A.Y.: Large scale distributed deep networks. In: NIPS 2012: Neural Information Processing Systems (2012)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition (CVPR) (2016)
Hennessy, J.L., Patterson, D.A.: Computer Architecture: A Quantitative Approach, 3rd edn. Morgan Kaufmann Publishers, Inc., San Francisco (2003)
Ho, Q., Cipar, J., Cui, H., Kim, J.K., Lee, S., Gibbons, P.B., Gibson, G.A., Ganger, G.R., Xing, E.P.: More effective distributed ml via a stale synchronous parallel parameter server. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, NIPS 2013, pp. 1223–1231. Curran Associates Inc., USA (2013). http://dl.acm.org/citation.cfm?id=2999611.2999748
Jin, P., Yuan, Q., Jin, P., Keutzer, K.: How to scale distributed deep learning? In: ML Systems Workshop, NIPS 2016 (2016)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates Inc. (2012)
Le, Q., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G., Dean, J., Ng, A.: Building high-level features using large scale unsupervised learning. In: International Conference in Machine Learning (2012)
Leiserson, C.E.: Fat-trees: universal networks for hardware-efficient supercomputing. IEEE Trans. Comput. 34(10), 892–901 (1985)
Li, M., Andersen, D.G., Park, J.W., Smola, A.J., Ahmed, A., Josifovski, V., Long, J., Shekita, E.J., Su, B.Y.: Scaling distributed machine learning with the parameter server. In: 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2014), pp. 583–598. USENIX Association, Broomfield, October 2014. https://www.usenix.org/conference/osdi14/technical-sessions/presentation/li_mu
Seide, F., Fu, H., Droppo, J., Li, G., Yu, D.: 1-bit stochastic gradient descent and application to data-parallel distributed training of speech dnns. In: Interspeech 2014, September 2014
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Computer Vision and Pattern Recognition (CVPR) (2015). http://arxiv.org/abs/1409.4842
Thakur, R., Gropp, W.D.: Improving the performance of MPI collective communication on switched networks. Technical report, ANL/MCS-P1007-1102, Argonne National Laboratory, November 2002
Valiant, L.G.: A bridging model for parallel computation. Commun. ACM 33(8), 103–111 (1990). http://doi.acm.org/10.1145/79173.79181
Zhao, H., Canny, J.: Butterfly mixing: accelerating incremental-update algorithms on clusters. In: Proceedings of the 2013 SIAM International Conference on Data Mining (2013)
Acknowledgement
This paper is based on results obtained from a project commissioned by the New Energy and Industrial Technology Development Organization (NEDO). This work was supported by JSPS KAKENHI Grant Number JP16K00116.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Li, M., Tanimura, Y., Nakada, H. (2018). A Quantitative Analysis on Required Network Bandwidth for Large-Scale Parallel Machine Learning. In: Nicosia, G., Pardalos, P., Giuffrida, G., Umeton, R. (eds) Machine Learning, Optimization, and Big Data. MOD 2017. Lecture Notes in Computer Science(), vol 10710. Springer, Cham. https://doi.org/10.1007/978-3-319-72926-8_32
Download citation
DOI: https://doi.org/10.1007/978-3-319-72926-8_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-72925-1
Online ISBN: 978-3-319-72926-8
eBook Packages: Computer ScienceComputer Science (R0)