Advertisement

Deep Learning Training Management Platform Based on Distributed Technologies in Resource-Constrained Scenarios

  • Jie Li
  • Guoteng Wang
  • Changsheng ZhangEmail author
  • Bin Zhang
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1074)

Abstract

Deep learning has attracted a lot of research attention in the past few years for its efficiency and accuracy. However, there exist two problems of their study. Firstly, the computing power of one single machine is limited and not suitable for handling with training deep learning models with massive cells. Secondly, it costs much to train models on different deep learning frameworks. Motivated by these problems, this paper proposed a deep learning training management platform based on distributed technologies, which integrates different kinds of deep learning frameworks through virtualization technologies and coordinates machines through distributed technologies. Specially, specific algorithms are proposed to solve the multi-task scheduling problem, the computing resources allocation problem and the fault tolerance problem in resource limited scenarios. It turns out that the platform can be widely used in small and medium-sized research teams.

Keywords

Deep learning Distributed technologies Virtualization technologies Automation 

References

  1. 1.
    Abadi, M., Barham, P., Chen, J., et al.: TensorFlow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2016), pp. 265–283 (2016)Google Scholar
  2. 2.
    Paszke, A., Gross, S., Chintala, S., et al.: PyTorch: tensors and dynamic neural networks in Python with strong GPU acceleration (2017). https://github.com/pytorch/pytorch
  3. 3.
    Jia, Y., Shelhamer, E., Donahue, J., et al.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675–678. ACM (2014)Google Scholar
  4. 4.
    Chen, T., Li, M., Li, Y., et al.: MXNet: a flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274 (2015)
  5. 5.
    Chollet, F.: Keras: the python deep learning library. Astrophysics Source Code Library (2018)Google Scholar
  6. 6.
    Yeager, L., Bernauer, J., Gray, A., et al.: DIGITS: the deep learning GPU training system. In: ICML 2015 AutoML Workshop (2015)Google Scholar
  7. 7.
    NVIDIA Docker. https://github.com/NVIDIA/nvidia-docker. Accessed 09 July 2017
  8. 8.
    Hunt, P., Konar, M., Junqueira, F.P., et al.: ZooKeeper: wait-free coordination for internet-scale systems. In: USENIX Annual Technical Conference, vol. 8, no. 9 (2010)Google Scholar
  9. 9.
    Chodorow, K.: MongoDB: The Definitive Guide: Powerful and Scalable Data Storage. O’Reilly Media, Inc., Sebastopol (2013)Google Scholar
  10. 10.
    Hightower, K., Burns, B., Beda, J.: Kubernetes: Up and Running: Dive Into the Future of Infrastructure. O’Reilly Media, Inc., Sebastopol (2017)Google Scholar
  11. 11.
    Dean, J., Corrado, G., Monga, R., et al.: Large scale distributed deep networks. In: Advances in Neural Information Processing Systems, pp. 1223–1231 (2012)Google Scholar
  12. 12.
    Sun, S., Chen, W., Bian, J., et al.: Slim-DP: a multi-agent system for communication-efficient distributed deep learning. In: Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, pp. 721–729. International Foundation for Autonomous Agents and Multiagent Systems (2018)Google Scholar
  13. 13.
    Ooi, B.C., Tan, K.L., Wang, S., et al.: SINGA: a distributed deep learning platform. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 685–688. ACM (2015)Google Scholar
  14. 14.
    Gupta, S., Zhang, W., Wang, F.: Model accuracy and runtime tradeoff in distributed deep learning: a systematic study. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 171–180. IEEE (2016)Google Scholar
  15. 15.
    Cui, H., Zhang, H., Ganger, G.R., et al.: GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server. In: Proceedings of the Eleventh European Conference on Computer Systems, p. 4. ACM (2016)Google Scholar
  16. 16.
    Frederic, J.: How to automate deep learning training with Kubernetes GPU-cluster [EB/OL], 07 September 2018. https://github.com/Langhalsdino/Kubernetes-GPU-Guide

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Jie Li
    • 1
  • Guoteng Wang
    • 1
  • Changsheng Zhang
    • 1
    Email author
  • Bin Zhang
    • 1
  1. 1.Northeastern UniversityShenyangPeople’s Republic of China

Personalised recommendations