Skip to main content
SpringerLink
Log in

GPU-based embedded edge server configuration and offloading for a neural network service

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Recently, emerging edge computing technology has been proposed as a new paradigm that compensates for the disadvantages of the current cloud computing. In particular, edge computing is used for service applications with low latency while using local data. For this emerging technology, a neural network approach is required to run large-scale machine learning on edge servers. In this paper, we propose a pod allocation method by adding various graphics processing unit (GPU) resources to increase the efficiency of a Kubernetes-based edge server configuration using a GPU-based embedded board and a TensorFlow-based neural network service application. As a result of experiments performed on the proposed edge server, the following are inferred: 1) The bandwidth, according to the time and data size, changes in local (20.4–42.4 Mbps) and Internet environments (6.31–25.5 Mbps) for service applications. 2) When two neural network applications are run on an edge server consisted with Xavier, TX2 and Nano, the network times of the object detection application are from 112.2 ms (Xavier) to 515.8 ms (Nano); the network times of the driver profiling application are from 321.8 ms (Xavier) to 495.7 ms (Nano). 3) The proposed pod allocation method demonstrates better performance than the default pod allocation method. We observe that the number of allocatable pods on three worker nodes increases from five to seven, and compared to other papers, the proposed offloading shows similar or better response times in environments where multiple deep learning applications are implemented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Duan Q (2017) Cloud service performance evaluation: status, challenges, and opportunities—a survey from the system modeling perspective. Digital Commun Networks 3(2):101–111

    Article  Google Scholar 

  2. Shirazi SN, Gouglidis A, Farshad A, Hutchison D (2017) The extended cloud: Review and analysis of mobile edge computing and fog from a security and resilience perspective. IEEE J Sel Areas Commun 35(11):2586–2595

    Article  Google Scholar 

  3. Burke B, Cearley D, Jones N, Smith D, Chandrasekaran A, Lu CK, Panetta K (2019) Gartner top 10 strategic technology trends for 2020-Smarter with Gartner

  4. Contributor IBM (2019, December 13) IBM BrandVoice: IBM Tech Trends To Watch In 2020 ... And Beyond. Retrieved from https://www.forbes.com/sites/ibm/2019/12/09/ibm-tech-trends-to-watch-in-2020--and-beyond/#280a11974c1c

  5. Shi W, Pallis G, Xu Z (2019) Edge computing [Scanning the Issue]. Proc IEEE 107(8):1474–1481

    Article  Google Scholar 

  6. Yousefpour A, Fung C, Nguyen T, Kadiyala K, Jalali F, Niakanlahiji A, Jue JP (2019) All one needs to know about fog computing and related edge computing paradigms: a complete survey. J Syst Architec 98:289–330

    Article  Google Scholar 

  7. Lyu X, Tian H, Jiang L, Vinel A, Maharjan S, Gjessing S, Zhang Y (2018) Selective offloading in mobile edge computing for the green internet of things. IEEE Network 32(1):54–60

    Article  Google Scholar 

  8. Markakis EK, Karras K, Sideris A, Alexiou G, & Pallis E (2017) Computing, caching, and communication at the edge: The cornerstone for building a versatile 5G ecosystem. In: IEEE Communications Magazine, 55(11), 152–157.]

  9. Kiani A, Ansari N (2017) Toward hierarchical mobile edge computing: an auction-based profit maximization approach. IEEE Internet Things J 4(6):2082–2091

    Article  Google Scholar 

  10. Ren J, Guo H, Xu C, Zhang Y (2017) Serving at the edge: a scalable IoT architecture based on transparent computing. IEEE Network 31(5):96–105

    Article  Google Scholar 

  11. Ryden M, Oh K, Chandra A & Weissman J (2014, March) Nebula: Distributed edge cloud for data intensive computing. In: 2014 IEEE International Conference on Cloud Engineering (pp. 57–66). IEEE

  12. Noreikis, M., Xiao, Y., & Ylä-Jaäiski, A. (2017, May). QoS-oriented capacity planning for edge computing. In: 2017 IEEE International Conference on Communications (ICC) (pp. 1–6). IEEE

  13. Malandrino F, Kirkpatrick S & Chiasserini CF (2016, December) How close to the edge? delay/utilization trends in mec. In: Proceedings of the 2016 ACM Workshop on Cloud-Assisted Networking (pp. 37–42)

  14. Kamiyama N, Nakano Y, Shiomoto K, Hasegawa G, Murata M & Miyahara H (2016, December) Analyzing effect of edge computing on reduction of web response time. In: 2016 IEEE Global Communications Conference (GLOBECOM) (pp. 1–6). IEEE

  15. Hou IH, Zhao T, Wang S & Chan K (2016, July) Asymptotically optimal algorithm for online reconfiguration of edge-clouds. In: Proceedings of the 17th ACM International Symposium on Mobile Ad Hoc Networking and Computing (pp. 291–300)

  16. Zhang W, Hu Y, Zhang Y & Raychaudhuri D (2016, December) Segue: Quality of service aware edge cloud service migration. In: 2016 IEEE International Conference on Cloud Computing Technology and Science (CloudCom) (pp. 344–351). IEEE

  17. Ismail BI, Goortani EM, Ab Karim MB, Tat WM, Setapa S, Luke JY & Hoe OH 2015, August) Evaluation of docker as edge computing platform. In: 2015 IEEE Conference on Open Systems (ICOS) (pp. 130–135). IEEE

  18. Pahl C, Helmer S, Miori L, Sanin J & Lee B (2016, August) A container-based edge cloud paas architecture based on raspberry pi clusters. In: 2016 IEEE 4th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW) (pp. 117–124). IEEE

  19. Helmer S, Pahl C, Sanin J, Miori L, Brocanelli S, Cardano F & Sharear AM (2016, November) Bringing the cloud to rural and remote areas via cloudlets. In: Proceedings of the 7th Annual Symposium on Computing for Development (pp. 1–10)

  20. Elkhatib Y, Porter B, Ribeiro HB, Zhani MF, Qadir J, Rivière E (2017) On using micro-clouds to deliver the fog. IEEE Internet Comput 21(2):8–15

    Article  Google Scholar 

  21. Zhang X, Wang Y & Shi W (2018) pcamp: Performance comparison of machine learning packages on the edges. In: {USENIX} Workshop on Hot Topics in Edge Computing (HotEdge 18)

  22. Pahl C & Lee B (2015, August) Containers and clusters for edge cloud architectures—a technology review. In: 2015 3rd international conference on future internet of things and cloud (pp. 379–386). IEEE

  23. Kim JH, Tulkinbekov K, Kim DH (2019) Benchmarking Kubernetes based Edge Server in Embedded Environment (pp. 49–52). In: The 5th International Conference on Next Generation Computing 2019 Proceeding

  24. Kubernetes Runtimes (2018, November 18) Retrieved April 20, 2020, from https://docs.google.com/spreadsheets/d/17ak_fVtWNUwMMJNpo7dNkoR8KK1ezTZVvh6v_tBcP7Y/edit#gid=0

  25. Bernstein D (2014) Containers and cloud: from lxc to docker to kubernetes. IEEE Cloud Comp 1(3):81–84

    Article  Google Scholar 

  26. Kang H, Le M & Tao S (2016, April) Container and microservice driven design for cloud infrastructure devops. In: 2016 IEEE International Conference on Cloud Engineering (IC2E) (pp. 202–211). IEEE

  27. Docker (2020, March 31) docker/classicswarm. Retrieved from https://github.com/docker/swarm/

  28. Production-Grade Container Orchestration (n.d.). Retrieved from https://kubernetes.io/

  29. Hindman B, Konwinski A., Zaharia M, Ghodsi A., Joseph AD, Katz RH & Stoica I (2011, March) Mesos: A platform for fine-grained resource sharing in the data center. In: NSDI (Vol. 11, No. 2011, pp. 22–22)

  30. Hoque S, de Brito MS, Willner A, Keil O & Magedanz T (2017, July) Towards container orchestration in fog computing infrastructures. In: 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC) (Vol. 2, pp. 294–299). IEEE

  31. NVIDIA DeepStream SDK (2020, April 25) Retrieved April 27, 2020, from https://developer.nvidia.com/deepstream-sdkNvidia. (2020, February 26). NVIDIA/nvidia-docker. Retrieved from https://github.com/NVIDIA/nvidia-docker

  32. Nvidia (2020, February 26) NVIDIA/nvidia-docker. Retrieved from https://github.com/NVIDIA/nvidia-docker

  33. GPU-Accelerated Innovation with NGC (n.d.). Retrieved from https://www.nvidia.com/en-us/gpu-cloud/

  34. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016, October). Ssd: Single shot multibox detector. In: European conference on computer vision (pp. 21–37). Springer, Cham

  35. Sandler M, Howard A., Zhu M, Zhmoginov A & Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4510–4520)

  36. Zhang J, Wu Z, Li F, Xie C, Ren T, Chen J, Liu L (2019) A deep learning framework for driving behavior identification on in-vehicle CAN-BUS sensor data. Sensors 19(6):1356

    Article  Google Scholar 

  37. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D & Zitnick CL (2014, September) Microsoft coco: Common objects in context. In: European conference on computer vision (pp. 740–755). Springer, Cham

  38. Kwak BI, Woo J & Kim HK (2016, December) Know your master: Driver profiling-based anti-theft method. In: 2016 14th Annual Conference on Privacy, Security and Trust (PST) (pp. 211–218). IEEE

  39. Tensorflow (2020, April 13). tensorflow/models. Retrieved from https://github.com/tensorflow/models/tree/master/research/object_detection

  40. Mortimer M. (2018) iperf3 documentation

  41. Ran X, Chen H, Zhu X, Liu Z & Chen J (2018, April) Deepdecision: A mobile deep learning framework for edge video analytics. In: IEEE INFOCOM 2018-IEEE Conference on Computer Communications (pp. 1421–1429). IEEE

  42. Liu P, Qi B & Banerjee S (2018, June) Edgeeye: An edge service framework for real-time intelligent video analytics. In: Proceedings of the 1st International Workshop on Edge Systems, Analytics and Networking (pp. 1–6)

  43. Jeong HJ, Jeong I, Lee HJ & Moon SM (2018, July) Computation offloading for machine learning web apps in the edge server environment. In: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS) (pp. 1492–1499). IEEE

  44. Ullah S and Kim DH 2020 Benchmarking Jetson Platform for 3D Point-Cloud and Hyper-Spectral Image Classification. In: 2020 IEEE International Conference on Big Data and Smart Computing (BigComp), pp. 477–482. IEEE

  45. Ullah S, Kim D-H (2020) Lightweight driver behavior identification model with sparse learning on In-Vehicle CAN-BUS sensor data. Sensors 20(18):5030

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2018R1D1A1B07042602) and in part by the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korean government (MSIT) (No. 2019-0-00064, Intelligent Mobile Edge Cloud Solution for Connected Car, No. 2019-0-00240, Deep Partition-and-Merge: Merging and Splitting Deep Neural Networks on Smart Embedded Devices for Real Time Inference).

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Inha University, Incheon, South Korea

    JooHwan Kim, Shan Ullah & Deok-Hwan Kim

Authors
  1. JooHwan Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shan Ullah
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Deok-Hwan Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deok-Hwan Kim.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, J., Ullah, S. & Kim, DH. GPU-based embedded edge server configuration and offloading for a neural network service. J Supercomput 77, 8593–8621 (2021). https://doi.org/10.1007/s11227-021-03623-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-03623-9

Keywords

Search

Navigation