Advertisement

Edge AIBench: Towards Comprehensive End-to-End Edge Computing Benchmarking

  • Tianshu Hao
  • Yunyou Huang
  • Xu Wen
  • Wanling Gao
  • Fan Zhang
  • Chen Zheng
  • Lei Wang
  • Hainan Ye
  • Kai Hwang
  • Zujie Ren
  • Jianfeng ZhanEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11459)

Abstract

In edge computing scenarios, the distribution of data and collaboration of workloads on different layers are serious concerns for performance, privacy, and security issues. So for edge computing benchmarking, we must take an end-to-end view, considering all three layers: client-side devices, edge computing layer, and cloud servers. Unfortunately, the previous work ignores this most important point. This paper presents the BenchCouncil’s coordinated effort on edge AI benchmarks, named Edge AIBench. In total, Edge AIBench models four typical application scenarios: ICU Patient Monitor, Surveillance Camera, Smart Home, and Autonomous Vehicle with the focus on data distribution and workload collaboration on three layers. Edge AIBench is publicly available from http://www.benchcouncil.org/EdgeAIBench/index.html. We also build an edge computing testbed with a federated learning framework to resolve performance, privacy, and security issues.

Keywords

Edge computing AI benchmarks Testbed Federated learning 

Notes

Acknowledgment

This work is supported by the Standardization Research Project of Chinese Academy of Sciences No. BZ201800001.

References

  1. 1.
    Peter, M., Timothy, G.: The NIST definition of cloud computing, recommendations of the National Institute of Standards and Technology. In: National Institute of Standards and Technology (NIST) Special Publication 800–145. Technical report (2011)Google Scholar
  2. 2.
  3. 3.
    Gartner Says the Internet of Things Will Transform the Data Centre. https://prwire.com.au/pr/42679/gartner-says-the-internet-of-things-will-transform-the-data-centre
  4. 4.
  5. 5.
    Vijay, J.R.: An ML benchmark suite for ML software frameworks and ML hardware accelerators in ML cloud and edge computing platforms. In: Report in BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (2018)Google Scholar
  6. 6.
  7. 7.
    Ignatov, A., et al.: AI benchmark: running deep neural networks on Android smartphones. In: Leal-Taixé, Laura, Roth, Stefan (eds.) ECCV 2018. LNCS, vol. 11133, pp. 288–314. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-11021-5_19CrossRefGoogle Scholar
  8. 8.
    Gao, W., et al.: AIBench: an industry standard internet service AI benchmark suite. Technical report (2019)Google Scholar
  9. 9.
    Gao W, et al.: AIBench: towards scalable and comprehensive datacenter AI benchmarking. In: BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench 2018) (2018)Google Scholar
  10. 10.
    Jiang, Z., et al.: HPC AI500: a benchmark suite for HPC AI systems. In: BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench 2018) (2018)Google Scholar
  11. 11.
    Luo, C., et al.: AIoT bench: towards comprehensive benchmarking mobile and embedded device intelligence. In: BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench 2018) (2018)Google Scholar
  12. 12.
    Wang, L., et al.: BigDataBench: a big data benchmark suite from internet services. In: 2014 IEEE 20th International Symposium on High Performance Computer Architecture (HPCA), 15 February 2014, pp. 488–499. IEEE (2014)Google Scholar
  13. 13.
    Jia, Z., Wang, L., Zhan, J., Zhang, L., Luo, C.: Characterizing data analysis workloads in data centers. In: 2013 IEEE International Symposium on Workload Characterization (IISWC), 22 September 2013, pp. 66–76. IEEE (2013)Google Scholar
  14. 14.
    Das, A., Patterson, S., Wittie, M.: EdgeBench: benchmarking edge computing platforms. In: 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion), 17 December 2018, pp. 175–180. IEEE (2018)Google Scholar
  15. 15.
    Johnson, A.E.W., et al.: MIMIC-III, a freely accessible critical care database. Sci. Data 3, 160035 (2016)CrossRefGoogle Scholar
  16. 16.
    Choi, E., Bahadori, M.T., Sun, J., Kulas, J., Schuetz, A., Stewart, W.: RETAIN: an interpretable predictive model for healthcare using reverse time attention mechanism. In: Advances in Neural Information Processing Systems, pp. 3504–3512 (2016)Google Scholar
  17. 17.
    Liu, L., Shen, J., Zhang, M., Wang, Z., Tang, J.: Learning the joint representation of heterogeneous temporal events for clinical endpoint prediction. In: Thirty-Second AAAI Conference on Artificial Intelligence, 25 April 2018Google Scholar
  18. 18.
    Amodei, D., et al.: Deep speech 2: end-to-end speech recognition in English and Mandarin. In: International Conference on Machine Learning, 11 June 2016, pp. 173–182 (2016)Google Scholar
  19. 19.
    Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: Librispeech: an ASR corpus based on public domain audio books. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 19 April 2015, pp. 5206–5210. IEEE (2015)Google Scholar
  20. 20.
    Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)Google Scholar
  21. 21.
    Huang, G.B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. In: Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition, October 2008Google Scholar
  22. 22.
    Yang, Q., Liu, Y., Chen, T., Tong, Y.: Federated machine learning: concept and applications. ACM Trans. Intell. Syst. Technol. (TIST) 10(2), 12 (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Tianshu Hao
    • 1
    • 2
  • Yunyou Huang
    • 1
    • 2
  • Xu Wen
    • 1
    • 2
  • Wanling Gao
    • 1
    • 3
  • Fan Zhang
    • 1
  • Chen Zheng
    • 1
    • 3
  • Lei Wang
    • 1
    • 3
  • Hainan Ye
    • 3
    • 4
  • Kai Hwang
    • 5
  • Zujie Ren
    • 6
  • Jianfeng Zhan
    • 1
    • 2
    • 3
    Email author
  1. 1.State Key Laboratory of Computer Architecture, Institute of Computing TechnologyChinese Academy of SciencesBeijingChina
  2. 2.University of Chinese Academy of SciencesBeijingChina
  3. 3.BenchCouncil (International Open Benchmark Council)DoverUSA
  4. 4.Beijing Academy of Frontier Sciences and TechnologyBeijingChina
  5. 5.Chinese University of Hongkong at ShenzhenShenzhenChina
  6. 6.Zhejiang LabZhejiangChina

Personalised recommendations