Abstract
Since the birth of artificial intelligence, the theory and the technology have become more mature, and the application field is expanding. In this paper, we build an artificial intelligence platform for heterogeneous computing, which supports deep learning frameworks such as TensorFlow and Caffe. We describe the overall architecture of the AI platform for a GPU cluster. In the GPU cluster, based on the scheduling layer, we propose Yarn by the Slurm scheduler to not only improve the distributed TensorFlow plug-in for the Slurm scheduling layer but also to extend YARN to manage and schedule GPUs. The front-end of the high-performance AI platform has the attributes of availability, scalability and efficiency. Finally, we verify the convenience, scalability, and effectiveness of the AI platform by comparing the performance of single-chip and distributed versions for the TensorFlow, Caffe and YARN systems.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Shanhai, W., Xinxing, J., Haiyan, Y.: Research on isolated word speech recognition based on deep learning neural network. Comput. Appl. Res. 32(8), 2289–2291 (2015)
Gantz, J., Reinsel, D.: Digital Universe in 2020 [EB/OL]. https://www.emc.com/collateral/analyst-reports/idc-the-digital-universe-in-2020.pdf. Accessed 1 Dec 2012
Christiansen, B., Garey, M., Hartung, I.: SlurmOveview [EB/OL]. https://slurm.schedmd.com/SC17/SlurmOverviewSC17.pdf. Accessed 12 Dec 2017
Jeff, D., Rajat, M., et al.: (9 November 2015) TensorFlow: Large-scale machine learning on heterogeneous systems (PDF). TensorFlow.org. Google Research. Accessed 10 Nov 2015
Cybulska, M.: Assessing yarn structure with image analysis methods1. Text. Res. J. 69(5), 369–373 (1999)
Pacelli, M., Caldani, L., Paradiso, R.: Performances evaluation of piezoresistive fabric sensors as function of yarn structure. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6502–6505 (2013)
Ozturk, M., Nergis, B.U.: Determining the dependence of colour values on yarn structure. Color. Technol. 124(3), 145–150 (2008)
Owens, J.D., Houston, M., Luebke, D., et al.: GPU computing. Proceed. IEEE 96(5), 879–899 (2008)
Hou, S., Tan, M.T., Luo, X.G.: Application driven multi DSP processor array in high performance computing. Comput. Appl. Res. 28(4), 1336–1338 (2011)
Nickolls, J., Dally, W.J.: The GPU computing era. IEEE Micro 30(2), 56–69 (2010)
Hwu, W.M.W.: Introduction - GPU computing gems emerald edition. GPU Comput. Gems Emerald Ed. 27, 599–600 (2011)
Pratx, G., Xing, L.: GPU computing in medical physics: a review. Med. Phys. 38(5), 2685–2697 (2011)
Li, X.: A design of secret information system against internal network attack based on software container. Inf. Comput. 10, 109–111 (2016)
Ubal, R., Schaa, D., Jang, B., et al.: Multi2Sim: a simulation framework for CPU-GPU computing, pp. 335–344 (2012)
IBM Platform LSF. http://www-03.ibm.com/systems/platformcomputing/products/lsf/
Wei, J.: Research on massive transaction record query system based on Hadoop. Nanjing University of Posts and Telecommunications, Nanjing (2013)
Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google file system. In: 19th ACM Symposium on Operating Systems Principles, October 2013
Wikipedia contributors. Apache Hadoop [EB/OL]. https://en.wikipedia.org/wiki/Apache_Hadoop. Accessed 13 Dec 2017
Braam, P.J.: The Lustre storage architecture. Cluster File Systems, Inc. (2004). http://www.clusterfs.com
Wikipedia contributors. Lustre_(file_system) [EB/OL]. https://en.wikipedia.org/wiki/Lustre_(file_system). Accessed 31 Jan 2018
Wang, J., Gao F., Vazquez-Poletti, J.L., Li, J.: Preface of high performance computing or advanced modeling and simulation of materials. Comput. Phys. Commun. (211) (2017). (IF: 3.653)
Martin, A., Raponi, S., Combe, T., et al.: Docker ecosystem – vulnerability analysis. Comput. Commun. 122, 30–43 (2018)
Slurm workload manager [EB/OL]. http://slurm.schedmd.com/slurm.html
Wang, J., Liu, C., Huang, Y.: Auto tuning for new energy dispatch problem: A case study. Future Gener. Comput. Syst. 54, 501–506 (2016)
HuaiTe, Z., et al.: Hadoop Authoritative Guide, 2nd edn. Tsinghua University Press, Beijing (2011)
Zhao, H., Zhang, Y., Bradford, P.D., et al.: Carbon nanotube yarn strain sensors. Nanotechnology 21(30), 305502 (2010)
Ramesh, M.C., Rajamanickam, R., Jayaraman, S.: The Prediction of yarn tensile properties by using artificial neural networks. J. Text. Inst. Proceed. Abstr. 86(3), 459–469 (1995)
Debo, L.: Research of GPU cluster system based on YARN, Sun Yat-sen University, Guangzhou (2014)
Schwarz, E.R.: Certain aspects of yarn structure. Text. Res. J. 21(3), 125–136 (1951)
Acknowledgments
This work was partly supported by the National Key R&D Program of China (No. 2017YFB0202202), the State Key Program of National Natural Science Foundation of China (No. 61702476).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, H. et al. (2018). Artificial Intelligence Platform for Heterogeneous Computing. In: Qiu, M. (eds) Smart Computing and Communication. SmartCom 2018. Lecture Notes in Computer Science(), vol 11344. Springer, Cham. https://doi.org/10.1007/978-3-030-05755-8_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-05755-8_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-05754-1
Online ISBN: 978-3-030-05755-8
eBook Packages: Computer ScienceComputer Science (R0)