Abstract
With the rapid development of Internet technology, global data are growing at an exponential rate, and human computing has undergone profound changes. The early stand-alone mode has not met people’s needs, and the network-based collaborative distributed gained more and more attention and favor. Distributed computing system is one of the hottest Internet research directions in the era of big data era. It has the characteristics of high efficiency, high capacity, dynamic processing, and so on. It shows great application value in the commercial field and scientific research field of society. Based on the development and present situation of distributed computing system, this paper reviews and summarizes two popular key technologies: grid technology and cloud computing technology and expounds the differences between the two. At the same time, the design principles and system performance of Hadoop, Storm, Spark, and other typical distributed computing platforms are compared and analyzed in detail, which is of theoretical significance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Coulouris G, Dollimore J, Kindberg T. Distributed systems: concepts and design—4th ed. [J]. Programmable Controllers, 2011, 18(95):182–231.
Mao zhen li, Mark Baker. The Grid core technologies, 2006: 254–262.
ZHANG Jian-xun, GU Zhi-min, ZHENG Chao. Survey of research progress on cloud computing [J]. Application Research of Computer. 2010, 27(02):429–433.
Werner Vogels. Web services are not distributed objects. Internet Computing, IEEE, 2003, 7(6): P59–66.
XIA Jing-bo, WEI Ze-kun, FU Kai, CHEN Zhen. Review of research and application on Hadoop on cloud computing [J]. Computer Science, 2016, 43(11):6–11.
Apache Software Foundation. Hadoop [EB/OL]. http://hadoop.apache.org/.
Anderson Q. Storm Real-Time Processing Cookbook [M]. Packt Publishing, 2013.
Apache spark [EB/OL].http://spark.apache.org/.
Zhang Wenfeng. The principle and design of distributed computing platform based on MapReduce [D]. Huazhong University of Science and Technology, 2010.
Zhou Xiao-feng, Wang Zhi-jian. Overview of distributed computing technology [J]. Computer Era, 2004(12):3–5.
GE Peng. A brief overview on distributed computing technology [J]. Micro Electronics & Computer, 2012, 29(5):201–204.
Foster I, Kesselman C, Tuecke S. The Anatomy of the Grid: Enabling Scalable Virtual Organizations [J]. International Journal of High Performance Computing Applications, 2001, 15(1):6.
Foster I. What is the Grid? A Three Point Checklist [J]. Grid Today, 2002, 1:32–36.
Nist S P. A NIST definition of Cloud computing [J]. Communications of the Acm, 2015, 53(6):50–50.
Hayes B. Cloud computing [J]. Communications of the Acm, 2008, 51(7):9–11.
Apache Software Foundation. Apache Nutch [EB/OL]. http://nutch.apache.org/.
YANG Junjie, LIAO Zhuofan, FENG Chaochao. Survey on big data storage framework and algorithm [J]. Journal of Computer Applications. 2016, 36(9):2465–2471.
Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters [C]. Conference on Symposium on Operating Systems Design & Implementation. USENIX Association, 2004:10–10.
Borthakur D. The Hadoop Distributed File System: Architecture and Design [J]. Hadoop Project Website, 2007, 11(11):1 – 10.
Konar M, Konar M, Konar M, et al. Apache Hadoop YARN: yet another resource negotiator [C]. Symposium on Cloud Computing. ACM, 2013:5.
Zaharia M, Chowdhury M, Franklin M J, et al. Spark: cluster computing with working sets [C]. Usenix Conference on Hot Topics in Cloud Computing. USENIX Association, 2010:10–10.
HU Jun1, HU Xian-De1, CHEN Jia-Xing. Big Data Hybrid Computing Mode Based on Spark [J]. Computer Systems & Applications, 2015, 24(4):214–218.
Apache Storm [EB/OL]. http://storm.apache.org/.
LI Chuan, E Hai-hong, SONG Mei-na. Research & Application of real-time compute framework based on Storm [J]. Software, 2014(10):16–20.
Github Inc. Storm Wiki [EB/OL]. [2014-11-02]. https://github.com/apache/storm.
Malewicz G, Austern M H, Bik A J C, et al. Pregel: a system for large-scale graph processing [C]. ACM SIGMOD International Conference on Management of Data. ACM, 2010:135–146.
Melnik S, Gubarev A, Long J J, et al. Dremel: Interactive Analysis of Web-Scale Datasets [J]. Communications of the Acm, 2011, 3(12):114–123.
Isard M, Budiu M, Yu Y, et al. Dryad: distributed data-parallel programs from sequential building blocks [J]. Acm Sigops Operating Systems Review, 2007, 41(3):59–72.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Xingang, W. (2019). A Research Review of Distributed Computing System. In: Patnaik, S., Jain, V. (eds) Recent Developments in Intelligent Computing, Communication and Devices. Advances in Intelligent Systems and Computing, vol 752. Springer, Singapore. https://doi.org/10.1007/978-981-10-8944-2_42
Download citation
DOI: https://doi.org/10.1007/978-981-10-8944-2_42
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-8943-5
Online ISBN: 978-981-10-8944-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)