Performance Modeling of Spark Computing Platform
Big Data has been widely used in all aspects of society. For solving the problem of massive data storing and analyzing, many big data solutions have been proposed. Spark is the newer solution of the universal parallel framework which like Hadoop MapReduce. Compare the Hadoop, Spark’s performance has been increased significantly. As a data analysis framework, researchers are particularly concerned about its performance. So in this paper, we use a stochastic process algebra (PEPA) to model the Spark architecture. This model will give the usability of the compositional approach in modeling and analysis Spark architecture. This research obtains an algorithm that generated the service flow of the PEPA model. In the end, we will state the benefit of this compositional method in modeling a large parallel system.
KeywordsBig Data Spark Stochastic process algebra Performance evaluation
The authors acknowledge the financial support by the National Natural Science Foundation of China under Grant 61472343.
- 1.What Is Apache Hadoop? May 2018. http://hadoop.apache.org/
- 5.Lu, H., Li, Y., Mu, S., Wang, D., Kim, H., Serikawa, S.: Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet Things J. (2017). https://doi.org/10.1109/JIOT.2017.2737479
- 6.Lu, H., Li, Y., Chen, M., Kim, H., Serikawa, S.: Brain intelligence: go beyond artificial intelligence. Mob. Netw. Appl. 1–8 (2017)Google Scholar
- 7.Lu, H., Bin, L., Zhu, J., Serikawa, S.: Wound intensity correction and segmentation with convolutional neural networks. Concurr. Comput.: Pract. Exp. (2017). https://doi.org/10.1002/cpe.3927
- 8.Lu, H., Li, Y., Uemura, T., Kim, H., Serikawa, S.: Low illumination underwater light field images reconstruction using deep convolutional neural networks. Future Gener. Comput. Syst. 142–148 (2018). https://doi.org/10.1016/j.future.2018.01.001
- 9.Xu, X., He, L., Lu, H., Gao, L., Ji, Y.: Deep adversarial metric learning for cross-modal retrieval. World Wide Web J. (2018). https://doi.org/10.1007/s11280-018-0541-x
- 10.Hillston, J.: A Compositional Approach to Performance Modelling. Cambridge University Press (1996)Google Scholar
- 11.Ding, J., Wang, R., Chen, X.: Performance modeling and evaluation of real-time traffic status query for intelligent traffic systems. In: Proceedings of 2016 22nd Asia-Pacific Conference on Communications (APCC) (2016)Google Scholar
- 13.Ding, J., Zhu, X., Wang, M.: Fluid analysis for a PEPA model. In: Proceedings of the 2015 Chinese Intelligent Systems Conference, vol. 2, pp. 181–190 (2015)Google Scholar
- 14.Hillston, J.: Fluid flow approximation of PEPA models. In: Second International Conference on the Quantitative Evaluation of Systems (2005)Google Scholar