Advertisement

Performance Modeling of Spark Computing Platform

  • Jie Ding
  • Yunyue XieEmail author
  • Meihua Zhou
Chapter
Part of the Studies in Computational Intelligence book series (SCI, volume 810)

Abstract

Big Data has been widely used in all aspects of society. For solving the problem of massive data storing and analyzing, many big data solutions have been proposed. Spark is the newer solution of the universal parallel framework which like Hadoop MapReduce. Compare the Hadoop, Spark’s performance has been increased significantly. As a data analysis framework, researchers are particularly concerned about its performance. So in this paper, we use a stochastic process algebra (PEPA) to model the Spark architecture. This model will give the usability of the compositional approach in modeling and analysis Spark architecture. This research obtains an algorithm that generated the service flow of the PEPA model. In the end, we will state the benefit of this compositional method in modeling a large parallel system.

Keywords

Big Data Spark Stochastic process algebra Performance evaluation 

Notes

Acknowledgements

The authors acknowledge the financial support by the National Natural Science Foundation of China under Grant 61472343.

References

  1. 1.
    What Is Apache Hadoop? May 2018. http://hadoop.apache.org/
  2. 2.
    Chandakanna, V.R.: REHDFS: a random read/write enhanced HDFS. J. Netw. Comput. Appl. 103, 85–100 (2018)CrossRefGoogle Scholar
  3. 3.
    Won, H., Nguyen, M.C., Gil, M.S., Moon, Y.S.: Advanced resource management with access control for multitenant hadoop. J. Commun. Netw. 17(6), 592–601 (2015)CrossRefGoogle Scholar
  4. 4.
    Serikawa, S., Huimin, L.: Underwater image dehazing using joint trilateral filter. Comput. Electr. Eng. 40(1), 41–50 (2014)CrossRefGoogle Scholar
  5. 5.
    Lu, H., Li, Y., Mu, S., Wang, D., Kim, H., Serikawa, S.: Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet Things J. (2017).  https://doi.org/10.1109/JIOT.2017.2737479
  6. 6.
    Lu, H., Li, Y., Chen, M., Kim, H., Serikawa, S.: Brain intelligence: go beyond artificial intelligence. Mob. Netw. Appl. 1–8 (2017)Google Scholar
  7. 7.
    Lu, H., Bin, L., Zhu, J., Serikawa, S.: Wound intensity correction and segmentation with convolutional neural networks. Concurr. Comput.: Pract. Exp. (2017).  https://doi.org/10.1002/cpe.3927
  8. 8.
    Lu, H., Li, Y., Uemura, T., Kim, H., Serikawa, S.: Low illumination underwater light field images reconstruction using deep convolutional neural networks. Future Gener. Comput. Syst. 142–148 (2018).  https://doi.org/10.1016/j.future.2018.01.001
  9. 9.
    Xu, X., He, L., Lu, H., Gao, L., Ji, Y.: Deep adversarial metric learning for cross-modal retrieval. World Wide Web J. (2018).  https://doi.org/10.1007/s11280-018-0541-x
  10. 10.
    Hillston, J.: A Compositional Approach to Performance Modelling. Cambridge University Press (1996)Google Scholar
  11. 11.
    Ding, J., Wang, R., Chen, X.: Performance modeling and evaluation of real-time traffic status query for intelligent traffic systems. In: Proceedings of 2016 22nd Asia-Pacific Conference on Communications (APCC) (2016)Google Scholar
  12. 12.
    Ding, J.: A comparison of fluid approximation and stochastic simulation for evaluating content adaptation systems. Wirel. Pers. Commun. 84, 231–250 (2015)CrossRefGoogle Scholar
  13. 13.
    Ding, J., Zhu, X., Wang, M.: Fluid analysis for a PEPA model. In: Proceedings of the 2015 Chinese Intelligent Systems Conference, vol. 2, pp. 181–190 (2015)Google Scholar
  14. 14.
    Hillston, J.: Fluid flow approximation of PEPA models. In: Second International Conference on the Quantitative Evaluation of Systems (2005)Google Scholar
  15. 15.
    Molloy, M.K.: Performance analysis using stochastic Petri Nets. IEEE Trans. Comput. 31, 913–917 (1982)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of Information EngineeringYangzhou UniversityYangzhouChina

Personalised recommendations