Skip to main content
SpringerLink
Log in

An Improved Container Scheduling Algorithm Based on PSO for Big Data Applications

  • Conference paper
  • First Online:
Cyberspace Safety and Security (CSS 2019)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 11982))

Included in the following conference series:

  • 1150 Accesses

Abstract

Existing big data computing and storage platforms are generally based on traditional virtual machine technology, which often results in low resource utilization, a long time for flexible scaling and expanding clusters. To deal with these problems, this paper proposes an improved container scheduling algorithm, Kubernetes-based Particle Swarm Optimization (K-PSO), for big data applications based on Particle Swarm Optimization (PSO). The K-PSO algorithm converges faster than the basic PSO algorithm, and the algorithm running time is reduced by about half. The K-PSO capacity for big data applications is implemented in the Kubernetes container cloud system. The experimental results show that the node resource utilization rate of the improved scheduling strategy based on K-PSO algorithm is about 20% higher than that of Kube-Scheduler default strategy, BalancedQosPriority strategy, ESS strategy, and PSO strategy while the average I/O performance and average computing performance of Hadoop cluster are not degraded.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Felter, W, Ferreira, A, Rajamony, R, et al.: An updated performance comparison of virtual machines and linux containers. In: 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 171–172. IEEE (2015)

    Google Scholar 

  2. Alfonso, C.D., Calatrava, A., Moltó, G.: Container-based virtual elastic clusters. J. Syst. Softw. 127, 1–11 (2017)

    Article  Google Scholar 

  3. Li, Z., Zhang, Y., Liu, Y.: Towards a full-stack DevOps environment (platform-as-a-service) for cloud-hosted applications. Tsinghua Sci. Technol. 22(1), 1–9 (2017)

    Article  Google Scholar 

  4. Pahl, C., Brogi, A., Soldani, J., et al.: Cloud container technologies: a state-of-the-art review. IEEE Trans. Cloud Comput. 99, 1–1 (2017)

    Google Scholar 

  5. Gandhi, A., Thota, S., Dube, P., et al.: Autoscaling for Hadoop clusters. In: 2016 IEEE International Conference on Cloud Engineering (IC2E). IEEE (2016)

    Google Scholar 

  6. Khan, M., Jin, Y., Li, M., et al.: Hadoop performance modeling for job estimation and resource provisioning. IEEE Trans. Parallel Distrib. Syst. 99, 441–454 (2015)

    Google Scholar 

  7. Naik, N.: Docker container-based big data processing system in multiple clouds for everyone. In: 2017 IEEE International Systems Engineering Symposium (ISSE), pp. 1–7. IEEE (2017)

    Google Scholar 

  8. Xu, Z., Yang, H.: Quality of service based on Kubernetes scheduler. Softw. Guide 17(11), 77–80 (2018)

    Google Scholar 

  9. Zhang, K., Peng, L., Lu, X., et al.: Kubernetes elastic scheduling on open source cloud. Comput. Technol. Dev. 29(02), 115–120 (2019)

    Google Scholar 

  10. Weiwei, L., Dejun, Q.: Review of cloud computing resource scheduling. Comput. Sci. 39(10), 1–6 (2012)

    Google Scholar 

  11. Fernández-Baca, D.: Allocating modules to processors in a distributed system. IEEE Trans. Softw. Eng. 15(11), 1427–1436 (1989)

    Article  Google Scholar 

  12. Bernstein, D.: Containers and cloud: from LXC to Docker to Kubernetes. IEEE Cloud Comput. 1(3), 81–84 (2014)

    Article  Google Scholar 

  13. Hindman, B., Konwinski, A., Zaharia, M., et al.: Mesos: a platform for fine-grained resource sharing in the data center. In: NSDI, vol. 11, no. 2011, p. 22 (2011)

    Google Scholar 

  14. Jie, L., Guangzhong, L.: Research on automated container deployment of Hadoop distributed cluster. Comput. Appl. Res. 33(11), 3404–3407 (2016)

    Google Scholar 

  15. Liu, B., Li, P., Lin, W., et al.: A new container scheduling algorithm based on multi-objective optimization. Soft Comput. 22(23), 7741–7752 (2018)

    Article  Google Scholar 

  16. Lin, W., Wang, Z.: Docker cluster scheduling strategy based on genetic algorithm. J. S. China Univ. Technol. (Nat. Sci. Ed.) 46(3), 19 (2018)

    Google Scholar 

  17. Sujana, J.A.J., Revathi, T., Priya, T.S.S., et al.: Smart PSO-based secured scheduling approaches for scientific workflows in cloud computing. Soft Comput. 23(5), 1745–1765 (2019)

    Article  Google Scholar 

  18. Zhou, Z., Chang, J., Hu, Z., et al.: A modified PSO algorithm for task scheduling optimization in cloud computing. Concurr. Comput. Pract. Exp. 30(24), e4970. 3404–3407 (2018)

    Google Scholar 

  19. Adhikari, M., Srirama, S.N.: Multi-objective accelerated particle swarm optimization with a container-based scheduling for Internet-of-Things in cloud environment. J. Netw. Comput. Appl. 137, 35–61 (2019)

    Article  Google Scholar 

  20. Zhang, L., Tang, Y., Hua, C., et al.: A new particle swarm optimization algorithm with adaptive inertia weight based on Bayesian techniques. Appl. Soft Comput. 28, 138–149 (2015)

    Article  Google Scholar 

  21. Nobile, M.S., Cazzaniga, P., Besozzi, D., et al.: Fuzzy self-tuning PSO: a settings-free algorithm for global optimization. Swarm Evol. Comput. 39, 70–85 (2018)

    Article  Google Scholar 

  22. Taherkhani, M., Safabakhsh, R.: A novel stability-based adaptive inertia weight for particle swarm optimization. Appl. Soft Comput. 38, 281–295 (2016)

    Article  Google Scholar 

  23. Deng, W., Yao, R., Zhao, H., et al.: A novel intelligent diagnosis method using optimal LS-SVM with improved PSO algorithm. Soft Comput. 23(7), 2445–2462 (2019)

    Article  Google Scholar 

  24. Clerc, M.: The swarm and the queen: towards a deterministic and adaptive particle swarm optimization. In: Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406). IEEE (2002)

    Google Scholar 

Download references

Acknowledgment

This work is supported by National Natural Science Foundation of China (Grant Nos. 61772205, 61872084), Guangdong Science and Technology Department (Grant No. 2017B010126002), Guangzhou Science and Technology Program key projects (Grant Nos. 201802010010, 201807010052, 201902010040 and 201907010001), Nansha Science and Technology Projects (Grant No. 2017GJ001), Guangzhou Development Zone Science and Technology (Grant No. 2018GH17) and the Fundamental Research Funds for the Central Universities, SCUT (Grant No. 2019ZD26).

Author information

Authors and Affiliations

  1. School of Computer Science, South China Normal University, Guangzhou, 510000, China

    Jiawei Li, Bo Liu, Pengfei Li & Qian Gao

  2. School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510000, China

    Weiwei Lin

  3. Guangdong Luan Indusrty and Commerce Co., Ltd., Guangzhou, 510520, China

    Weiwei Lin

Authors
  1. Jiawei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weiwei Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pengfei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qian Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weiwei Lin .

Editor information

Editors and Affiliations

  1. Rutgers University, Newark, NJ, USA

    Jaideep Vaidya

  2. Beihang University, Beijing, China

    Xiao Zhang

  3. Guangzhou University, Guangzhou, China

    Jin Li

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, J., Liu, B., Lin, W., Li, P., Gao, Q. (2019). An Improved Container Scheduling Algorithm Based on PSO for Big Data Applications. In: Vaidya, J., Zhang, X., Li, J. (eds) Cyberspace Safety and Security. CSS 2019. Lecture Notes in Computer Science(), vol 11982. Springer, Cham. https://doi.org/10.1007/978-3-030-37337-5_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-37337-5_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-37336-8

  • Online ISBN: 978-3-030-37337-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation