Advertisement

Light-Weight Cloud-Based Virtual Computing Infrastructure for Distributed Applications and Hadoop Clusters

  • Vladimir KorkhovEmail author
  • Sergey Kobyshev
  • Alexander Degtyarev
  • Alexander Bogdanov
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10408)

Abstract

Virtualized computing infrastructures are often used to create clusters of resources tailored to solve tasks taking into account particular requirements of these tasks. An important objective is to evaluate such requirements and request optimal amount of resources which becomes challenging for parallel tasks with intercommunication. In previous works we investigated how light-weight container-based virtualization can be used for creating virtual clusters running MPI applications. Such cluster is configured according to the requirements of particular application and allocates only necessary amount of resources from the physical infrastructure leaving space for co-allocated clusters running without conflicts or resource races. In this paper we investigate similar concepts for MapReduce applications based on Hadoop framework that use Cloudply virtualization tool to create and manage light-weight virtual Hadoop clusters on Amazon cloud resources. We investigate performance of several Hadoop benchmarks in different deployment scenarios and evaluate effects of resource sharing and limitation on application performance.

Keywords

Virtualization Containers Virtual cluster 

Notes

Acknowledgments

The research was supported by Russian Foundation for Basic Research (projects N 16-07-01111, 16-07-00886, 16-07-01113).

References

  1. 1.
    Korkhov, V., Kobyshev, S., Krosheninnikov, A.: Flexible configuration of application-centric virtualized computing infrastructure. In: Gervasi, O., Murgante, B., Misra, S., Gavrilova, M.L., Rocha, A.M.A.C., Torre, C., Taniar, D., Apduhan, B.O. (eds.) ICCSA 2015. LNCS, vol. 9158, pp. 342–353. Springer, Cham (2015). doi: 10.1007/978-3-319-21410-8_27 CrossRefGoogle Scholar
  2. 2.
    Korkhov, V., Kobyshev, S., Krosheninnikov, A., Degtyarev, A., Bogdanov, A.: Distributed computing infrastructure based on dynamic container clusters. In: Gervasi, O., Murgante, B., Misra, S., Rocha, A.M.A.C., Torre, C., Taniar, D., Apduhan, B.O., Stankova, E., Wang, S. (eds.) ICCSA 2016. LNCS, vol. 9787, pp. 263–275. Springer, Cham (2016). doi: 10.1007/978-3-319-42108-7_20 CrossRefGoogle Scholar
  3. 3.
    Apache Hadoop Project. http://hadoop.apache.org/
  4. 4.
    Apache Wiki: Virtual Hadoop. https://wiki.apache.org/hadoop/Virtual
  5. 5.
    Buell, J.: Virtualized Hadoop Performance with VMware vSphere 5.1. Performance Study. http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/vmware-virtualizing-apache-hadoop-white-paper.pdf
  6. 6.
    Buell, J.: Protecting Hadoop with VMware vSphere 5 Fault Tolerance. VMware Inc. (2012). http://www.vmware.com/resources/techresources/10301
  7. 7.
    Zhang, R., Li, M., Hildebrand, D.: Finding the big data sweet spot: towards automatically recommending configurations for Hadoop clusters on Docker containers. In: 2015 IEEE International Conference on Cloud Engineering, Tempe, AZ, pp. 365–368 (2015)Google Scholar
  8. 8.
    Rey, J., Cogorno, M., Nesmachnow, S., Steffenel, L.A.: Efficient prototyping of fault tolerant map-reduce applications with Docker-Hadoop. In: 2015 IEEE International Conference on Cloud Engineering, Tempe, AZ, pp. 369–376 (2015)Google Scholar
  9. 9.
    Qiao, Y., Wang, X., Fang, G., Lee, B.: Doopnet: an emulator for network performance analysis of Hadoop clusters using Docker and Mininet. In: 2016 IEEE Symposium on Computers and Communication (ISCC), Messina 2016, pp. 784–790 (2016)Google Scholar
  10. 10.
    Ivanov, T., Zicari, R., Izberovic, S., Tolle, K.: Performance evaluation of virtualized hadoop clusters. Technical report No. 2014-1, Frankfurt Big Data Lab oratory. https://arxiv.org/ftp/arxiv/papers/1411/1411.3811.pdf
  11. 11.
    Gankevich, I., Tipikin, Y., Korkhov, V., Gaiduchok, V., Degtyarev, A., Bogdanov, A.: Factory: master node high-availability for big data applications and beyond. In: Gervasi, O., Murgante, B., Misra, S., Rocha, A.M.A.C., Torre, C., Taniar, D., Apduhan, B.O., Stankova, E., Wang, S. (eds.) ICCSA 2016. LNCS, vol. 9787, pp. 379–389. Springer, Cham (2016). doi: 10.1007/978-3-319-42108-7_29 CrossRefGoogle Scholar
  12. 12.
    Gankevich, I., Tipikin, Y., Korkhov, V., Gaiduchok, V.: Factory: non-stop batch jobs without checkpointing. In: 2016 International Conference on High Performance Computing and Simulation, HPCS 2016, Art. no. 7568441, pp. 979–984 (2016)Google Scholar
  13. 13.
    Iakushkin, O.: Cloud middleware combining the functionalities of message passing and scaling control. In: EPJ Web of Conferences, vol. 108, Art. no. 02029 (2016). doi: 10.1051/epjconf/201610802029
  14. 14.
    Iakushkin, O.: Intellectual scaling in a distributed cloud application architecture: a message classification algorithm, In: Proceedings of International Conference on Stability and Control Processes in Memory of V.I. Zubov, SCP 2015, art. no. 7342245, pp. 634–637 (2015). doi: 10.1109/SCP.2015.7342245
  15. 15.
    Raba, N.O., Stankova, E.N.: On the problem of numerical modeling of dangerous convective phenomena: possibilities of real-time forecast with the help of multi-core processors. In: Murgante, B., Gervasi, O., Iglesias, A., Taniar, D., Apduhan, B.O. (eds.) ICCSA 2011. LNCS, vol. 6786, pp. 633–642. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-21934-4_51 CrossRefGoogle Scholar
  16. 16.
    Raba, N., Stankova, E., Ampilova, N.: On investigation of parallelization effectiveness with the help of multi-core processors. In: Proceedings of 10th International Conference on Computational Science (ICCS) 2010. Procedia Computer Science, vol. 1(1), pp. 2763–2768 (2010). doi: 10.1016/j.procs.2010.04.310
  17. 17.
    Bogdanov, A., Ivashchenko, A., Belezeko, A., Korkhov, V., Kulabukhova, N., Khmel, D., Suslova, S., Milova, E., Smirnov, K.: Building a virtual cluster for 3D graphics applications. In: Gervasi, O., Murgante, B., Misra, S., Rocha, A.M.A.C., Torre, C., Taniar, D., Apduhan, B.O., Stankova, E., Wang, S. (eds.) ICCSA 2016. LNCS, vol. 9787, pp. 276–291. Springer, Cham (2016). doi: 10.1007/978-3-319-42108-7_21 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Vladimir Korkhov
    • 1
    Email author
  • Sergey Kobyshev
    • 1
  • Alexander Degtyarev
    • 1
  • Alexander Bogdanov
    • 1
  1. 1.St. Petersburg State UniversitySt. PetersburgRussia

Personalised recommendations