Cluster Computing

, Volume 22, Supplement 6, pp 14061–14071 | Cite as

Towards optimal resource provisioning for Hadoop-MapReduce jobs using scale-out strategy and its performance analysis in private cloud environment

  • Ramakrishnan RamanathanEmail author
  • B. Latha


Cloud computing always provides IT resources on demand basis, without additional waiting time. Therefore, data analytics is one of the most significant areas that can be benefited from Cloud Computing. MapReduce programs in the cloud computing to optimize the resource provisioning and finish the MapReduce jobs with quantified time. The efficacy as well as the accuracy of performance of the performance model based on regression used for predicting the MapReduce job completion time has been suggested in our OpenStack private cloud Hadoop cluster using linear regression method. In order to satisfy the user jobs with deadline requirements, Cloud service providers do not have a resource provisioning technique or polices. The contemporary system requires a cloud user to estimate the needed quantity of resources for running jobs in the cloud. Our proposed scalability strategy of Scale-Out methods used to obtain the accurate prediction of job completion times through our experimental results shows the performance level of MapReduce benchmark in the open stack private cloud. The regression based performance model predicting and evaluating the execution time of 5 popular MapReduce benchmark applications over our private cloud environment with better resource utilization which depicts 99% of accuracy results.


Cloud Computing Hadoop MapReduce Performance Model Resource Provisioning Efficiency 


  1. 1.
    Khan, M., Jin, Y., Li, M., Xiang, Y., Jiang, C.: Hadoop performance modeling for job estimation and resource provisioning. IEEE Trans. Parallel Distrib. Syst. 27(2), 441–454 (2016)CrossRefGoogle Scholar
  2. 2.
    Lin, X., Meng, Z., Xu, C., Wang, M.: A practical performance model for Hadoop MapReduce. In: Proceedings of IEEE International Conference on Cluster Computing. Workshops, pp. 231–239 (2012)Google Scholar
  3. 3.
    Cui, X., Lin, X., Hu, C., Zhang, R., Wang, C.: Modeling the performance of MapReduce under resource contentions and task failures. In: Proceedings of IEEE 5th International Conference on Cloud Computing Technology and Science, vol. 1, pp. 158–163 (2013)Google Scholar
  4. 4.
    Liu, J., Zhang, Y., Zhou, Y., Zhang, D., Liu, H.: Aggressive resource provisioning for ensuring QoS in virtualized environments. IEEE Trans. Cloud Comput. 3(2), 119–131 (2015)CrossRefGoogle Scholar
  5. 5.
    Mao, M., Humphrey, M.: Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In: IEEE Xplore SC ‘11, Proceedings of 2011 International Conference for High Performance Computing, Networking Storage and Analysis, p. 49 (2011)Google Scholar
  6. 6.
    Zhang, Q., Cherkasova, L., Smimi, E.: Regression-based analytic model for dynamic resource provisioning of multi-tier applications. In: Proceedings of the Fourth International conference on Autonomic Computing, Jacksonville, Florida, USA (2007)Google Scholar
  7. 7.
    Davis, I.J., Hemmati, H., Holt, R.C., Godfrey, M.W., Neuse D.M., Mankovskii, S.: Regression-based utilization prediction algorithms: an empirical investigation. CASCON’13 Proceedings of the 2013, ACM, (2013)Google Scholar
  8. 8.
    Marshall, P., Keahey, K., Freeman,T.: Elastic site using clouds to elastically extend site resources. InCluster, Cloud and Grid Computing (CCGrid), 2010 10th IEEE/ACM International Conference on IEEE, pp. 43–52 (2010)Google Scholar
  9. 9.
    Hwang, K., Bai, X., Shi, Y., Li, M., Chen, W.G., Wu, Y.: Cloud performances modeling with benchmark evaluation of elastic scaling strategies. IEEE Trans. Parallel Distrib. Syst. 27(1), 130–143 (2016)CrossRefGoogle Scholar
  10. 10.
    Ostermann, S., Iosup, A., Yigitbasi, N., Prodan, R., Fahringer, T., Epema, D.: A performance analysis of EC2 cloud computing services for scientific computing, International Conference on Cloud Computing, vol. 34, pp. 115–131. Springer, New York (2009)CrossRefGoogle Scholar
  11. 11.
    Chen, K., Powers, J., Guo, S., Tian, F.: CRESP: towards optimal resource provisioning for MapReduce computing in public clouds. IEEE Trans. Parallel Distrib. Syst. 25(6), 1403–1412 (2014)CrossRefGoogle Scholar
  12. 12.
    Li, D., Chen, C., Guan, J., Zhang, Y., Zhu, J., Yu, R.: DCloud: deadline-aware resource allocation for cloud computing jobs. IEEE Trans. Parallel Distrib. Syst. 27(8), 2248–2260 (2016)CrossRefGoogle Scholar
  13. 13.
    da Rosa Right, R., Rodrigues, V.F., Da Costa, C.A., Galante, G., de Bona, L.C.E., Ferreto, T.: AutoElastic:automatic resource elasticity for high performance applications in the cloud. IEEE Trans. Cloud Comput. 4(1), 16–19 (2016)Google Scholar
  14. 14.
    Rodriguez, M.A., Buyya, R.: Deadline based resource provisioning and scheduling algorithm for scientific workflows on cloud. IEEE Trans. Cloud Comput. 2(2), 222–235 (2014)CrossRefGoogle Scholar
  15. 15.
    Mashayekhy, L., Nejad, M.M., Grosu, D.: A PTAS mechanism for provisioning and allocation of heterogeneous cloud resources. IEEE Trans. Parallel Distrib. Syst. 26(9), 2386–2399 (2015)CrossRefGoogle Scholar
  16. 16.
    Dai, W., Bassiouni, M.: An improved task assignment scheme for Hadoop running in the clouds. J. Cloud Comput. 2(1), 23 (2013)CrossRefGoogle Scholar
  17. 17.
    Pastorelli, M., Carra, D., Dell Amico, M., Michiardi, P.: HFSP: bringing size-based scheduling to hadoop. IEEE Trans. Cloud Comput. 5(1), 43–56 (2013)CrossRefGoogle Scholar
  18. 18.
    Calheiros, R.N., Masoumi, E., Ranjan, R., Buyya, R.: Workload prediction using ARIMA model and its impact on cloud applications’ QoS. IEEE Trans. Cloud Comput. 3(4), 449–458 (2015)CrossRefGoogle Scholar
  19. 19.
    Ji, C., Li, Y., Qiu, W., Awada, U., Li, K: Big data processing in cloud computing environments. In: International Symposium Pervasive Systems, Algorithms and Networks, pp. 17–23 (2012)Google Scholar
  20. 20.
    Zhang, Z., Cherkasova, L., Loo, B.T.: Performance modeling of MapReduce jobs in heterogeneous cloud environments. In: IEEE Sixth international Conference on Cloud Computing (2013)Google Scholar
  21. 21.
    Assuncao, M.D., Calheiros, R.N., Bianchi, S., Netto, M.A., Buyya, R.: Big data computing and clouds: trends and future directions. J. Parallel Distrib. Comput. 79, 3–15 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Information TechnologyDhanalakshmi College of EngineeringChennaiIndia
  2. 2.Department of Computer Science and EngineeringSri Sairam Engineering CollegeChennaiIndia

Personalised recommendations