Abstract
Nowadays, Big data processing in cloud is becoming an inevitable trend. For Big data processing, a specially designed cloud resource allocation approach is required. However, it is challenging how to efficiently allocate resources dynamically based on Big data applications’ QoS demands and support energy and cost savings by optimizing the number of servers in use. In order to solve this problem, a general problem formulation is established in this paper. By giving certain assumptions, we prove that the reduction of resource waste has a direct relation with cost minimization. Based on that, we develop efficient heuristic algorithms with tuning parameters to find cost minimized dynamic resource allocation solutions for the above-mentioned problem. In paper, we study and test the workload of Big data by running a group of typical Big data jobs, i.e., video surveillance services, on Amazon Cloud EC2. Then we create a large simulation scenario and compare our proposed method with other approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Demchenko, Y., Zhao, Z., Grosso, P., Wibisono, A., de Laat, C.: Addressing big data challenges for scientific data infrastructure. In: 2012 IEEE 4th International Conference on Cloud Computing Technology and Science (CloudCom), pp. 614–617. IEEE (2012)
Ji, C., Li, Y., Qiu, W., Awada, U., Li, K.: Big data processing in cloud computing environments. In: 2012 12th International Symposium on Pervasive Systems, Algorithms and Networks (ISPAN), pp. 17–23. IEEE (2012)
Guo, S., Xiong, J., Wang, W., Lee, R.: Mastiff: A mapreduce-based system for time-based big data analytics. In: 2012 IEEE International Conference on Cluster Computing (CLUSTER), pp. 72–80. IEEE (2012)
Zhang, G., Li, C., Zhang, Y., Xing, C., Yang, J.: An efficient massive data processing model in the cloud – a preliminary report. In: 2012 Seventh ChinaGrid Annual Conference (ChinaGrid), pp. 148–155 (2012)
Speitkamp, B., Bichler, M.: A mathematical programming approach for server consolidation problems in virtualized data centers. IEEE Transactions on Services Computing 3(4), 266–278 (2010)
Guo, J., Zhu, Z.-M., Zhou, X.-M., Zhang, G.-X.: An instances placement algorithm based on disk i/o load for big data in private cloud. In: 2012 International Conference on Wavelet Active Media Technology and Information Processing (ICWAMTIP), pp. 287–290 (2012)
Kaushik, R.T., Nahrstedt, K.: T: a data-centric cooling energy costs reduction approach for big data analytics cloud. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, p. 52. IEEE Computer Society Press (2012)
Mo, X., Wang, H.: Asynchronous index strategy for high performance real-time big data stream storage. In: 2012 3rd IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC), pp. 232–236. IEEE (2012)
Jung, N.G., Gnanasambandam, Mukherjee, T.: Synchronous parallel processing of big-data analytics services to optimize performance in federated clouds. In: 2012 IEEE 5th International Conference on Cloud Computing (CLOUD), pp. 811–818 (2012)
Rahman, M., Li, X., Palit, H.: Hybrid heuristic for scheduling data analytics workflow applications in hybrid cloud environment. In: 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum (IPDPSW), pp. 966–974. IEEE (2011)
Ferreto, T.C., Netto, M.A.S., Calheiros, R.N., De Rose, C.A.F.: Server consolidation with migration control for virtualized data centers. Future Gener. Comput. Syst. 27, 1027–1034 (2011)
Jain, N., Menache, I., Naor, J., Yaniv, J.: Near-optimal scheduling mechanisms for deadline-sensitive jobs in large computing clusters. In: Proceedings of the 24th ACM Symposium on Parallelism in Algorithms and Architectures, pp. 255–266. ACM (2012)
Kou, L.T., Markowsky, G.: Multidimensional bin packing algorithms. IBM J. Res. Dev. 21, 443–448 (1977)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Hassan, M.M., Song, B., Hossain, M.S., Alamri, A. (2014). Efficient Resource Scheduling for Big Data Processing in Cloud Platform. In: Fortino, G., Di Fatta, G., Li, W., Ochoa, S., Cuzzocrea, A., Pathan, M. (eds) Internet and Distributed Computing Systems. IDCS 2014. Lecture Notes in Computer Science, vol 8729. Springer, Cham. https://doi.org/10.1007/978-3-319-11692-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-11692-1_5
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-11691-4
Online ISBN: 978-3-319-11692-1
eBook Packages: Computer ScienceComputer Science (R0)