Abstract
Nowadays, with the constantly increasing amount of data, we are facing a growing number of users, who are characterized by a frequent and a massively concurrent data access. The large number of users pose multiple query optimization problems. In a distributed data warehousing system such as Hadoop/Hive, queries are evaluated one at a time and processed with the MapReduce paradigm. The massive query execution usually overloads and slows down the entire distributed environment mainly due to multiple data scan tasks. In this paper we aim to optimize the multiple query execution performance on Hive. We propose Shared Distributed Datasets (\( S2D \)), a method that dynamically looks for and shares common data among queries. The evaluation shows that, compared to Hive, \( S2D \) consumes on average 20% less memory in the Map-scan task and it is 12% faster regarding the execution time of interactive and reporting queries from TPC-DS.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Elghandour, I., Aboulnaga, A.: ReStore: reusing results of MapReduce jobs. Proc. VLDB Endowment 5(6), 586–597 (2012)
Herodotou, H.: Automatic tuning of data-intensive analytical workloads. Ph.D. thesis, Duke University (2012)
Herodotou, H., Lim, H., Luo, G., Borisov, N., Dong, L., Cetin, F.B., Babu, S.: Starfish: a self-tuning system for big data analytics. CIDR 11, 261–272 (2011)
Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., Goldberg, A.: Quincy: fair scheduling for distributed computing clusters. In: Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, pp. 261–276 (2009)
Marcus, R., Papaemmanouil, O.: WiSeDB: a learning-based workload management advisor for cloud databases. CoRR abs/1601.08221 (2016)
Nykiel, T., Potamias, M., Mishra, C., Kollios, G., Koudas, N.: MRShare: sharing across multiple queries in mapreduce. Proc. VLDB Endowment 3(1–2), 494–505 (2010)
Sellis, T.K.: Multiple-query optimization. ACM Trans. Database Syst. 13, 23–52 (1988)
Wang, G., Chan, C.-Y.: Multi-query optimization in MapReduce framework. Proc. VLDB Endowment 7, 145–156 (2013)
Zaharia, M., Borthakur, D., Sarma, J.S., Elmeleegy, K., Shenker, S., Stoica, I.: Job scheduling for multi-user mapreduce clusters. EECS Department, University of California, Berkeley, Technical report. UCB/EECS-2009-55 (2009)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings 6th Symposium on Operating System Design and Implementation OSDI, pp. 137–150 (2004)
Nambiar, R.O., Poess, M.: The making of TPC-DS. In: Proceedings of the 32nd International Conference on Very Large Data Bases, pp. 1049–1058. VLDB Endowment (2006)
Poess, M., Nambiar, R.O., Walrath, D.: Why you should run TPC-DS: a workload analysis. In: Proceedings of the 33rd International Conference on Very Large Data Bases. VLDB Endowment (2007)
Ratsimbazafy R., Boussaid O., Bentayeb F.: Stratégie pour le traitement des processus décisionnels massifs dans un big data warehouse. In: Proceedings of the EDA 2016, Aix-en-Provence (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Ratsimbazafy, R., Boussaid, O., Bentayeb, F. (2017). S2D: Shared Distributed Datasets, Storing Shared Data for Multiple and Massive Queries Optimization in a Distributed Data Warehouse. In: Bellatreche, L., Chakravarthy, S. (eds) Big Data Analytics and Knowledge Discovery. DaWaK 2017. Lecture Notes in Computer Science(), vol 10440. Springer, Cham. https://doi.org/10.1007/978-3-319-64283-3_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-64283-3_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-64282-6
Online ISBN: 978-3-319-64283-3
eBook Packages: Computer ScienceComputer Science (R0)