Abstract
Multiway spatial join has drawn significant interest in research community because of its wide range of applications. Multiway spatial join further enjoys lots of applications in location based services. The analysis of communication cost is vital in the performance analysis of computing distributed multiway spatial join due to the skew observed in real world data. We analyze the performance of multiway spatial join using two strategies for addressing skew (a) whether to have a constraint on the number of reducers or (b) to have a constraint on the size of the input to the reducer (reducer is a computing facility). Our study gives a solution to address the issue of skew and to minimize the cost for communication in a network. We propose two algorithms, which study the trade-offs between the two strategies. We conducted experiments on real world datasets shows the performance in various scenarios. Based on the learning we provide insights into the selection of appropriate strategies for a given task.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Afrati, F.N., Stasinopoulos, N., Ullman, J.D., Vassilakopoulos, A.: SharesSkew: an algorithm to handle skew for joins in mapreduce. Inform. Syst. 77, 129–150 (2018)
Afrati, F.N., Ullman, J.D.: Optimizing joins in a map-reduce environment. In: Proceedings of the 13th International Conference on Extending Database Technology, pp. 99–110. ACM (2010)
Beame, P., Koutris, P., Suciu, D.: Communication steps for parallel query processing. In: Proceedings of the 32nd ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, pp. 273–284. ACM (2013)
Beame, P., Koutris, P., Suciu, D.: Skew in parallel query processing. In: Proceedings of the 33rd ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp. 212–223. ACM (2014)
Cheng, L., Kotoulas, S., Liu, Q., Wang, Y.: Load-balancing distributed outer joins through operator decomposition. J. Parallel Distrib. Comput. (2019)
Cho, E., Myers, S.A., Leskovec, J.: Friendship and mobility: user movement in location-based social networks. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1082–1090. ACM (2011)
Chu, S., Balazinska, M., Suciu, D.: From theory to practice: efficient join query evaluation in a parallel database system. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 63–78. ACM (2015)
Gavagsaz, E., Rezaee, A., Javadi, H.H.S.: Load balancing in join algorithms for skewed data in mapreduce systems. J. Supercomput. 75(1), 228–254 (2019)
Irandoost, M.A., Rahmani, A.M., Setayeshi, S.: MapReduce data skewness handling: a systematic literature review. Int. J. Parallel Program. 1–44 (2019)
Joglekar, M., Re, C.: It’s all a matter of degree: using degree information to optimize multiway joins. arXiv preprint arXiv:1508.01239 (2015)
Koutris, P., Beame, P., Suciu, D.: Worst-case optimal algorithms for parallel query processing. In: LIPIcs-Leibniz International Proceedings in Informatics, vol. 48. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2016)
Kwon, Y., Balazinska, M., Howe, B., Rolia, J.: Skewtune in action: mitigating skew in mapreduce applications. Proc. VLDB Endow. 5(12), 1934–1937 (2012)
Ngo, H.Q., Ré, C., Rudra, A.: Skew strikes back: new developments in the theory of join algorithms. arXiv preprint arXiv:1310.3314 (2013)
Shi, Y., Qian, K.: LBMM: a load balancing based task scheduling algorithm for cloud. In: Future of Information and Communication Conference, pp. 706–712. Springer (2019)
Wang, Z., Chen, Q., Suo, B., Pan, W., Li, Z.: Reducing partition skew on mapreduce: an incremental allocation approach. Front. Comput. Sci. 13(5), 960–975 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Potluri, A., Bhattu, S.N., Kumar, N.V.N., Subramanyam, R.B.V. (2020). Design Strategies for Handling Data Skew in MapReduce Framework. In: Smys, S., Bestak, R., Rocha, Á. (eds) Inventive Computation Technologies. ICICIT 2019. Lecture Notes in Networks and Systems, vol 98. Springer, Cham. https://doi.org/10.1007/978-3-030-33846-6_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-33846-6_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33845-9
Online ISBN: 978-3-030-33846-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)