Advertisement

An Adaptive Skew Handling Join Algorithm for Large-scale Data Analysis

  • Di Wu
  • Tengjiao WangEmail author
  • Yuxin Chen
  • Shun Li
  • Hongyan Li
  • Kai Lei
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9098)

Abstract

Join plays an essential role in large-scale data analysis, but the performance is severely degraded by data skew. Existing works can’t adaptively handle data skew very well and reduce communication cost simultaneously. To address these problems, we firstly propose a mixed data structure comprising Bloom Filter and Histogram(BFH). Based on BFH, Bloom Filter and Histogram Join(BFHJ) is proposed to handle data skew adaptively. BFHJ can reduce communication cost by filtering unnecessary records. Furthermore, BFHJ adopts a heuristic partitioning strategies to balance workload. Experiments on TPC-H demonstrate that BFHJ outperforms the state-of-the-art methods in terms of communication cost, load balance and query time.

Keywords

Skew handling join Adaptive Partitioning strategy 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Communications of the ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  2. 2.
    Walton, C.B., Dale, A.G., Jenevein, R.M.: A taxonomy and performance model of data skew effects in parallel joins. In: VLDB, vol. 91, pp. 537–548 (1991)Google Scholar
  3. 3.
    Thusoo, A., Sarma, J.S., Jain, N., Shao, Z., Chakka, P., Anthony, S., Liu, H., Wyckoff, P., Murthy, R.: Hive: a warehousing solution over a map-reduce framework. Proceedings of the VLDB Endowment 2(2), 1626–1629 (2009)CrossRefGoogle Scholar
  4. 4.
    Atta, F., Viglas, S.D., Niazi, S.: Sand join skew handling join algorithm for google’s mapreduce framework. In: 2011 IEEE 14th International on Multitopic Conference (INMIC), pp. 170–175. IEEE (2011)Google Scholar
  5. 5.
    Gates, A.: Programming Pig. O’Reilly (2011)Google Scholar
  6. 6.
    Bloom, B.H.: Space/time trade-offs in hash coding with allowable errors. Communications of the ACM 13(7), 422–426 (1970)zbMATHCrossRefGoogle Scholar
  7. 7.
    Council, T.P.P.: Tpc-h benchmark specification (2008). Published at http://www.tpc.org/tpch/

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Di Wu
    • 1
    • 3
  • Tengjiao Wang
    • 1
    • 2
    • 3
    Email author
  • Yuxin Chen
    • 2
    • 3
  • Shun Li
    • 5
  • Hongyan Li
    • 2
    • 4
  • Kai Lei
    • 1
  1. 1.School of Electronics and Computer Engineering (ECE)Peking UniversityShenzhenChina
  2. 2.School of Electronics Engineering and Computer SciencePeking UniversityBeijingChina
  3. 3.Key Laboratory of High Confidence Software TechnologiesPeking University, Ministry of EducationBeijingChina
  4. 4.Key Laboratory of Machine PerceptionPeking University, Ministry of EducationBeijingChina
  5. 5.University of International RelationsBeijingChina

Personalised recommendations