Exploring Spark-SQL-Based Entity Resolution Using the Persistence Capability

  • Xiao ChenEmail author
  • Roman Zoun
  • Eike Schallehn
  • Sravani Mantha
  • Kirity Rapuru
  • Gunter Saake
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 928)


Entity Resolution (ER) is a task to identify records that refer to the same real-world entities. A naive way to solve ER tasks is to calculate the similarity of the Cartesian product of all records, which is called pair-wise ER and leads to quadratic time complexity. Faced with an exploding data volume, pair-wise ER is challenged to achieve high efficiency and scalability. To tackle this challenge, parallel computing is proposed for speeding up the ER process. Due to the difficulty of distributed programming, big data processing frameworks are often used as tools to ease the realization of parallel ER, supporting data partitioning, workload balancing, and fault tolerance. However, the efficiency and scalability of parallel ER is also influenced by the adopted framework. In the area of parallel ER, the adoption of Apache Spark, a general framework supporting in-memory computation, still is not widely studied. Furthermore, though Apache Spark provides both low-level (RDD-based) and high-level APIs (Datasets-based), to date, only RDD-based APIs have been adopted in parallel ER research. In this paper, we have implemented a Spark-SQL-based ER process and explored its persistence capability to see the performance benefits. We have evaluated its speedup and compared its efficiency to Spark-RDD-based ER. We observed that different persistence options have a large impact on the efficiency of Spark-SQL-based ER, requiring a careful consideration for choosing it. By adopting the best persistence option, the efficiency of our Spark-SQL-based ER implementation is improved up to 3 times on different datasets, over a baseline without any persistence option or with misconfigured persistence.


Apache Spark (Spark SQL) Entity Resolution Record linkage Data matching Parallel computing 



The authors would like to thank China Scholarship Council [No. 201408080093] to fund our work. Besides, we are very grateful to Gabriel Campero Durand, David Broneske and Yusra Shakeel to provide us valuable feedback.


  1. 1.
    Abouzeid, A., Bajda-Pawlikowski, K., Abadi, D., Silberschatz, A., Rasin, A.: HadoopDB: an architectural hybrid of MapReduce and DBMS technologies for analytical workloads. Proc. VLDB Endow. 2(1), 922–933 (2009)CrossRefGoogle Scholar
  2. 2.
    Apache: Apache spark. Accessed 10 Apr 2017
  3. 3.
    Armbrust, M., et al.: Spark SQL: relational data processing in spark. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 1383–1394. ACM (2015)Google Scholar
  4. 4.
    Benjelloun, O., et al.: D-Swoosh: a family of algorithms for generic, distributed entity resolution. In: 27th International Conference on Distributed Computing Systems, ICDCS 2007, p. 37. IEEE (2007)Google Scholar
  5. 5.
    Bowes, R.: Facebook names dataset. Accessed 15 June 2017
  6. 6.
    Chen, D., Shen, C., Feng, J., Le, J.: An efficient parallel top-k similarity join for massive multidimensional data using spark. Int. J. Database Theory Appl. 8(3), 57–68 (2015)CrossRefGoogle Scholar
  7. 7.
    Chen, X., Schallehn, E., Saake, G.: Cloud-scale entity resolution: current state and open challenges. Open J. Big Data (OJBD) 4(1), 30–51 (2018)Google Scholar
  8. 8.
    Christen, P.: Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection. Springer, Heidelberg (2012). Scholar
  9. 9.
    Christen, P., Vatsalan, D.: Flexible and extensible generation and corruption of personal data. In: Proceedings of the 22nd ACM International Conference on Information & Knowledge Management, CIKM 2013, pp. 1165–1168. ACM, New York (2013)Google Scholar
  10. 10.
    Cohen, W., Ravikumar, P., Fienberg, S.: A comparison of string metrics for matching names and records. In: KDD Workshop on Data Cleaning and Object Consolidation, vol. 3, pp. 73–78 (2003)Google Scholar
  11. 11.
    Elmagarmid, A.K., Ipeirotis, P.G., Verykios, V.S.: Duplicate record detection: a survey. IEEE Trans. Knowl. Data Eng. 19(1), 1–16 (2007)CrossRefGoogle Scholar
  12. 12.
    Getoor, L., Machanavajjhala, A.: Entity resolution for big data. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 1527. ACM (2013)Google Scholar
  13. 13.
    Hameurlain, A., Morvan, F.: Big data management in the cloud: evolution or crossroad? In: Kozielski, S., Mrozek, D., Kasprowski, P., Małysiak-Mrozek, B., Kostrzewa, D. (eds.) BDAS 2015-2016. CCIS, vol. 613, pp. 23–38. Springer, Cham (2016). Scholar
  14. 14.
    Hortonworks: Hortonworks data platform. Accessed 10 July 2017
  15. 15.
    Karau, H., Warren, R.: High Performance Spark. O’Reilly Media, Sebastopol (2017)Google Scholar
  16. 16.
    Kolb, L., Thor, A., Rahm, E.: Dedoop: efficient deduplication with Hadoop. Proc. VLDB Endow. 5(12), 1878–1881 (2012)CrossRefGoogle Scholar
  17. 17.
    Mestre, D.G., Pires, C.E.S., Nascimento, D.C., de Queiroz, A.R.M., Santos, V.B., Araujo, T.B.: An efficient spark-based adaptive windowing for entity matching. J. Syst. Softw. 128, 1–10 (2017)CrossRefGoogle Scholar
  18. 18.
    Pita, R., Pinto, C., Melo, P., Silva, M., Barreto, M., Rasella, D.: A spark-based workflow for probabilistic record linkage of healthcare data. In: EDBT/ICDT Workshops, pp. 17–26 (2015)Google Scholar
  19. 19.
    Rong, C., Lu, W., Du, X., Zhang, X.: Efficient duplicate detection on cloud using a new signature scheme. In: Wang, H., Li, S., Oyama, S., Hu, X., Qian, T. (eds.) WAIM 2011. LNCS, vol. 6897, pp. 251–263. Springer, Heidelberg (2011). Scholar
  20. 20.
    Tran, K.N., Vatsalan, D., Christen, P.: GeCo: an online personal data generator and corruptor. In: Proceedings of the 22nd ACM International Conference on Information & Knowledge Management, CIKM 2013, pp. 2473–2476. ACM, New York (2013)Google Scholar
  21. 21.
    Wang, C., Karimi, S.: Parallel duplicate detection in adverse drug reaction databases with spark. In: EDBT, pp. 551–562 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Xiao Chen
    • 1
    Email author
  • Roman Zoun
    • 1
  • Eike Schallehn
    • 1
  • Sravani Mantha
    • 2
  • Kirity Rapuru
    • 1
  • Gunter Saake
    • 1
  1. 1.Otto-von-Guericke-University of MagdeburgMagdeburgGermany
  2. 2.German Research Center For Artificial IntelligenceBerlinGermany

Personalised recommendations