Advertisement

Journal of Computer Science and Technology

, Volume 33, Issue 6, pp 1204–1218 | Cite as

Modeling Topic-Based Human Expertise for Crowd Entity Resolution

  • Sai-Sai Gong
  • Wei HuEmail author
  • Wei-Yi Ge
  • Yu-Zhong Qu
Regular Paper
  • 19 Downloads

Abstract

Entity resolution (ER) aims to identify whether two entities in an ER task refer to the same real-world thing. Crowd ER uses humans, in addition to machine algorithms, to obtain the truths of ER tasks. However, inaccurate or erroneous results are likely to be generated when humans give unreliable judgments. Previous studies have found that correctly estimating human accuracy or expertise in crowd ER is crucial to truth inference. However, a large number of them assume that humans have consistent expertise over all the tasks, and ignore the fact that humans may have varied expertise on different topics (e.g., music versus sport). In this paper, we deal with crowd ER in the Semantic Web area. We identify multiple topics of ER tasks and model human expertise on different topics. Furthermore, we leverage similar task clustering to enhance the topic modeling and expertise estimation. We propose a probabilistic graphical model that computes ER task similarity, estimates human expertise, and infers the task truths in a unified framework. Our evaluation results on real-world and synthetic datasets show that, compared with several state-of-the-art approaches, our proposed model achieves higher accuracy on the task truth inference and is more consistent with the human real expertise.

Keywords

entity resolution crowdsourcing human expertise topic modeling task similarity 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

11390_2018_1882_MOESM1_ESM.pdf (76 kb)
ESM 1 (PDF 76 kb)

References

  1. [1]
    Heflin J, Song D. Ontology instance linking: Towards interlinked knowledge graphs. In Proc. the 30th AAAI Conf. Artificial Intelligence, February 2016, pp.4163-4169.Google Scholar
  2. [2]
    Hu W, Jia C. A bootstrapping approach to entity linkage on the Semantic Web. Journal of Web Semantics, 2015, 34: 1-12.Google Scholar
  3. [3]
    Wang J, Kraska T, Franklin M J, Feng J. CrowdER: Crowdsourcing entity resolution. Proceedings of the VLDB Endowment, 2012, 5(11): 1483-1494.Google Scholar
  4. [4]
    Yalavarthi V K, Ke X, Khan A. Select your questions wisely: For entity resolution with crowd errors. In Proc. the 26th Int. Conf. Information and Knowledge Management, November 2017, pp.317-326.Google Scholar
  5. [5]
    Ma F, Li Y, Li Q, Qiu M, Gao J, Zhi S, Su L, Zhao B, Ji H, Han J. FaitCrowd: Fine grained truth discovery for crowdsourced data aggregation. In Proc. the 21st ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, August 2015, pp.745-754.Google Scholar
  6. [6]
    Yan Y, Rosales R, Fung G, Dy J G. Active learning from crowds. In Proc. the 28th Int. Conf. Machine Learning, June 2011, pp.1161-1168.Google Scholar
  7. [7]
    Raykar V C, Yu S, Zhao L H, Valadez G H, Florin C, Bogoni L, Moy L. Learning from crowds. Journal of Machine Learning Research, 2010, 11: 1297-1322.Google Scholar
  8. [8]
    Fang M, Yin J, Tao D. Active learning for crowdsourcing using knowledge transfer. In Proc. the 28th AAAI Conf. Artificial Intelligence, July 2014, pp.1809-1815.Google Scholar
  9. [9]
    Kuncheva L I, Whitaker C J, Shipp C A, Duin R P. Limits on the majority vote accuracy in classifier fusion. Pattern Analysis and Applications, 2003, 6(1): 22-31.Google Scholar
  10. [10]
    Whitehill J, Ruvolo P, Wu T, Bergsma J, Movellan J R. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proc. the 23rd Annual Conf. Neural Information Processing Systems, December 2009, pp.2035-2043.Google Scholar
  11. [11]
    Snow R, O’Connor B, Jurafsky D, Ng A Y. Cheap and fast — But is it good? Evaluating non-expert annotations for natural language tasks. In Proc. the 2008 Conf. Empirical Methods in Natural Language Processing, October 2008, pp.254-263.Google Scholar
  12. [12]
    Fan J, Li G, Ooi B C, Tan K L, Feng J. iCrowd: An adaptive crowdsourcing framework. In Proc. the 2015 ACM SIGMOD Int. Conf. Management of Data, May 2015, pp.1015-1030.Google Scholar
  13. [13]
    Blei D M, Ng A Y, Jordan M I. Latent Dirichlet allocation. Journal of Machine Learning Research, 2003, 3: 993-1022.Google Scholar
  14. [14]
    Bhattacharya I, Getoor L. A latent Dirichlet model for unsupervised entity resolution. In Proc. the 6th SIAM Int. Conf. Data Mining, April 2006, pp.47-58.Google Scholar
  15. [15]
    Li G, Wang J, Zheng Y, Franklin M J. Crowdsourced data management: A survey. IEEE Trans. Knowledge and Data Engineering, 2016, 28(9): 2296-2319.Google Scholar
  16. [16]
    Li G, Zheng Y, Fan J, Wang J, Cheng R. Crowdsourced data management: Overview and challenges. In Proc. the 2017 ACM SIGMOD Int. Conf. Management of Data, May 2017, pp.1711-1716.Google Scholar
  17. [17]
    Acosta M, Zaveri A, Simperl E, Kontokostas D, Auer S, Lehmann J. Crowdsourcing linked data quality assessment. In Proc. the 12th Int. Semantic Web Conf., October 2013, pp.260-276.Google Scholar
  18. [18]
    Demartini G, Difallah D E, Cudré-Mauroux P. ZenCrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In Proc. the 21st Int. Conf. World Wide Web, April 2012, pp.469-478.Google Scholar
  19. [19]
    Chai C, Li G, Li J, Deng D, Feng J. Cost effective crowdsourced entity resolution: A partial-order approach. In Proc. the 2016 ACM SIGMOD Int. Conf. Management of Data, June 2016, pp.969-984.Google Scholar
  20. [20]
    Vesdapunt N, Bellare K, Dalvi N. Crowdsourcing algorithms for entity resolution. Proceedings of the VLDB Endowment, 2014, 7(12): 1071-1082.Google Scholar
  21. [21]
    Hassan U, Zaveri A, Marx E, Curry E, Lehmann J. ACRyLIQ: Leveraging DBpedia for adaptive crowdsourcing in linked data quality assessment. In Proc. the 20th Int. Conf. Knowledge Engineering and Knowledge Management, November 2016, pp.681-696.Google Scholar
  22. [22]
    Kontokostas D, Zaveri A, Auer S, Lehmann J. TripleCheck-Mate: A tool for crowdsourcing the quality assessment of linked data. In Proc. the 4th Int. Conf. Knowledge Engineering and the Semantic Web, October 2013, pp.265-272.Google Scholar
  23. [23]
    Fang Y L, Sun H L, Chen P P, Deng T. Improving the quality of crowdsourced image labeling via label similarity. Journal of Computer Science and Technology, 2017, 32(5): 877-889.Google Scholar
  24. [24]
    Zhuang Y, Li G, Zhong Z, Feng J. Hike: A hybrid humanmachine method for entity alignment in large-scale knowledge bases. In Proc. the 2017 Int. Conf. Information and Knowledge Management, November 2017, pp.1917-1926.Google Scholar
  25. [25]
    Li G, Chai C, Fan J, Weng X, Li J, Zheng Y, Li Y, Yu X, Zhang X, Yuan H. CDB: Optimizing queries with crowdbased selections and joins. In Proc. the 2017 ACM SIGMOD Int. Conf. Management of Data, May 2017, pp.1463-1478.Google Scholar
  26. [26]
    Zheng Y, Cheng R, Maniu S, Mo L. On optimality of jury selection in crowdsourcing. In Proc. the 18th Int. Conf. Extending Database Technology, March 2015, pp.193-204.Google Scholar
  27. [27]
    Li Q, Ma F, Gao J, Su L, Quinn C J. Crowdsourcing high quality labels with a tight budget. In Proc. the 9th ACM Int. Conf. Web Search and Data Mining, February 2016, pp.237-246.Google Scholar
  28. [28]
    Yuan D, Li G, Li Q, Zheng Y. Sybil defense in crowdsourcing platforms. In Proc. the 2017 Int. Conf. Information and Knowledge Management, November 2017, pp.1529-1538.Google Scholar
  29. [29]
    Li Q, Li Y, Gao J, Zhao B, Fan W, Han J. Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation. In Proc. the 2014 ACM SIGMOD Int. Conf. Management of Data, June 2014, pp.1187-1198.Google Scholar
  30. [30]
    Xiao H, Gao J, Li Q, Ma F, Su L, Feng Y, Zhang A. Towards confidence in the truth: A bootstrapping based truth discovery approach. In Proc. the 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, August 2016, pp.1935-1944.Google Scholar
  31. [31]
    Ma F, Meng C, Xiao H, Li Q, Gao J, Su L, Zhang A. Unsupervised discovery of drug side-effects from heterogeneous data sources. In Proc. the 23rd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, August 2017, pp.967-976.Google Scholar
  32. [32]
    Wang Y, Ma F, Su L, Gao J. Discovering truths from distributed data. In Proc. the 2017 IEEE Int. Conf. Data Mining, November 2017, pp.505-515.Google Scholar
  33. [33]
    Meng C, Jiang W, Li Y, Gao J, Su L, Ding H, Cheng Y. Truth discovery on crowd sensing of correlated entities. In Proc. the 13th ACM Conf. Embedded Networked Sensor Systems, November 2015, pp.169-182.Google Scholar
  34. [34]
    Zhang H, Li Q, Ma F, Xiao H, Li Y, Gao J, Su L. Influenceaware truth discovery. In Proc. the 25th ACM Int. Conf. Information and Knowledge Management, October 2016, pp.851-860.Google Scholar
  35. [35]
    Hu H, Zheng Y, Bao Z, Li G, Feng J, Cheng R. Crowdsourced POI labelling: Location-aware result inference and task assignment. In Proc. the 32nd IEEE Int. Conf. Data Engineering, May 2016, pp.61-72.Google Scholar
  36. [36]
    Zheng Y, Wang J, Li G, Cheng R, Feng J. QASCA: A quality-aware task assignment system for crowdsourcing applications. In Proc. the 2015 ACM SIGMOD Int. Conf. Management of Data, May 2015, pp.1031-1046.Google Scholar
  37. [37]
    Fang M, Zhu X, Li B, Ding W, Wu X. Self-taught active learning from crowds. In Proc. the 12th IEEE Int. Conf. Data Mining, December 2012, pp.858-863.Google Scholar
  38. [38]
    Zheng Y, Li G, Cheng R. DOCS: Domain-aware crowdsourcing system. Proceedings of the VLDB Endowment, 2016, 10(4): 361-372.Google Scholar
  39. [39]
    Zheng Y, Li G, Li Y, Shan C, Cheng R. Truth inference in crowdsourcing: Is the problem solved? Proceedings of the VLDB Endowment, 2017, 10(5): 541-552.Google Scholar
  40. [40]
    Li Y, Gao J, Meng C, Li Q, Su L, Zhao B, Fan W, Han J. A survey on truth discovery. ACM SIGKDD Explorations Newsletter, 2016, 17(2): 1-16.Google Scholar
  41. [41]
    Wainwright M J, Jordan M I. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 2008, 1(1/2): 1-305.Google Scholar
  42. [42]
    Qu Y, Gong S, Cheng G, Xu J, Li X, Zheng L, Jiang J. SView: Smart views for browsing linked entities. In Proc. ISWC Semantic Web Challenge 2014, October 2014.Google Scholar
  43. [43]
    Köpcke H, Thor A, Rahm E. Evaluation of entity resolution approaches on real-world match problems. Proceedings of the VLDB Endowment, 2010, 3(1): 484-493.Google Scholar
  44. [44]
    Kejriwal M, Miranker D P. An unsupervised instance matcher for schema-free RDF data. Journal of Web Semantics, 2015, 35: 102-123.Google Scholar
  45. [45]
    Abdullah M B. On a robust correlation coefficient. The Statistician, 1990, 39(4): 455-460.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.State Key Laboratory for Novel Software TechnologyNanjing UniversityNanjingChina
  2. 2.Science and Technology on Information Systems Engineering LaboratoryNanjingChina

Personalised recommendations