Cognition and Statistical-Based Crowd Evaluation Framework for ER-in-House Crowdsourcing System: Inbound Contact Center

  • Morteza SaberiEmail author
  • Omar Khadeer Hussain
  • Naeem Khalid Janjua
  • Elizabeth Chang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9093)


Entity identification and resolution has been a hot topic in computer science from last three decades. The ever increasing amount of data and data quality issues such as duplicate records pose great challenge to organizations to efficiently and effectively perform their business operations such as customer relationship management, marketing, contact centers management etc. Recently, crowdsourcing technique has been used to improve the accuracy of entity resolution that make use of human intelligence to label the data and make it ready for further processing by entity resolution (ER) algorithms. However, labelling of data by humans is an error prone process that affects the process of entity resolution and eventually overall performance of crowd. Thus controlling the quality of labeling task is an essential for crowdsourcing systems. However, this task becomes more challenging due to unavailability of ground data. In this paper, we address the above mentioned challenge and design and develop framework for evaluating performance of ER-In-house crowdsourcing system using cognition and statistical-based techniques. Our methodology is divided into two phases namely before-hand evaluation and in-process evaluation. In before-hand evaluation a cognitive approach is used to filter out workers with an inappropriate cognitive style for ER-labeling task. To this end, analytic hierarchy process (AHP) is used to classify the existing four primary cogitative styles discussed in the literature either as suitable or not-suitable for labelling task under consideration. To control the quality of work by crowd-workers, we extend and use the statistical approach proposed by Joglekar et al. during second phase i.e. in-process evaluation. To illustrate effectiveness of our approach; we have considered the domain of Inbound Contact Center and using Customer Service Representatives (CSRs) knowledge for ER-labeling task. In the proposed ER-In-house crowdsourcing system CSRs are considered as crowd-workers. Synthetic dataset is used to demonstrate the applicability of the proposed cognition and statistical-based CSRs evaluation approaches.


Contact centers Crowd evaluation Entity resolution Cognitive styles 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hart, M., Mwendia, K., Singh, I.: Managing knowledge about customers in inbound contact centres. In: Proceedings of the European Conference on Knowledge Management. ECKM 2009Google Scholar
  2. 2.
    Reichheld, F.F.: Loyalty rules!: How today’s leaders build lasting relationships. Harvard Business Press (2001)Google Scholar
  3. 3.
    Millard, N.: Learning from the ‘wow’factor—how to engage customers through the design of effective affective customer experiences. BT Technology Journal 24(1), 11–16 (2006)CrossRefGoogle Scholar
  4. 4.
    LaValle, S., Lesser, E., Shockley, R., Hopkins, M., Kruschwitz, N.: Big Data, Analytics and the Path From Insights to Value. MIT Sloan Management Review 52(2), 21–32 (2011)Google Scholar
  5. 5.
    Kim, W., Choi, B.-J., Hong, E.-K., Kim, S.-K., Lee, D.: A taxonomy of dirty data. Data mining and knowledge discovery 7(1), 81–99 (2003)CrossRefMathSciNetGoogle Scholar
  6. 6.
    Turing, A.M.: Computing machinery and intelligence. Mind, 433–460 (1950)Google Scholar
  7. 7.
    Davidson, S.B., Khanna, S., Milo, T., Roy, S.: Using the crowd for top-k and group-by queries. In: Book Using the Crowd for Top-k and Group-by Queries, pp. 225–236. ACM (2013)Google Scholar
  8. 8.
    Wang, F.-Y., Carley, K.M., Zeng, D., Mao, W.: Social computing: From social informatics to social intelligence. Intelligent Systems, IEEE 22(2), 79–83 (2007)CrossRefGoogle Scholar
  9. 9.
    Szuba, T.M.: Computational collective intelligence. John Wiley & Sons, Inc. (2001)Google Scholar
  10. 10.
    Sarma, A.D., Parameswaran, A., Garcia-Molina, H., Halevy, A.: Finding with the crowd. In: Book Finding with the Crowd (2012)Google Scholar
  11. 11.
    Brabham, D.C.: Crowdsourcing as a model for problem solving an introduction and cases. Convergence: the international journal of research into new media technologies 14(1), 75–90 (2008)Google Scholar
  12. 12.
    Yi, J., Jin, R., Jain, A.K., Jain, S.: Crowdclustering with sparse pairwise labels: a matrix completion approach. In: Book Crowdclustering with Sparse Pairwise Labels: A Matrix Completion Approach, pp. 1–7 (2012)Google Scholar
  13. 13.
    Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, B., White, S.: Vizwiz: nearly real-time answers to visual questions. In: Book Vizwiz: Nearly Real-Time Answers to Visual Questions, pp. 333–342. ACM (2010)Google Scholar
  14. 14.
    Pak, A., Paroubek, P.: Twitter as a corpus for sentiment analysis and opinion mining. In: Book Twitter as a Corpus for Sentiment Analysis and Opinion Mining, pp. 1320–1326 (2010)Google Scholar
  15. 15.
    Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In: Book Cheap and Fast—but is it Good?: Evaluating Non-Expert Annotations for Natural Language Tasks, pp. 254–263. Association for Computational Linguistics (2008)Google Scholar
  16. 16.
    Kittur, A., Chi, E.H., Suh, B.: Crowdsourcing user studies with mechanical turk. In: Book Crowdsourcing User Studies with Mechanical Turk, pp. 453–456. ACM (2008)Google Scholar
  17. 17.
    Mason, W., Suri, S.: Conducting behavioral research on Amazon’s Mechanical Turk. Behavior research methods 44(1), 1–23 (2012)CrossRefGoogle Scholar
  18. 18.
    Schmidt, L.: Crowdsourcing for human subjects research. In: Proceedings of CrowdConf (2010)Google Scholar
  19. 19.
    Whang, S.E., Lofgren, P., Garcia-Molina, H.: Question selection for crowd entity resolution. Proceedings of the VLDB Endowment 6(6), 349–360 (2013)CrossRefGoogle Scholar
  20. 20.
    Doan, A., Franklin, M.J., Kossmann, D., Kraska, T.: Crowdsourcing applications and platforms: A data management perspective. Proceedings of the VLDB Endowment 4(12), 1508–1509 (2011)Google Scholar
  21. 21.
    Feng, A., Franklin, M., Kossmann, D., Kraska, T., Madden, S., Ramesh, S., Wang, A., Xin, R.: Crowddb: Query processing with the vldb crowd. Proceedings of the VLDB Endowment 4(12) (2011)Google Scholar
  22. 22.
    Gokhale, C., Das, S., Doan, A., Naughton, J.F., Rampalli, R., Shavlik, J., Zhu, X.: Corleone: hands-off crowdsourcing for entity matching. In: Book Corleone: Hands-Off Crowdsourcing for Entity MatchingGoogle Scholar
  23. 23.
    Jiang, L., Wang, Y., Hoffart, J., Weikum, G.: Crowdsourced entity markup. In: Book Crowdsourced Entity Markup, pp. 1–10 (2013)Google Scholar
  24. 24.
    Demartini, G., Difallah, D.E., Cudré-Mauroux, P.: ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In: Book ZenCrowd: Leveraging Probabilistic Reasoning and Crowdsourcing Techniques for Large-Scale Entity Linking, pp. 469–478. ACM (2012)Google Scholar
  25. 25.
    Yang, Y., Singh, P., Yao, J., Au Yeung, C.-m., Zareian, A., Wang, X., Cai, Z., Salvadores, M., Gibbins, N., Hall, W., Shadbolt, N.: Distributed human computation framework for linked data co-reference resolution. In: Antoniou, G., Grobelnik, M., Simperl, E., Parsia, B., Plexousakis, D., De Leenheer, P., Pan, J. (eds.) ESWC 2011, Part I. LNCS, vol. 6643, pp. 32–46. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  26. 26.
    Mozafari, B., Sarkar, P., Franklin, M.J., Jordan, M.I., Madden, S.: Active learning for crowd-sourced databases, CoRR, abs/1209.3686 (2012)Google Scholar
  27. 27.
    Venetis, P., Garcia-Molina, H.: Quality control for comparison microtasks. In: Book Quality Control for Comparison Microtasks, pp. 15–21. ACM (2012)Google Scholar
  28. 28.
    Mason, W., Watts, D.J.: Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter 11(2), 100–108 (2010)CrossRefGoogle Scholar
  29. 29.
    Feldman, M., Bernstein, A.: Cognition-based Task Routing: Towards Highly-Effective Task-Assignments in Crowdsourcing Settings (2014)Google Scholar
  30. 30.
    Joglekar, M., Garcia-Molina, H., Parameswaran, A.: Evaluating the crowd with confidence. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2013)Google Scholar
  31. 31.
    Khattak, F.K., Salleb-Aouissi, A.: Improving crowd labeling through expert evaluation. In: Book Improving Crowd Labeling Through Expert Evaluation (2012)Google Scholar
  32. 32.
    Su, H., Zheng, K., Huang, J., Liu, T., Wang, H., Zhou, X.: A crowd-based route recommendation system-CrowdPlanner. In: Book A Crowd-Based Route Recommendation System-CrowdPlanner, pp. 1178–1181. IEEE (2014)Google Scholar
  33. 33.
    Lease, M.: On Quality Control and Machine Learning in Crowdsourcing. In: Book On Quality Control and Machine Learning in Crowdsourcing (2011)Google Scholar
  34. 34.
    Driver, M.J.: Decision style: Past, present, and future research, International perspectives on individual differences, pp. 41–64 (2000)Google Scholar
  35. 35.
    Saberi, M., Hussain, O.K., Janjua, N.K., Chang, E.: In-house crowdsourcing-based entity resolution: dealing with common names. In: Book In-House Crowdsourcing-Based Entity Resolution: Dealing with Common Names, pp. 83–88. IEEE (2014)Google Scholar
  36. 36.
    Saaty, T.L.: The analytic hierarchy process: planning, priority setting, resources allocation. McGraw, New York (1980)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Morteza Saberi
    • 1
    Email author
  • Omar Khadeer Hussain
    • 1
  • Naeem Khalid Janjua
    • 1
  • Elizabeth Chang
    • 1
  1. 1.School of BusinessUNSW CanberraCanberraAustralia

Personalised recommendations