Advertisement

The VLDB Journal

, Volume 28, Issue 1, pp 1–23 | Cite as

Optimized group formation for solving collaborative tasks

  • Habibur RahmanEmail author
  • Senjuti Basu Roy
  • Saravanan Thirumuruganathan
  • Sihem Amer-Yahia
  • Gautam Das
Regular Paper
  • 171 Downloads

Abstract

Many popular applications, such as collaborative document editing, sentence translation, or citizen science, resort to collaborative crowdsourcing, a special form of human-based computing, where, crowd workers with appropriate skills and expertise are required to form groups to solve complex tasks. While there has been extensive research on workers’ task assignment for traditional microtask-based crowdsourcing, they often ignore the critical aspect of collaboration. Central to any collaborative crowdsourcing process is the aspect of solving collaborative tasks that requires successful collaboration among the workers. Our formalism considers two main collaboration-related factors—affinity and upper critical mass—appropriately adapted from organizational science and social theories. Our contributions are threefold. First, we formalize the notion of collaboration among crowd workers and propose a comprehensive optimization model for task assignment in a collaborative crowdsourcing environment. Next, we study the hardness of the task assignment optimization problem and propose a series of efficient exact and approximation algorithms with provable theoretical guarantees. Finally, we present a detailed set of experimental results stemming from two real-world collaborative crowdsourcing application using Amazon Mechanical Turk.

Keywords

Crowdsourcing Collaboration Group formation Algorithms 

References

  1. 1.
    Ahmad, S., Battle, A., Malkani, Z., Kamvar, S.: The Jabberwocky programming environment for structured social computing. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 53–64. ACM (2011)Google Scholar
  2. 2.
    Amer-Yahia, S., Basu Roy, S.: From complex object exploration to complex crowdsourcing. In: Proceedings of the 24th International Conference on World Wide Web, pp. 1531–1532. ACM (2015)Google Scholar
  3. 3.
    Anagnostopoulos, A., Becchetti, L., Castillo, C., Gionis, A., Leonardi, S.: Power in unity: forming teams in large-scale community systems. In: CIKM ’10 (2010)Google Scholar
  4. 4.
    Anagnostopoulos, A., Becchetti, L., Castillo, C., Gionis, A., Leonardi, S.: Online team formation in social networks. In: WWW ’12 (2012)Google Scholar
  5. 5.
    Andres, H.P.: Team cognition using collaborative technology: a behavioral analysis. J. Manage. Psychol. 28, 38–54 (2013)CrossRefGoogle Scholar
  6. 6.
    Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., Panovich, K.: Soylent: a word processor with a crowd inside. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, pp. 313–322. ACM (2010)Google Scholar
  7. 7.
    Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1999)zbMATHGoogle Scholar
  8. 8.
    Bisschop, J.: AIMMS - optimization modeling. Lulu.com (2006). ISBN:1411698991 Google Scholar
  9. 9.
    Boudreau, K., Gaule, P., Lakhani, K.R., Riedl, C., Woolley, A.W.: From crowds to collaborators: initiating effort and catalyzing interactions among online creative workers (2014)Google Scholar
  10. 10.
    Chai, K., Potdar, V., Dillon, T.: Content quality assessment related frameworks for social media. In: ICCSA(2009)Google Scholar
  11. 11.
    Chen, D.L., Dolan, W.B.: Building a persistent workforce on mechanical turk for multilingual data collection. In: HCOMP (2011)Google Scholar
  12. 12.
    Chhabra, M., Das, S., Szymanski, B.: Team formation in social networks. In: Computer and Information Sciences III, pp. 291–299. Springer (2013)Google Scholar
  13. 13.
    Daniela Retelny, A.T. M.B., Robaszkiewicz, S.: Expert crowdsourcing with flash teams. In: CrowdConf 2013 PosterGoogle Scholar
  14. 14.
    Difallah, D.E., Catasta, M., Demartini, G., Ipeirotis, P.G., Cudré-Mauroux, P.: The dynamics of micro-task crowdsourcing: the case of Amazon MTurk. In: Proceedings of the 24th International Conference on World Wide Web, pp. 238–247. ACM (2015)Google Scholar
  15. 15.
    Downs, J.S., Holbrook, M.B., Sheng, S., Cranor, L.F.: Are your participants gaming the system? Screening mechanical turk workers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’10, pp. 2399–2402. ACM, New York, NY (2010).  https://doi.org/10.1145/1753326.1753688
  16. 16.
    Duh, K., Hirao, T., Kimura, A., Ishiguro, K., Iwata, T., Yeung, C.-M.A.: Creating stories: Social curation of twitter messages. In: ICWSM (2012)Google Scholar
  17. 17.
    Gajewar, A., Sarma, A.D.: Multi-skill collaborative teams based on densest subgraphs. In: Proceedings of the 2012 SIAM International Conference on Data Mining, pp. 165–176 (2012)Google Scholar
  18. 18.
    Gao, D., Tong, Y., She, J., Song, T., Chen, L., Xu, K.: Top-k team recommendation in spatial crowdsourcing. In: International Conference on Web-Age Information Management, pp. 191–204. Springer (2016)Google Scholar
  19. 19.
    Gao, D., Tong, Y., She, J., Song, T., Chen, L., Xu, K.: Top-k team recommendation and its variants in spatial crowdsourcing. Data Sci. Eng. 2(2), 136–150 (2017)CrossRefGoogle Scholar
  20. 20.
    Garey, M.R., Johnson, D.S.: Computers and Intractability: A Guide to the Theory of NP-Completeness. W.H. Freeman and Company, New York (1979)zbMATHGoogle Scholar
  21. 21.
    Grotschel, M., Lovász, L.: Combinatorial optimization. Handb. Comb. 2, 1541–1597 (1995)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Guo, S., Parameswaran, A.G., Garcia-Molina, H.: So who won? Dynamic max discovery with the crowd. In: SIGMOD Conference, pp. 385–396 (2012)Google Scholar
  23. 23.
    Guttmann-Beck, N., Hassin, R.: Approximation algorithms for minimum k-cut. Algorithmica 27, 198–207 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Hertel, G.: Synergetic effects in working teams. J. Manage. Psychol. 26, 176–184 (2011)CrossRefGoogle Scholar
  25. 25.
    Hffmeier, J., Hertel, G.: When the whole is more than the sum of its parts: group motivation gains in the wild. J. Exp. Soc. Psychol. 47(2), 455–459 (2011)CrossRefGoogle Scholar
  26. 26.
    Ikeda, K., Morishima, A., Rahman, H., Roy, S.B., Thirumuruganathan, S., Amer-Yahia, S., Das, G.: Collaborative crowdsourcing with Crowd4U. Proc. VLDB Endow. 9(13), 1497–1500 (2016)CrossRefGoogle Scholar
  27. 27.
    Jøsang, A., Ismail, R., Boyd, C.: A survey of trust and reputation systems for online service provision. Decis. Support Syst. 43(2), 618–644 (2007)CrossRefGoogle Scholar
  28. 28.
    Kaplan, H., Lotosh, I., Milo, T., Novgorodov, S.: Answering planning queries with the crowd. PVLDB 6(9), 697–708 (2013)Google Scholar
  29. 29.
    Krumke, S., et. al.: Compact location problems. Comput. Sci. (1996)Google Scholar
  30. 30.
    Kargar, M., An, A.: Discovering top-k teams of experts with/without a leader in social networks. In: CIKM ’11 (2011)Google Scholar
  31. 31.
    Kargar, M., An, A., Zihayat, M.: Efficient bi-objective team formation in social networks. In: Flach, P., Bie, T., Cristianini, N. (eds.) Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, vol. 7524, pp. 483–498. Springer, Berlin (2012)CrossRefGoogle Scholar
  32. 32.
    Kargar, M., Zihayat, M., An, A.: Finding affordable and collaborative teams from a network of experts In: Proceedings of the 2013 SIAM International Conference on Data Mining, pp. 587–595 (2013)Google Scholar
  33. 33.
    Karpinski, M.: Approximability of the minimum bisection problem: an algorithmic challenge. In: Mathematical Foundations of Computer Science (2002)Google Scholar
  34. 34.
    Katz, D., Kahn, R.L.: The social psychology of organizations, vol. 2. Wiley, New York (1978)Google Scholar
  35. 35.
    Kenna, R., Berche, B.: Managing research quality: critical mass and optimal academic research group size. IMA J. Manage. Math. 23, 195–207 (2012)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Kim, J., Agrawala, M., Bernstein, M.S.: Mosaic: designing online creative communities for sharing works-in-progress. In: Design Thinking Research, pp. 105–129. Springer (2018)Google Scholar
  37. 37.
    Kim, J., Sterman, S., Cohen, A.A.B., Bernstein, M.S.: Mechanical novel: crowdsourcing complex work through reflection and revision. In: Design Thinking Research, pp. 79–104. Springer (2018)Google Scholar
  38. 38.
    Kittur, A., Kraut, R.E.: Harnessing the wisdom of crowds in Wikipedia: quality through coordination. In: Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work, CSCW ’08, pp. 37–46. ACM, New York, NY, USA (2008)Google Scholar
  39. 39.
    Kittur, A., Nickerson, J.V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., Horton, J.: The future of crowd work. In: CSCW ’13 (2013)Google Scholar
  40. 40.
    Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: Crowdforge: Crowdsourcing complex work. In: UIST (2011)Google Scholar
  41. 41.
    Kittur, A., Suh, B., Pendleton, B.A., Chi, E.H.: He says, she says: conflict and coordination in Wikipedia. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 453–462. ACM (2007)Google Scholar
  42. 42.
    Klakow, D., Peters, J.: Testing the correlation of word error rate and perplexity. Speech Commun. 38(1), 19–28 (2002)CrossRefzbMATHGoogle Scholar
  43. 43.
    Kulkarni, A., Can, M., Hartmann, B.: Collaboratively crowdsourcing workflows with turkomatic. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, pp. 1003–1012. ACM (2012)Google Scholar
  44. 44.
    Lappas, T., Liu, K., Terzi. E.: Finding a team of experts in social networks. In: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 467–476 (2009)Google Scholar
  45. 45.
    Lappas, T., Liu, K., Terzi, E.: Finding a team of experts in social networks. In: SIGKDD, pp. 467–476 (2009)Google Scholar
  46. 46.
    Lasecki, W.S., Murray, K.I., White, S., Miller, R.C., Bigham, J.P.: Real-time crowd control of existing interfaces. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11, pp. 23–32. ACM, New York, NY, USA (2011)Google Scholar
  47. 47.
    Lawler, E.L., Wood, D.E.: Branch-and-bound methods: a survey. Oper. Res. 14(4), 699–719 (1966)MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Li, G., Wang, J., Zheng, Y., Franklin, M.J.: Crowdsourced data management: a survey. IEEE Trans. Knowl. Data Eng. 28(9), 2296–2319 (2016)CrossRefGoogle Scholar
  49. 49.
    Li, G., Zheng, Y., Fan, J., Wang, J., Cheng, R.: Crowdsourced data management: overview and challenges. In: Proceedings of the 2017 ACM International Conference on Management of Data, pp. 1711–1716. ACM (2017)Google Scholar
  50. 50.
    Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, pp. 57–66. ACM (2010)Google Scholar
  51. 51.
    Majumder, A., Datta, S., Naidu, K.: Capacitated team formation problem on social networks. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’12, pp. 1005–1013. ACM, New York, NY, USA (2012)Google Scholar
  52. 52.
    Marcus, A., Parameswaran, A., et al.: Crowdsourced data management: industry and academic perspectives. Found. Trends Databases 6(1–2), 1–161 (2015)CrossRefGoogle Scholar
  53. 53.
    Marwell, G., Oliver, P.E., Prahl, R.: Social networks and collective action: a theory of the critical mass. Am. J. Sociol. 94, 502–534 (1988)CrossRefGoogle Scholar
  54. 54.
    Myers, I.B., McCaulley, M.H.: Myers–Briggs Type Indicator: MBTI. Consulting Psychologists Press, Palo Alto (1988)Google Scholar
  55. 55.
    Ojha, P., Talukdar, P.: Quality estimation of workers in collaborative crowdsourcing using group testing. In: Fourth AAAI Conference on Human Computation and Crowdsourcing (2016)Google Scholar
  56. 56.
    Ravi, S.S., et. al.: Facility dispersion problems: heuristics and special cases. In: WADS (1991)Google Scholar
  57. 57.
    Rahman, H., Roy, S.B., Thirumuruganathan, S., Amer-Yahia, S., Das, G.: Task assignment optimization in collaborative crowdsourcing. In: 2015 IEEE International Conference on Data Mining (ICDM), pp. 949–954. IEEE (2015)Google Scholar
  58. 58.
    Rahman, H., Thirumuruganathan, S., Roy, S.B., Amer-Yahia, S., Das, G.: Worker skill estimation in team-based tasks. Proc. VLDB Endow. 8(11), 1142–1153 (2015)CrossRefGoogle Scholar
  59. 59.
    Rosenkrantz, D.J., Tayi, G.K., Ravi, S.S.: Facility dispersion problems under capacity and cost constraints. J. Comb. Optim. 4, 7–33 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  60. 60.
    Roy, S.B., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., Das, G.: Crowds, not drones: modeling human factors in interactive crowdsourcing. In: DBCrowd (2013)Google Scholar
  61. 61.
    Roy, S.B., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., Das, G.: Optimization in knowledge-intensive crowdsourcing. CoRR, arXiv:1401.1302 (2014)
  62. 62.
    Salehi, N., McCabe, A., Valentine, M., Bernstein, M.: Huddler: convening stable and familiar crowd teams despite unpredictable availability. ArXiv preprint arXiv:1610.08216 (2016)
  63. 63.
    Schrijver, A.: Theory of Linear and Integer Programming. Wiley, New York (1986)zbMATHGoogle Scholar
  64. 64.
    Snehalkumar, G., Chhibber, N., Sehgal, V., Ballav, A., Mullings, C., Nasser, A., Richmond-Fuller, A., Gilbee, A., Gamage, D., Whiting, M., et al.: Prototype tasks: improving crowdsourcing results through rapid, iterative task design. ArXiv preprint arXiv:1707.05645 (2017)
  65. 65.
    Solomon, J., Wash, R.: Critical mass of what? Exploring community growth in wikiprojects. In: ICWSM (2014)Google Scholar
  66. 66.
    Surowiecki, J.: The wisdom of crowds: why the many are smarter than the few and how collective wisdom shapes business. Economies, Societies and Nations (2004)Google Scholar
  67. 67.
    Tong, Y., She, J., Ding, B., Wang, L., Chen, L.: Online mobile micro-task allocation in spatial crowdsourcing. In: 2016 IEEE 32nd International Conference on Data Engineering (ICDE), pp. 49–60. IEEE (2016)Google Scholar
  68. 68.
    Valentine, M.A., Retelny, D., To, A., Rahmati, N., Doshi, T., Bernstein, M.S.: Flash organizations: crowdsourcing complex work by structuring crowds as organizations. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 3523–3537. ACM (2017)Google Scholar
  69. 69.
    van Dongen, S., Enright, A.J.: Metric distances derived from cosine similarity and Pearson and Spearman correlations. CoRR arXiv:1208.3145 (2012)
  70. 70.
    Yan, R., Gao, M., Pavlick, E., Callison-Burch, C.: Are two heads better than one? Crowdsourced translation via a two-step collaboration of non-professional translators and editors In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol 1, pp. 1134–1144 (2014)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.UT ArlingtonArlingtonUSA
  2. 2.New Jersey Institute of TechnologyNewarkUSA
  3. 3.QCRIHBKUDohaQatar
  4. 4.CNRSLIGGrenobleFrance

Personalised recommendations