Skip to main content

Advertisement

SpringerLink
Log in

Optimized group formation for solving collaborative tasks

  • Regular Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

Many popular applications, such as collaborative document editing, sentence translation, or citizen science, resort to collaborative crowdsourcing, a special form of human-based computing, where, crowd workers with appropriate skills and expertise are required to form groups to solve complex tasks. While there has been extensive research on workers’ task assignment for traditional microtask-based crowdsourcing, they often ignore the critical aspect of collaboration. Central to any collaborative crowdsourcing process is the aspect of solving collaborative tasks that requires successful collaboration among the workers. Our formalism considers two main collaboration-related factors—affinity and upper critical mass—appropriately adapted from organizational science and social theories. Our contributions are threefold. First, we formalize the notion of collaboration among crowd workers and propose a comprehensive optimization model for task assignment in a collaborative crowdsourcing environment. Next, we study the hardness of the task assignment optimization problem and propose a series of efficient exact and approximation algorithms with provable theoretical guarantees. Finally, we present a detailed set of experimental results stemming from two real-world collaborative crowdsourcing application using Amazon Mechanical Turk.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Notes

  1. This work is the extension of our paper [57]. We extend our previous work by providing (i) an additional technique for task assignment referred to as Cons-cost-K-ApprxGrp, (ii) detail proofs of our algorithms and (iii) additional experiments on both real and synthetic data

  2. Notice that posing affinity as a constraint does not fully exploit the effect of “group synergy.”

  3. Star graph is a tree on v nodes with one node having degree \(v-1\) and other \(v-1\) nodes with degree 1.

  4. Without triangle inequality assumption, no theoretical guarantee could be ensured [59].

References

  1. Ahmad, S., Battle, A., Malkani, Z., Kamvar, S.: The Jabberwocky programming environment for structured social computing. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 53–64. ACM (2011)

  2. Amer-Yahia, S., Basu Roy, S.: From complex object exploration to complex crowdsourcing. In: Proceedings of the 24th International Conference on World Wide Web, pp. 1531–1532. ACM (2015)

  3. Anagnostopoulos, A., Becchetti, L., Castillo, C., Gionis, A., Leonardi, S.: Power in unity: forming teams in large-scale community systems. In: CIKM ’10 (2010)

  4. Anagnostopoulos, A., Becchetti, L., Castillo, C., Gionis, A., Leonardi, S.: Online team formation in social networks. In: WWW ’12 (2012)

  5. Andres, H.P.: Team cognition using collaborative technology: a behavioral analysis. J. Manage. Psychol. 28, 38–54 (2013)

    Article  Google Scholar 

  6. Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., Panovich, K.: Soylent: a word processor with a crowd inside. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, pp. 313–322. ACM (2010)

  7. Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1999)

    MATH  Google Scholar 

  8. Bisschop, J.: AIMMS - optimization modeling. Lulu.com (2006). ISBN:1411698991

  9. Boudreau, K., Gaule, P., Lakhani, K.R., Riedl, C., Woolley, A.W.: From crowds to collaborators: initiating effort and catalyzing interactions among online creative workers (2014)

  10. Chai, K., Potdar, V., Dillon, T.: Content quality assessment related frameworks for social media. In: ICCSA(2009)

  11. Chen, D.L., Dolan, W.B.: Building a persistent workforce on mechanical turk for multilingual data collection. In: HCOMP (2011)

  12. Chhabra, M., Das, S., Szymanski, B.: Team formation in social networks. In: Computer and Information Sciences III, pp. 291–299. Springer (2013)

  13. Daniela Retelny, A.T. M.B., Robaszkiewicz, S.: Expert crowdsourcing with flash teams. In: CrowdConf 2013 Poster

  14. Difallah, D.E., Catasta, M., Demartini, G., Ipeirotis, P.G., Cudré-Mauroux, P.: The dynamics of micro-task crowdsourcing: the case of Amazon MTurk. In: Proceedings of the 24th International Conference on World Wide Web, pp. 238–247. ACM (2015)

  15. Downs, J.S., Holbrook, M.B., Sheng, S., Cranor, L.F.: Are your participants gaming the system? Screening mechanical turk workers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’10, pp. 2399–2402. ACM, New York, NY (2010). https://doi.org/10.1145/1753326.1753688

  16. Duh, K., Hirao, T., Kimura, A., Ishiguro, K., Iwata, T., Yeung, C.-M.A.: Creating stories: Social curation of twitter messages. In: ICWSM (2012)

  17. Gajewar, A., Sarma, A.D.: Multi-skill collaborative teams based on densest subgraphs. In: Proceedings of the 2012 SIAM International Conference on Data Mining, pp. 165–176 (2012)

  18. Gao, D., Tong, Y., She, J., Song, T., Chen, L., Xu, K.: Top-k team recommendation in spatial crowdsourcing. In: International Conference on Web-Age Information Management, pp. 191–204. Springer (2016)

  19. Gao, D., Tong, Y., She, J., Song, T., Chen, L., Xu, K.: Top-k team recommendation and its variants in spatial crowdsourcing. Data Sci. Eng. 2(2), 136–150 (2017)

    Article  Google Scholar 

  20. Garey, M.R., Johnson, D.S.: Computers and Intractability: A Guide to the Theory of NP-Completeness. W.H. Freeman and Company, New York (1979)

    MATH  Google Scholar 

  21. Grotschel, M., Lovász, L.: Combinatorial optimization. Handb. Comb. 2, 1541–1597 (1995)

    MathSciNet  MATH  Google Scholar 

  22. Guo, S., Parameswaran, A.G., Garcia-Molina, H.: So who won? Dynamic max discovery with the crowd. In: SIGMOD Conference, pp. 385–396 (2012)

  23. Guttmann-Beck, N., Hassin, R.: Approximation algorithms for minimum k-cut. Algorithmica 27, 198–207 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  24. Hertel, G.: Synergetic effects in working teams. J. Manage. Psychol. 26, 176–184 (2011)

    Article  Google Scholar 

  25. Hffmeier, J., Hertel, G.: When the whole is more than the sum of its parts: group motivation gains in the wild. J. Exp. Soc. Psychol. 47(2), 455–459 (2011)

    Article  Google Scholar 

  26. Ikeda, K., Morishima, A., Rahman, H., Roy, S.B., Thirumuruganathan, S., Amer-Yahia, S., Das, G.: Collaborative crowdsourcing with Crowd4U. Proc. VLDB Endow. 9(13), 1497–1500 (2016)

    Article  Google Scholar 

  27. Jøsang, A., Ismail, R., Boyd, C.: A survey of trust and reputation systems for online service provision. Decis. Support Syst. 43(2), 618–644 (2007)

    Article  Google Scholar 

  28. Kaplan, H., Lotosh, I., Milo, T., Novgorodov, S.: Answering planning queries with the crowd. PVLDB 6(9), 697–708 (2013)

    Google Scholar 

  29. Krumke, S., et. al.: Compact location problems. Comput. Sci. (1996)

  30. Kargar, M., An, A.: Discovering top-k teams of experts with/without a leader in social networks. In: CIKM ’11 (2011)

  31. Kargar, M., An, A., Zihayat, M.: Efficient bi-objective team formation in social networks. In: Flach, P., Bie, T., Cristianini, N. (eds.) Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, vol. 7524, pp. 483–498. Springer, Berlin (2012)

    Chapter  Google Scholar 

  32. Kargar, M., Zihayat, M., An, A.: Finding affordable and collaborative teams from a network of experts In: Proceedings of the 2013 SIAM International Conference on Data Mining, pp. 587–595 (2013)

  33. Karpinski, M.: Approximability of the minimum bisection problem: an algorithmic challenge. In: Mathematical Foundations of Computer Science (2002)

  34. Katz, D., Kahn, R.L.: The social psychology of organizations, vol. 2. Wiley, New York (1978)

  35. Kenna, R., Berche, B.: Managing research quality: critical mass and optimal academic research group size. IMA J. Manage. Math. 23, 195–207 (2012)

    Article  MathSciNet  Google Scholar 

  36. Kim, J., Agrawala, M., Bernstein, M.S.: Mosaic: designing online creative communities for sharing works-in-progress. In: Design Thinking Research, pp. 105–129. Springer (2018)

  37. Kim, J., Sterman, S., Cohen, A.A.B., Bernstein, M.S.: Mechanical novel: crowdsourcing complex work through reflection and revision. In: Design Thinking Research, pp. 79–104. Springer (2018)

  38. Kittur, A., Kraut, R.E.: Harnessing the wisdom of crowds in Wikipedia: quality through coordination. In: Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work, CSCW ’08, pp. 37–46. ACM, New York, NY, USA (2008)

  39. Kittur, A., Nickerson, J.V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., Horton, J.: The future of crowd work. In: CSCW ’13 (2013)

  40. Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: Crowdforge: Crowdsourcing complex work. In: UIST (2011)

  41. Kittur, A., Suh, B., Pendleton, B.A., Chi, E.H.: He says, she says: conflict and coordination in Wikipedia. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 453–462. ACM (2007)

  42. Klakow, D., Peters, J.: Testing the correlation of word error rate and perplexity. Speech Commun. 38(1), 19–28 (2002)

    Article  MATH  Google Scholar 

  43. Kulkarni, A., Can, M., Hartmann, B.: Collaboratively crowdsourcing workflows with turkomatic. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, pp. 1003–1012. ACM (2012)

  44. Lappas, T., Liu, K., Terzi. E.: Finding a team of experts in social networks. In: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 467–476 (2009)

  45. Lappas, T., Liu, K., Terzi, E.: Finding a team of experts in social networks. In: SIGKDD, pp. 467–476 (2009)

  46. Lasecki, W.S., Murray, K.I., White, S., Miller, R.C., Bigham, J.P.: Real-time crowd control of existing interfaces. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11, pp. 23–32. ACM, New York, NY, USA (2011)

  47. Lawler, E.L., Wood, D.E.: Branch-and-bound methods: a survey. Oper. Res. 14(4), 699–719 (1966)

    Article  MathSciNet  MATH  Google Scholar 

  48. Li, G., Wang, J., Zheng, Y., Franklin, M.J.: Crowdsourced data management: a survey. IEEE Trans. Knowl. Data Eng. 28(9), 2296–2319 (2016)

    Article  Google Scholar 

  49. Li, G., Zheng, Y., Fan, J., Wang, J., Cheng, R.: Crowdsourced data management: overview and challenges. In: Proceedings of the 2017 ACM International Conference on Management of Data, pp. 1711–1716. ACM (2017)

  50. Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, pp. 57–66. ACM (2010)

  51. Majumder, A., Datta, S., Naidu, K.: Capacitated team formation problem on social networks. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’12, pp. 1005–1013. ACM, New York, NY, USA (2012)

  52. Marcus, A., Parameswaran, A., et al.: Crowdsourced data management: industry and academic perspectives. Found. Trends Databases 6(1–2), 1–161 (2015)

    Article  Google Scholar 

  53. Marwell, G., Oliver, P.E., Prahl, R.: Social networks and collective action: a theory of the critical mass. Am. J. Sociol. 94, 502–534 (1988)

    Article  Google Scholar 

  54. Myers, I.B., McCaulley, M.H.: Myers–Briggs Type Indicator: MBTI. Consulting Psychologists Press, Palo Alto (1988)

    Google Scholar 

  55. Ojha, P., Talukdar, P.: Quality estimation of workers in collaborative crowdsourcing using group testing. In: Fourth AAAI Conference on Human Computation and Crowdsourcing (2016)

  56. Ravi, S.S., et. al.: Facility dispersion problems: heuristics and special cases. In: WADS (1991)

  57. Rahman, H., Roy, S.B., Thirumuruganathan, S., Amer-Yahia, S., Das, G.: Task assignment optimization in collaborative crowdsourcing. In: 2015 IEEE International Conference on Data Mining (ICDM), pp. 949–954. IEEE (2015)

  58. Rahman, H., Thirumuruganathan, S., Roy, S.B., Amer-Yahia, S., Das, G.: Worker skill estimation in team-based tasks. Proc. VLDB Endow. 8(11), 1142–1153 (2015)

    Article  Google Scholar 

  59. Rosenkrantz, D.J., Tayi, G.K., Ravi, S.S.: Facility dispersion problems under capacity and cost constraints. J. Comb. Optim. 4, 7–33 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  60. Roy, S.B., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., Das, G.: Crowds, not drones: modeling human factors in interactive crowdsourcing. In: DBCrowd (2013)

  61. Roy, S.B., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., Das, G.: Optimization in knowledge-intensive crowdsourcing. CoRR, arXiv:1401.1302 (2014)

  62. Salehi, N., McCabe, A., Valentine, M., Bernstein, M.: Huddler: convening stable and familiar crowd teams despite unpredictable availability. ArXiv preprint arXiv:1610.08216 (2016)

  63. Schrijver, A.: Theory of Linear and Integer Programming. Wiley, New York (1986)

    MATH  Google Scholar 

  64. Snehalkumar, G., Chhibber, N., Sehgal, V., Ballav, A., Mullings, C., Nasser, A., Richmond-Fuller, A., Gilbee, A., Gamage, D., Whiting, M., et al.: Prototype tasks: improving crowdsourcing results through rapid, iterative task design. ArXiv preprint arXiv:1707.05645 (2017)

  65. Solomon, J., Wash, R.: Critical mass of what? Exploring community growth in wikiprojects. In: ICWSM (2014)

  66. Surowiecki, J.: The wisdom of crowds: why the many are smarter than the few and how collective wisdom shapes business. Economies, Societies and Nations (2004)

  67. Tong, Y., She, J., Ding, B., Wang, L., Chen, L.: Online mobile micro-task allocation in spatial crowdsourcing. In: 2016 IEEE 32nd International Conference on Data Engineering (ICDE), pp. 49–60. IEEE (2016)

  68. Valentine, M.A., Retelny, D., To, A., Rahmati, N., Doshi, T., Bernstein, M.S.: Flash organizations: crowdsourcing complex work by structuring crowds as organizations. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 3523–3537. ACM (2017)

  69. van Dongen, S., Enright, A.J.: Metric distances derived from cosine similarity and Pearson and Spearman correlations. CoRR arXiv:1208.3145 (2012)

  70. Yan, R., Gao, M., Pavlick, E., Callison-Burch, C.: Are two heads better than one? Crowdsourced translation via a two-step collaboration of non-professional translators and editors In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol 1, pp. 1134–1144 (2014)

Download references

Author information

Authors and Affiliations

  1. UT Arlington, Arlington, USA

    Habibur Rahman & Gautam Das

  2. New Jersey Institute of Technology, Newark, USA

    Senjuti Basu Roy

  3. QCRI, HBKU, Doha, Qatar

    Saravanan Thirumuruganathan

  4. CNRS, LIG, Grenoble, France

    Sihem Amer-Yahia

Authors
  1. Habibur Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Senjuti Basu Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saravanan Thirumuruganathan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sihem Amer-Yahia
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gautam Das
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Habibur Rahman.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rahman, H., Roy, S.B., Thirumuruganathan, S. et al. Optimized group formation for solving collaborative tasks. The VLDB Journal 28, 1–23 (2019). https://doi.org/10.1007/s00778-018-0516-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-018-0516-7

Keywords

Search

Navigation