The Dimensions of Crowdsourcing Task Design

  • Ilio Catallo
  • Davide MartinenghiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10360)


Crowdsourcing, i.e., the provision of micro-tasks to be executed by a large pool of possibly anonymous workers, is attracting an increasing research attention, because it promises to help solving many scientific and practical problems where the harmonic cooperation of humans and machines delivers superior results. This paper proposes a systematic view of the crowdsourcing task design space and categorizes the dimensions that qualify the design decisions in crowdsourcing applications. For each dimension, we discuss the main open research problems and the most significant contributions, thereby offering guidelines for a principled understanding of current crowdsourcing marketplaces.


Intrinsic Motivation Extrinsic Motivation Collective Intelligence Task Significance Skill Variety 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors acknowledge support from the H2020-EU.3.3.1 “ENCOMPASS” project (ID: 723059).


  1. 1.
    Antin, J., Shaw, A.: Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and india. In: SIGCHI, pp. 2925–2934. ACM (2012)Google Scholar
  2. 2.
    Bernaschina, C., et al.: A web tool for the execution of crowdsourcing campaigns. In: WWW - Companion Volume, pp. 171–174 (2015)Google Scholar
  3. 3.
    Bernaschina, C., et al.: On the role of task design in crowdsourcing campaigns. In: HCOMP, pp. 4–5 (2015)Google Scholar
  4. 4.
    Catallo, I., et al.: A workload-dependent task assignment policy for crowdsourcing. WWW J. (2017). doi: 10.1007/s11280-016-0428-7
  5. 5.
    Chandler, D., Kapelner, A.: Breaking monotony with meaning: motivation in crowdsourcing markets. J. Econ. Behav. Organiz. 90, 123–133 (2013)CrossRefGoogle Scholar
  6. 6.
    Ciceri, E., et al.: Crowdsourcing for top-K query processing over uncertain data. IEEE TKDE 28(1), 41–53 (2016)MathSciNetGoogle Scholar
  7. 7.
    Deci, E.L., Ryan, R.M.: Self-Determination. Wiley Online Library (2010)Google Scholar
  8. 8.
    Doan, A., Ramakrishnan, R., Halevy, A.Y.: Crowdsourcing systems on the world-wide web. Commun. ACM 54(4), 86–96 (2011)CrossRefGoogle Scholar
  9. 9.
    Gadiraju, U., Kawase, R., Dietze, S.: A taxonomy of microtasks on the web. In HYPERTEXT, pp. 218–223. ACM (2014)Google Scholar
  10. 10.
    Geiger, D., et al.: Crowdsourcing information systems - definition, typology, and design. In: ICIS (2012)Google Scholar
  11. 11.
    Hackman, J.R., Oldham, G.R.: Motivation through the design of work: test of a theory. Organ. Behav. Hum. Perform. 16(2), 250–279 (1976)CrossRefGoogle Scholar
  12. 12.
    Heer, J., Bostock, M.: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: SIGCHI, pp. 203–212. ACM (2010)Google Scholar
  13. 13.
    Khatib, F., et al.: Algorithm discovery by protein folding game players. Proc. Nat. Acad. Sci. 108(47), 18949–18953 (2011)CrossRefGoogle Scholar
  14. 14.
    Kittur, A., et al.: The future of crowd work. In: CSWC, pp. 1301–1318 (2013)Google Scholar
  15. 15.
    Loni, B., et al.: Fashion-focused creative commons social dataset. In: MMSys, pp. 72–77 (2013)Google Scholar
  16. 16.
    Malone, T.W., Laubacher, R., Dellarocas, C.: Harnessing crowds: mapping the genome of collective intelligence. MIT Sloan School of Management (2009)Google Scholar
  17. 17.
    Prendergast, C.: The provision of incentives in firms. J. Econ. Lit. 37, 7–63 (1999)CrossRefGoogle Scholar
  18. 18.
    Quinn, A.J., Bederson, B.B.: Human computation: a survey and taxonomy of a growing field. In: SIGCHI, pp. 1403–1412. ACM (2011)Google Scholar
  19. 19.
    Ryan, R.M., Deci, E.L.: Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp. Educ. Psychol. 25(1), 54–67 (2000)CrossRefGoogle Scholar
  20. 20.
    Schulze, T., Krug, S., Schader, M.: Workers’ task choice in crowdsourcing and human computation markets. In: ICIS (2012)Google Scholar
  21. 21.
    Surowiecki, J.: The wisdom of crowds. Random House LLC (2005)Google Scholar
  22. 22.
    Von Ahn, L.: Human computation. In: DAC 2009 46th ACM/IEEE Design Automation Conference, pp. 418–419. IEEE (2009)Google Scholar
  23. 23.
    Whitla, P.: Crowdsourcing and its application in marketing activities. Contemp. Manag. Res. 5(1), 15–28 (2009)CrossRefGoogle Scholar
  24. 24.
    Yan, Y., Rosales, R., Fung, G., Dy, J.G.: Active learning from crowds. In: ICML, pp. 1161–1168 (2011)Google Scholar
  25. 25.
    Yu, B., Willis, M., Sun, P., Wang, J.: Crowdsourcing participatory evaluation of medical pictograms using amazon mechanical turk. J. Med. Int. Res. 15(6), e108 (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.DEIB - Politecnico di MilanoMilanoItaly

Personalised recommendations