Abstract
The conceptual distinction between microtasks and macrotasks has been made relatively early on in the crowdsourcing literature. However, only recently a handful of research works has explored it explicitly. These works, for the most part, have focused on simply discussing macrotasks within the confines of their own work (e.g., in terms of creativity), without taking into account the multiple facets that working with such tasks involves. This has resulted in the term “macrotask” to be severely convoluted and largely meaning different things to different individuals. More importantly, it has resulted in disregarding macrotask crowdsourcing as a new labor model of its own right. To address this scholarly gap, in this paper we discuss macrotask crowdsourcing from a multitude of dimensions, namely the nature of the problem it can solve, the crowdworker skills it involves, and the work management structures it necessitates. In view of our analysis, we provide a first integrated definition of macrotask crowdsourcing.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Altshuller, G. (2005). The innovation algorithm: TRIZ, systematic innovation and technical creativity. Worchesrter, MA: Technical Innovation Center.
Ambati, V., Vogel, S., & Carbonell, J. (2012). Collaborative workflow for crowdsourcing translation. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW ’12) (pp. 1191–1194). ACM, New York, NY, USA.
Basu Roy, S., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., & Das, G. (2015). Task assignment optimization in knowledge-intensive crowdsourcing. The VLDB Journal—The International Journal on Very Large Data Bases, 24(4), 467–491.
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D. & Panovich, K. (2010). Soylent: A word processor with a crowd inside. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology (UIST ’10) (pp. 313–322). ACM, New York, NY, USA.
Chan, J., Dang, S., & Dow, S. P. (2016). Improving crowd innovation with expert facilitation. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW ’16) (pp. 1223–1235). ACM, New York, NY, USA.
Cheng, J., Teevan, J., Iqbal, S. T., & Bernstein, M. S. (2015, April). Break it down: A comparison of macro-and microtasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 4061–4064). ACM.
Chilton, L. B., Little, G., Edge, D., Weld, D. S., & Landay, J. A. (2013). Cascade: Crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13) (pp. 1999–2008). ACM, New York, NY, USA.
Creative and Cultural Skills. (2017). Building a creative nation: Current and future skills need. https://ccskills.org.uk/downloads/Building_a_Creative_Nation_-_Current_and_Future_Skills_Needs.pdf.
Edmondson, A. C., Bohmer, R. M., & Pisano, G. P. (2001). Disrupted routines: Team learning and new technology implementation in hospitals. Administrative Science Quarterly, 46(4), 685–716.
Goel, G., Nikzad, A., & Singla, A. (2014). Allocating tasks to workers with matching constraints: Truthful mechanisms for crowdsourcing markets. In Proceedings of the Companion Publication of the 23rd International Conference on World Wide Web Companion (WWW Companion ’14) (pp. 279–280). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland.
Gray, M. L., Suri, S., Ali, S. S., & Kulkarni, D. (2016, February). The crowd is a collaborative network. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 134–147). ACM.
Grier, D. A. (2013). Crowdsourcing for dummies. Wiley.
Haas, D., Ansel, J., Gu, L., & Marcus, A. (2015). Argonaut: Macrotask crowdsourcing for complex data processing. Proceedings of the VLDB Endowment, 8(12), 1642–1653.
Ho, C. J., & Vaughan, J. W. (2012, July). Online task assignment in crowdsourcing markets. In Twenty-sixth AAAI Conference on Artificial Intelligence.
Huang, S., & Holden, D. (2016). The R&D boundaries of the firm: A problem solving perspective. International Journal of the Economics of Business, 23(3), 287–317.
Ipeirotis, P. G., & Gabrilovich, E. (2014, April). Quizz: Targeted crowdsourcing with a billion (potential) users. In Proceedings of the 23rd International Conference on World Wide Web (pp. 143–154). ACM.
Irani, L. C., & Silberman, M. (2013, April). Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 611–620). ACM.
Jansson, D. G., & Smith, S. M. (1991). Design fixation. Design Studies, 12(1), 3–11.
Kim, J., Cheng, J., Bernstein, & M.S. (2014) Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 745–755). CSCW ’14, ACM, New York, NY, USA. https://doi.org/10.1145/2531602.2531638, http://doi.acm.org/10.1145/2531602.2531638.
Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’08) (pp. 453–456). ACM, New York, NY, USA
Lawler, E. E., & Worley, C. G. (2006). Designing organizations that are built to change. MIT Sloan Management Review, 48(1), 19–23.
Li, G., Wang, J., Zheng, Y., & Franklin, M. J. (2016). Crowdsourced data management: A survey. IEEE Transactions on Knowledge and Data Engineering, 28(9), 2296–2319.
Little, G. Chilton, L. B., Goldman, M., & Miller, R. C. (2010). Exploring Iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP ’10). ACM, New York, NY, USA, 68–76.
Machado, L., Pereira, G., Prikladnicki, R., Carmel, E., & de Souza, C. R. (2014, November). Crowdsourcing in the Brazilian IT industry: What we know and what we don’t know. In Proceedings of the 1st International Workshop on Crowd-based Software Development Methods and Technologies (pp. 7–12). ACM.
Martin, D., Hanrahan, B. V., O’Neill, J., & Gupta, N. (2014, February). Being a turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 224–235). ACM.
Mason, W., & Watts, D. J. (2009, June). Financial incentives and the performance of crowds. In Proceedings of the ACM SIGKDD Workshop on Human Computation (pp. 77–85). ACM.
Mavridis, P., Gross-Amblard, D., & Miklós, Z. (2016, April). Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing. In Proceedings of the 25th International Conference on World Wide Web (pp. 843–853). International World Wide Web Conferences Steering Committee.
Majchrzak, A., & Malhotra, A. (2013). Towards an information systems perspective and research agenda on crowdsourcing for innovation. The Journal of Strategic Information Systems, 22(4), 257–268.
Morris, M. R., Bigham, J. P., Brewer, R., Bragg, J., Kulkarni, A., Li, J., & Savage, S. (2017, May). Subcontracting microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 1867–1876). ACM.
Musthag, M., & Ganesan, D. (2013). Labor dynamics in a mobile micro-task market. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13) (pp. 641–650). ACM, New York, NY, USA.
Nickerson, J. A., & Zenger, T. R. (2004). A knowledge-based theory of the firm–the problem-solving perspective. Organization Science, 15(6), 617–632.
Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., … & Bernstein, M. S. (2014, October). Expert crowdsourcing with flash teams. In Proceedings of the 27th annual ACM Symposium on User Interface Software and Technology (pp. 75–85). ACM.
Retelny, D., Bernstein, M. S., & Valentine, M. A. (2017). No workflow can ever be enough: How crowdsourcing workflows constrain complex work. In Proceedings of the ACM on Human-Computer Interaction, 1(CSCW), 89.
Salehi, N., & Bernstein, M. S. (2018). Hive: Collective design through network rotation. In Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 151.
Schmitz, H., & Lykourentzou, I. (2018). Online sequencing of non-decomposable macrotasks in expert crowdsourcing. ACM Transactions on Social Computing, 1(1), 1.
Sieg, J. H, Wallin, M. W., & Von Krogh, G. (2010). Managerial challenges in open innovation: a study of innovation intermediation in the chemical industry. R&D Management, 40(3), 281–291.
Teevan, J., Iqbal, S. T., Cai, C. J., Bigham, J. P., Bernstein, M. S., & Gerber, E. M. (2016). Productivity decomposed: Getting big things done with microtasks. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’16) (pp. 3500–3507). ACM, New York, NY, USA.
Teodoro, R., Ozturk, P., Naaman, M., Mason, W., & Lindqvist, J. (2014, February). The motivations and experiences of the on-demand mobile workforce. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 236–247). ACM.
Valentine, M. A., Retelny, D., To, A., Rahmati, N., Doshi, T., & Bernstein, M. S. (2017, May). Flash organizations: Crowdsourcing complex work by structuring crowds as organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 3523–3537). ACM.
Voigt, P., & Von dem Bussche, A. (2017). The EU General Data Protection Regulation (GDPR). In A practical guide, 1st edn. Cham: Springer.
Vondrick, C., Patterson, D., & Ramanan, D. (2013). Efficiently scaling up crowdsourced video annotation. International Journal of Computer Vision, 101(1), 184–204.
Wagner, T. (2014). The global achievement gap: Why even our best schools don’t teach the new survival skills our children need—and what we can do about it. Basic Books.
Wood, A. J., Graham, M., Lehdonvirta, V., & Hjorth, I. (2019). Networked but commodified: The (Dis) embeddedness of digital labour in the gig economy. Sociology, 0038038519828906.
Xie, H., & Lui, J. C. (2018). Incentive mechanism and rating system design for crowdsourcing systems: Analysis, tradeoffs and inference. IEEE Transactions on Services Computing, 11(1), 90–102.
Zheng, H., Li, D., & Hou, W. (2011). Task design, motivation, and participation in crowdsourcing contests. International Journal of Electronic Commerce, 15(4), 57–88.
Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., & Horvitz, E. (2012). Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12) (pp. 217–226). ACM, New York, NY, USA.
Zaidan, O. F., & Callison-Burch, C. (2011). Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (HLT ’11) (pp. 1220–1229). Association for Computational Linguistics, Stroudsburg, PA, USA.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Lykourentzou, I., Khan, VJ., Papangelis, K., Markopoulos, P. (2019). Macrotask Crowdsourcing: An Integrated Definition. In: Khan, VJ., Papangelis, K., Lykourentzou, I., Markopoulos, P. (eds) Macrotask Crowdsourcing. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-030-12334-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-12334-5_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-12333-8
Online ISBN: 978-3-030-12334-5
eBook Packages: Computer ScienceComputer Science (R0)