Abstract
We propose a framework of micro-tasking that intrinsically supports the development of workers’ skills. It aims to help developers of micro-tasking systems add skill development capabilities to their systems with minimal development costs. This will allow micro-tasking of skill-intensive work and improve the sustainability of micro-tasking systems. Based on the results of the micro-tasking projects we have carried out, our framework has three core modules: tutorial producer, task dispatcher, and feedback visualizer, which are supported by a back-end skill assessment engine. In closing, we discuss ways to apply the proposed framework to realistic micro-tasking situations.
Chapter PDF
References
Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: Crowdforge: Crowdsourcing complex work. In: Proc. UIST 2011, pp. 43–52. ACM (2011)
Kulkarni, A., Can, M., Hartmann, B.: Collaboratively crowdsourcing workflows with turkomatic. In: Proc. CSCW 2012, pp. 1003–1012. ACM (2012)
Satzger, B., Psaier, H., Schall, D., Dustdar, S.: Auction-based crowdsourcing supporting skill management. Information Systems 38(4), 547–560 (2013)
Kittur, A., Nickerson, J.V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., Horton, J.: The future of crowd work. In: Proc. CSCW 2013, pp. 1301–1318. ACM (2013)
Weld, D.S., Adar, E., Chilton, L., Hoffmann, R., Horvitz, E., Koch, M., Landay, J., Lin, C.H., Mausam, M.: Personalized online educationa crowdsourcing challenge. In: Proc. HCOMP 2012 (2012)
Kochhar, S., Mazzocchi, S., Paritosh, P.: The anatomy of a large-scale human computation engine. In: Proc. HCOMP 2010, pp. 10–17. ACM (2010)
Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., Movellan, J.R.: Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In: Proc. NIPS 2009, pp. 2035–2043 (2009)
Welinder, P., Branson, S., Belongie, S., Perona, P.: The multidimensional wisdom of crowds. In: Proc. NIPS 2010, pp. 2424–2432 (2010)
Lin, C.H., Mausam, M., Weld, D.S.: Crowdsourcing control: Moving beyond multiple choice. In; Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence (2012)
Yan, Y., Fung, G.M., Rosales, R., Dy, J.G.: Active learning from crowds. In: Proc. ICML 2011, pp. 1161–1168 (2011)
Wauthier, F.L., Jordan, M.I.: Bayesian bias mitigation for crowdsourcing. In: Proc. NIPS 2011, pp. 1800–1808 (2011)
Donmez, P., Carbonell, J., Schneider, J.: A probabilistic framework to learn from multiple annotators with time-varying accuracy. In: Proc. SD 2010, pp. 826–837 (2010)
Kacorri, H., Shinkawa, K., Saito, S.: Introducing game elements in crowdsourced video captioning by non-experts. In: Proc. W4A 20114. ACM (2014) (in press)
Sobhi, A., Nagatsuma, R., Saitoh, T.: Collaborative caption editing system—enhancing the quality of caption editing system. In: CSUN 2012 (2012)
von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Proc. CHI 2004, pp. 319–326. ACM (2004)
Ishihara, T., Itoko, T., Sato, D., Tzadok, A., Takagi, H.: Transforming Japanese archives into accessible digital books. In: Proc. JCDL 2012, pp. 91–100. ACM (2012)
Kobayashi, M., Ishihara, T., Itoko, T., Takagi, H., Asakawa, C.: Age-based Task Specialization for Crowdsourced Proofreading. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2013, Part II. LNCS, vol. 8010, pp. 104–112. Springer, Heidelberg (2013)
Itoko, T., Arita, S., Kobayashi, M., Takagi, H.: Involving senior workers in crowdsourced proofreading. In: Stephanidis, C., Antona, M. (eds.) UAHCI/HCII 2014, Part III. LNCS, vol. 8515, pp. 106–117. Springer, Heidelberg (2014)
Leung, R., Tang, C., Haddad, S., McGrenere, J., Graf, P., Ingriany, V.: How Older Adults Learn to Use Mobile Devices: Survey and Field Investigations. ACM Trans. Access. Comput. 4(3), Article 11 (2012)
Kobayashi, M., Ishihara, T., Kosugi, A., Takagi, H., Asakawa, C.: Question-Answer Cards for an Inclusive Micro-Tasking Framework for the Elderly. In: Kotzé, P., Marsden, G., Lindgaard, G., Wesson, J., Winckler, M. (eds.) INTERACT 2013, Part III. LNCS, vol. 8119, pp. 590–607. Springer, Heidelberg (2013)
Huang, J., Twidale, M.B.: Graphstract: Minimal graphical help for computers. In: Proc. UIST 200707, pp. 203–212. ACM (2007)
Rowe, S.C., Cohen, C.J., Tang, K.K.: Applying computer game tutorial design techniques to simulation-based training. In: Proc. 2004 Fall SIW (2004)
Isbister, K., Schaffer, N.: “gamenics” and its potential. In: Game Usability: Advancing the Player Experience. CRC Press (2008)
DiMicco, J., Millen, D.R., Geyer, W., Dugan, C., Brownholtz, B., Muller, M.: Motivations for social networking at work. In: Proc. CSCW 2008, pp. 711–720. ACM (2008)
Kaufmann, N., Schulze, T., Veit, D.: More than fun and money. worker motivation in crowdsourcing-a study on mechanical turk. In: Proc. AMCIS 2011 (2011)
Dai, P., Mausam, W.D.S.: Decision-theoretic control of crowd-sourced workflows. In: Twenty-Fourth AAAI Conference on Artificial Intelligence (2010)
Mason, W., Watts, D.J.: Financial incentives and the “performance of crowds”. SIGKDD Explor. Newsl. 11(2), 100–108 (2010)
Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., Vukovic, M.: An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In: Proc. ICWSM 2011 (2011)
oDesk, http://www.odesk.com
CrowdFlower, http://crowdflower.com
Amazon Mechanical Turk, http://www.mturk.com
Duolingo, http://www.duolingo.com
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Saito, S., Watanabe, T., Kobayashi, M., Takagi, H. (2014). Skill Development Framework for Micro-Tasking. In: Stephanidis, C., Antona, M. (eds) Universal Access in Human-Computer Interaction. Universal Access to Information and Knowledge. UAHCI 2014. Lecture Notes in Computer Science, vol 8514. Springer, Cham. https://doi.org/10.1007/978-3-319-07440-5_37
Download citation
DOI: https://doi.org/10.1007/978-3-319-07440-5_37
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07439-9
Online ISBN: 978-3-319-07440-5
eBook Packages: Computer ScienceComputer Science (R0)