Abstract
In this chapter, we discuss detailed statistics of the popular Amazon Mechanical Turk (AMT) marketplace to provide insights in task properties and requester behavior. We present a model to automatically infer requester communities based on task keywords. Hierarchical clustering is used to identify relations between keywords associated with tasks. We present novel techniques to rank communities and requesters by using a graph-based algorithm. Furthermore, we introduce models and methods for the discovery of relevant crowdsourcing brokers who are able to act as intermediaries between requesters and platforms such as AMT.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alonso, O., Rose, D.E., Stewart, B.: Crowdsourcing for relevance evaluation. SIGIR Forum 42(2), 9–15 (2008)
Barabasi, A.-L., Albert, R.: Emergence of scaling in random networks. Science 286, 509 (1999)
Bhattacharyya, P., Garg, A., Wu, S.: Analysis of user keyword similarity in online social networks. Soc. Netw. Anal. Min. 1, 143–158 (2011). doi:10.1007/s13278-010-0006-4
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)
Branting, L.: Context-sensitive detection of local community structure. Soc. Netw. Anal. Min. 1, 1–11 (2012). doi:10.1007/s13278-011-0035-7
Burt, R.S.: Structural Holes: The Social Structure of Competition. Harvard University Press, Cambridge (1992)
Callison-Burch, C., Dredze M.: Creating speech and language data with amazon’s mechanical turk. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10. Association for Computational Linguistics, pp. 1–12. Stroudsburg, PA, USA (2010)
Carvalho, V.R., Lease, M., Yilmaz, E.: Crowdsourcing for search evaluation. SIGIR Forum 44(2), 17–22 (2011)
Cazabet, R., Takeda, H., Hamasaki, M., Amblard, F.: Using dynamic community detection to identify trends in user-generated content. Soc. Netw. Anal. Min. 1–11 (2012). doi:10.1007/s13278-012-0074-8
Chakrabarti, S.: Dynamic personalized pagerank in entity-relation graphs. In: Proceedings of the 16th International Conference on World Wide Web, WWW ’07, pp. 571–580. ACM, New York (2007)
Chang, J., Boyd-Graber, J., Gerrish, S., Wang, C., Blei, D.: Reading tea leaves: How humans interpret topic models. In: Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C.K.I., Culotta, A. (eds.) Advances in Neural Information Processing Systems 22, pp. 288–296. MIT press, Cambridge (2009)
ClickWorker: http://www.clickworker.com/ (2012). Accessed 20 Aug
CrowdFlower: http://crowdflower.com/ (2012). Accessed 20 Aug
Doan, A., Ramakrishnan, R., Halevy, A.Y.: Crowdsourcing systems on the world-wide web. Commun. ACM 54(4), 86–96 (2011)
Eda, T., Yoshikawa, M., Yamamuro, M.: Locally expandable allocation of folksonomy tags in a directed acyclic graph. In: Proceedings of the 9th International Conference on Web Information Systems Engineering, WISE ’08, pp. 151–162. Springer, Berlin, Heidelberg (2008)
Fazeen, M., Dantu, R., Guturu, P.: Identification of leaders, lurkers, associates and spammers in a social network: context-dependent and context-independent approaches. Soc. Netw. Anal. Min. 1, 241–254 (2011). doi:10.1007/s13278-011-0017-9
Fisher, D., Smith, M., Welser, H.T.: You are who you talk to: detecting roles in usenet newsgroups. In: Proceedings of the 39th Annual Hawaii International Conference on System Sciences, HICSS ’06, Vol. 3, p. 59.2, IEEE Computer Society, Washington, DC, USA (2006)
Flickr: http://www.flickr.com/ (2012). Accessed 20 Aug
Fogaras, D., Rácz, B., Csalogány, K., Sarlós, T.: Towards scaling fully personalized pagerank: algorithms, lower bounds, and experiments. Internet Math. 2(3), 333–358 (2005)
Franklin, M.J., Kossmann, D., Kraska, T., Ramesh, S., Xin, R.: Crowddb: answering queries with crowdsourcing. In: Proceedings of the 2011 International Conference on Management of Data, SIGMOD ’11, pp. 61–72. ACM, New York (2011)
Gemmell, J., Shepitsen, A., Mobasher, B., Burke, R.: Personalizing navigation in folksonomies using hierarchical tag clustering. In: Proceedings of the 10th International Conference on Data Warehousing and Knowledge Discovery, DaWaK ’08, pp. 196–205. Springer, Berlin, Heidelberg (2008)
Golder, S., Huberman, B.A.: Usage patterns of collaborative tagging systems. J. Inform. Sci. 32(2), 198–208 (2006)
Haveliwala, T.H.: Topic-sensitive pagerank. In: Proceedings of the 11th International Conference on World Wide Web, WWW ’02, pp. 517–526. ACM, New York (2002)
Heer, J., Bostock, M.: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: Proceedings of the 28th International Conference on Human factors in Computing Systems, CHI ’10, pp. 203–212. ACM, New York (2010)
Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)
Heymann, P., Garcia-Molina, H.: Collaborative creation of communal hierarchical taxonomies in social tagging systems. Technical report, Computer Science Department, Standford University, April (2006)
Howe, J.: The rise of crowdsourcing. Wired 14(14), 1–5 (2006)
Howe, J.: Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business. Crown Business, New York (2008)
Ipeirotis, P.G.: Analyzing the amazon mechanical turk marketplace. XRDS 17, 16–21 (2010)
Jeh, G., Widom, J.: Scaling personalized web search. In: Proceedings of the 12th International Conference on World Wide Web, WWW ’03, pp. 271–279. ACM, New York (2003)
Kittur, A., Chi, E.H., Suh, B.: Crowdsourcing user studies with mechanical turk. In: Proceedings of the 26th Annual SIGCHI Conference on Human factors in Computing Systems, CHI ’08, pp. 453–456. ACM, New York (2008)
Kleinberg, J.M.: Authoritative sources in a hyperlinked environment. J. ACM 46(5), 604–632 (1999)
Kourtellis, N., Alahakoon, T., Simha, R., Iamnitchi, A., Tripathi, R.: Identifying high betweenness centrality nodes in large social networks. Soc. Netw. Anal. Min. 1–16 (2012). doi:10.1007/s13278-012-0076-6
Lampe, C., Resnick, P.: Slash(dot) and burn: distributed moderation in a large online conversation space. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’04, pp. 543–550. ACM, New York (2004)
Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23rd Annual ACM symposium on User Interface Software and Technology, UIST ’10, pp. 57–66. ACM, New York (2010)
Marge, M., Banerjee, S., Rudnicky, A.I.: Using the amazon mechanical turk for transcription of spoken language. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 5270–5273, 2010
Michlmayr, E., Cayzer, S.: Learning user profiles from tagging data and leveraging them for personal(ized) information access. In: Tagging and Metadata for Social Information Organization, Workshop, WWW07, 2007
Munro, R., Bethard, S., Kuperman, V., Lai, V.T., Melnick, R., Potts,C., Schnoebelen, T., Tily, H.: Crowdsourcing and language studies: the new generation of linguistic data. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10, pp. 122–130. Association for Computational Linguistics, Stroudsburg, PA, USA (2010)
oDesk: http://www.odesk.com/ (2012). Accessed 20 Aug
Page, L., Brin, S., Motwani, R., Winograd, T.: Bringing order to the web. The pagerank citation ranking (1999)
Parameswaran, A., Park, H., Garcia-Molina, H., Polyzotis, N., Widom, J.: Deco: declarative crowdsourcing. Technical report, Stanford University (2011)
Psaier, H., Skopik, F., Schall, D., Dustdar, S.: Resource and agreement management in dynamic crowdcomputing environments. In: EDOC, pp. 193–202. IEEE Computer Society, Washington, DC (2011)
Quinn, A.J., Bederson, B.B.: Human computation: a survey and taxonomy of a growing field. In: Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, CHI ’11, pp. 1403–1412. ACM, New York (2011)
Romesburg, C.: Cluster Analysis for Researchers. Krieger, Florida (2004)
Rosvall, M., Bergstrom, C.T.: Maps of random walks on complex networks reveal community structure. PNAS 105, 1118 (2008)
Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Inf. Process. Manage. 24(5), 513–523 (1988)
Samasource: http://samasource.org/ (2012). Accessed 20 Aug
Satzger, B., Psaier, H., Schall, D., Dustdar, S.: Stimulating skill evolution in market-based crowdsourcing. In: BPM, pp. 66–82. Springer, Berlin (2011)
Schall, D.: A human-centric runtime framework for mixed service-oriented systems. Distributed and Parallel Databases, 29, 333–360 (2011). doi: 10.1007/s10619-011-7081-z(Springer, Berlin)
Schall, D.: Expertise ranking using activity and contextual link measures. Data Knowl. Eng. 71(1), 92–113 (2012). doi:10.1016/j.datak.2011.08.001
Schall, D., Skopik, F.: An analysis of the structure and dynamics of large-scale q/a communities. In: Eder, J., Bieliková, M., Tjoa, A.M. (eds.) ADBIS, Lecture Notes in Computer Science, vol. 6909, pp. 285–301. Springer, Berlin (2011)
Schall, D., Skopik, F., Psaier, H., Dustdar, S.: Bridging socially-enhanced virtual communities. In: Chu, W.C., Wong, W.E., Palakal, M.J., Hung, C.-C. (eds.) SAC, pp. 792–799. ACM, New York (2011)
Shepitsen, A., Gemmell, J., Mobasher, B., Burke, R.: Personalized recommendation in social tagging systems using hierarchical clustering. In: Proceedings of the 2008 ACM Conference on Recommender Systems, RecSys’08, pp. 259–266. ACM, New York (2008)
Sigurbjörnsson, B., van Zwol, R.: Flickr tag recommendation based on collective knowledge. In: Proceedings of the 17th International Conference on World Wide Web, WWW’08, pp. 327–336. ACM, New York (2008)
Skopik, F., Schall, D., Dustdar, S.: Start trusting strangers? bootstrapping and prediction of trust. In: Vossen, G., Long, D.D.E., Yu, J.X. (eds.) WISE, Lecture Notes in Computer Science, vol. 5802, pp. 275–289. Springer, Berlin (2009)
SmartSheet. http://www.smartsheet.com/ (2012). Accessed 20 Aug
SpeechInk. http://www.speechink.com/ (2012). Accessed 20 Aug
Vukovic, M.: Crowdsourcing for enterprises. In: Proceedings of the 2009 Congress on Services-I, SERVICES ’09, pp. 686–692. IEEE Computer Society, Washington, DC (2009)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2012 The Author(s)
About this chapter
Cite this chapter
Schall, D. (2012). Crowdsourcing Task Marketplaces. In: Service-Oriented Crowdsourcing. SpringerBriefs in Computer Science. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-5956-9_2
Download citation
DOI: https://doi.org/10.1007/978-1-4614-5956-9_2
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-5955-2
Online ISBN: 978-1-4614-5956-9
eBook Packages: Computer ScienceComputer Science (R0)