Abstract
Crowdsourcing in the form of human-based electronic services provides a powerful way of outsourcing so called human intelligence tasks (HITs) to a large workforce of people over the Internet. Because of the limited control over that workforce, it is challenging to ensure the quality of the work results. Several approaches have been proposed that can be applied to specific types of HITs. However, it is difficult to identify a suitable quality management approach for any given type of HIT. This paper aims to provide a first sketch of a decision matrix.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Barr, J., Cabrera, L.F.: AI gets a brain. ACM Queue 4(4), 24–29 (2006)
Kern, R., Bauer, C., Thies, H., Satzger, G.: Validating results of human-based electronic services leveraging multiple reviewers. In: Proceedings of the 16th Americas Conference on Information Systems (AMCIS), Lima, Peru (2010) (forthcoming)
Kern, R., Zirpins, C., Agarwal, S.: Managing quality of Human-Based eServices. In: Feuerlicht, G., Lamersdorf, W. (eds.) Service-Oriented Computing - ICSOC 2008 Workshops. LNCS, vol. 5472, pp. 304–309. Springer, Heidelberg (2008)
Rowe, G., Wright, G.: The delphi technique as a forecasting tool: issues and analysis. International journal of forecasting 15(4), 353–375 (1999)
Snow, R., OConnor, B., Jurafsky, D., Ng, A.Y.: Cheap and fastbut is it good? evaluating non-expert annotations for natural language tasks. In: EMNLP 2008: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 254–263. ACL, Stroudsburg (2008)
Sorokin, A., Forsyth, D.: Utility data annotation with amazon mechanical turk. In: CVPRW 2008: Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–8. IEEE Computer Society, Washington (June 2008)
Surowiecki, J.: The Wisdom of Crowds, 1st edn. Doubleday, New York (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kern, R., Thies, H., Bauer, C., Satzger, G. (2010). Quality Assurance for Human-Based Electronic Services: A Decision Matrix for Choosing the Right Approach. In: Daniel, F., Facca, F.M. (eds) Current Trends in Web Engineering. ICWE 2010. Lecture Notes in Computer Science, vol 6385. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-16985-4_39
Download citation
DOI: https://doi.org/10.1007/978-3-642-16985-4_39
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-16984-7
Online ISBN: 978-3-642-16985-4
eBook Packages: Computer ScienceComputer Science (R0)