Abstract
Crowdsourcing has emerged as an effective paradigm for accomplishing various intelligent tasks at low costs. However, the labels provided by non-expert crowdsourcing labelers often appear various quality as labelers possess wide-ranging levels of competence. This raises the significant challenges of estimating the true answers for tasks and the reliability of the labelers. Of numerous approaches to estimating labeler quality, expectation-maximization (EM) is widely used by maximizing the likelihood estimates of labeler quality from the observed multiple labels. However, EM-based approaches are easily trapped into local optima. In this paper we use a weight vector to represent the quality (reliability) of corresponding labelers and then using differential evolution (DE) to search optimal weights for different labelers. The experimental results validate the effectiveness of the proposed approach.
This work was partially supported by NSFC (U1711267, 61773355).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bazi, Y., Alajlan, N., Melgani, F., AlHichri, H., Malek, S.: Differential evolution extreme learning machine for the classification of hyperspectral images. IEEE Geosci. Remote Sens. Lett. 11(6), 1066–1070 (2014)
Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. Appl. Stat., 20–28 (1979)
Demartini, G., Difallah, D.E., Cudr-Mauroux, P.: ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In: Proceedings of the 21st International Conference on World Wide Web, pp. 469–478. ACM (2012)
Georgescu, M., Zhu, X.: Aggregation of crowdsourced labels based on worker history. In: Proceedings of the 4th International Conference on Web Intelligence, Mining and Semantics, pp. 1–11. ACM (2014)
Howe, J.: The rise of crowdsourcing. Wired Mag. 14(6), 1–4 (2006)
Maulik, U., Bandyopadhyay, S., Saha, I.: Integrating clustering and supervised learning for categorical data analysis. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 40(4), 664–675 (2010)
Raykar, V.C., Yu, S., Zhao, L.H.: Supervised learning from multiple experts: whom to trust when everyone lies a bit. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 889–896. ACM (2009)
Raykar, V.C., Yu, S., Zhao, L.H.: Learning from crowds. J. Mach. Learn. Res. 11, 1297–1322 (2010)
Sheng, V.S.: Simple multiple noisy label utilization strategies. In: 2011 IEEE 11th International Conference on Data Mining, pp. 635–644. IEEE (2011)
Whitehill, J., Wu, T.F., Bergsma, J., Movellan, J.R., Ruvolo, P.L.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In: Advances in Neural Information Processing Systems, pp. 2035–2043 (2009)
Witten, I.H., Frank, E., Hall, M.A.: Data Mining: Practical Machine Learning Tools and Techniques, 3rd edn. Morgan Kaufmann, San Francisco (2011)
Ye, B., Wang, Y., Liu, L.: Crowd trust: a context-aware trust model for worker selection in crowdsourcing environments. In: ICWS, pp. 121–128. IEEE (2015)
Zhang, Y., Chen, X., Zhou, D., Jordan, M.I.: Spectral methods meet EM: a provably optimal algorithm for crowdsourcing. In: NIPS, pp. 1260–1268 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Qiu, C., Jiang, L., Cai, Z. (2018). Using Differential Evolution to Estimate Labeler Quality for Crowdsourcing. In: Geng, X., Kang, BH. (eds) PRICAI 2018: Trends in Artificial Intelligence. PRICAI 2018. Lecture Notes in Computer Science(), vol 11013. Springer, Cham. https://doi.org/10.1007/978-3-319-97310-4_19
Download citation
DOI: https://doi.org/10.1007/978-3-319-97310-4_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-97309-8
Online ISBN: 978-3-319-97310-4
eBook Packages: Computer ScienceComputer Science (R0)