Abstract
The quality of labels is one of the major issues in crowdsourced labeling tasks. A convenient method for ensuring the quality of labels is to assign the same labeling task to multiple workers and aggregate the labels. Several statistical aggregation methods for single-label classification tasks have been proposed; however, for multi-label classification tasks has not been well studied. Although the existing aggregation methods for single-label classification tasks can be applied to the multi-label classification tasks, they are not designed to incorporate relationships among classes, or they require large computation time. To address these issues, we propose to use RAndom k-labELsets (RAkEL). By incorporating an existing aggregation method for single-label classification tasks into RAkEL, we propose a novel quality control method for crowdsourced multi-label classification. We demonstrate that our method achieves better quality than the existing methods with real data especially when spammers are included in the worker pool.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bragg, J., Mausam, Weld, D.S.: Crowdsourcing multi-label classification for taxonomy creation. In: HCOMP (2013)
Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. Appl. Stat. 28, 20–28 (1979)
Demartini, G., Difallah, D.E., Cudré-Mauroux, P.: Large-scale linked data integration using probabilistic reasoning and crowdsourcing. VLDB J. 22, 665–687 (2013)
Duan, L., Oyama, S., Kurihara, M., Sato, H.: Crowdsourced semantic matching of multi-label annotations. In: Proceedings of the 24th International Conference on Artificial Intelligence, pp. 3483–3489 (2015)
Duan, L., Oyama, S., Sato, H., Kurihara, M.: Separate or joint? Estimation of multiple labels from crowdsourced annotations. Expert Syst. Appl. 41(13), 5723–5732 (2014)
Ekman, P.: An argument for basic emotions. Cogn. Emotion 6(3–4), 169–200 (1992)
Godbole, S., Sarawagi, S.: Discriminative methods for multi-labeled classification. In: Dai, H., Srikant, R., Zhang, C. (eds.) PAKDD 2004. LNCS, vol. 3056, pp. 22–30. Springer, Heidelberg (2004). doi:10.1007/978-3-540-24775-3_5
Nakamura, A.: Kanjo Hyogen Jiten [Dictionary of Emotive Expressions]. Tokyodo (1993)
Oyama, S., Baba, Y., Sakurai, Y., Kashima, H.: Accurate integration of crowdsourced labels using workers’ self-reported confidence scores. In: Proceedings of the 23rd International Joint Conference on Artificial Intelligence, pp. 2554–2560 (2013)
Tsoumakas, G., Katakis, I.: Multi-label classification: an overview. Int. J. Data Warehouse. Min. 3(3), 1–11 (2007)
Tsoumakas, G., Katakis, I., Vlahavas, I.: Random k-labelsets for multi-label classification. IEEE Trans. Knowl. Data Eng. 23(7), 1079–1089 (2011)
Welinder, P., Branson, S., Perona, P., Belongie, S.J.: The multidimensional wisdom of crowds. In: Advances in Neural Information Processing Systems, vol. 23, pp. 2424–2432 (2010)
Whitehill, J., Fan Wu, T., Bergsma, J., Movellan, J.R., Ruvolo, P.L.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In: Advances in Neural Information Processing Systems, vol. 22, pp. 2035–2043 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Yoshimura, K., Baba, Y., Kashima, H. (2017). Quality Control for Crowdsourced Multi-label Classification Using RAkEL. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, ES. (eds) Neural Information Processing. ICONIP 2017. Lecture Notes in Computer Science(), vol 10634. Springer, Cham. https://doi.org/10.1007/978-3-319-70087-8_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-70087-8_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70086-1
Online ISBN: 978-3-319-70087-8
eBook Packages: Computer ScienceComputer Science (R0)