Skip to main content
SpringerLink
Log in

Imprecise Label Aggregation Approach Under the Belief Function Theory

  • Conference paper
  • First Online:
Book cover Intelligent Systems Design and Applications (ISDA 2018 2018)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 941))

Abstract

Crowdsourcing has become a phenomenon of increasing interest in several research fields such as artificial intelligence. It typically uses human cognitive ability in order to effectively solve tasks that can hardly be addressed by automated computation. The major problem however is that so far studies could not completely control the quality of obtained data since contributors are uncertainly reliable. In this work, we propose an approach that aggregates labels using the belief function theory under the assumption that these labels could be partial hence imprecise. Simulated data demonstrate that our method produces more reliable aggregation results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Zheng, Y., Wang, J., Li, G., Feng, J.: QASCA: a quality-aware task assignment system for crowdsourcing applications. In: International Conference on Management of Data, pp. 1031–1046 (2015)

    Google Scholar 

  2. Yan, T., Kumar, V., Ganesan, D.: Designing games with a purpose. Commun. ACM 51(8), 58–67 (2008)

    Google Scholar 

  3. Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast but is it good? Evaluation non-expert annotations for natural language tasks. In: The Conference on Empirical Methods in Natural Languages Processing, pp. 254–263 (2008)

    Google Scholar 

  4. Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: International Conference on Knowledge Discovery and Data Mining, pp. 614–622 (2008)

    Google Scholar 

  5. Shafer, G.: A Mathematical Theory of Evidence, vol. 1. Princeton University Press, Princeton (1976)

    MATH  Google Scholar 

  6. Dempster, A.P.: Upper and lower probabilities induced by a multivalued mapping. In: The Annals of Mathematical Statistics, pp. 325–339 (1967)

    Article  MathSciNet  Google Scholar 

  7. Jousselme, A.-L., Grenier, D., Bossé, É.: A new distance between two bodies of evidence. In: Information Fusion, pp. 91–101 (2001)

    Article  Google Scholar 

  8. Lefèvre, E., Elouedi, Z.: How to preserve the confict as an alarm in the combination of belief functions? Decis. Support Syst. 56, 326–333 (2013)

    Article  Google Scholar 

  9. Smets, P.: The combination of evidence in the transferable belief model. IEEE Trans. Pattern Anal. Mach. Intell. 12(5), 447–458 (1990)

    Article  Google Scholar 

  10. Raykar, V.C., Yu, S.: Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J. Mach. Learn. Res. 13, 491–518 (2012)

    MathSciNet  MATH  Google Scholar 

  11. Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. Appl. Stat. 28, 20–28 (2010)

    Article  Google Scholar 

  12. Khattak, F.K., Salleb, A.: Quality control of crowd labeling through expert evaluation. In: The Neural Information Processing Systems 2nd Workshop on Computational Social Science and the Wisdom of Crowds, pp. 27–29 (2011)

    Google Scholar 

  13. Smets, P., Mamdani, A., Dubois, D., Prade, H.: Non Standard Logics for Automated Reasoning, pp. 253–286. Academic Press, London (1988)

    Google Scholar 

  14. Ben Rjab, A., Kharoune, M., Miklos, Z., Martin, A.: Characterization of experts in crowdsourcing platforms. In: International Conference on BELIEF 2016, pp. 97–104 (2016)

    Google Scholar 

  15. Watanabe, M., Yamaguchi, K.: The EM Algorithm and Related Statistical Models, 250 p. CRC Press, Boca Raton (2003)

    Google Scholar 

  16. Whitehill, J., Wu, T., Bergsma, J., Movellan, J.R., Ruvolo, P.L.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In: Neural Information Processing Systems, pp. 2035–2043 (2009)

    Google Scholar 

  17. Abassi, L., Boukhris, I.: Crowd label aggregation under a belief function framework. In: International Conference on Knowledge Science, Engineering and Management, pp. 185–196. Springer (2016)

    Google Scholar 

  18. Abassi, L., Boukhris, I.: A gold standards-based crowd label aggregation within the belief function theory. In: International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pp. 97–106. Springer (2017)

    Google Scholar 

  19. Abassi, L., Boukhris, I.: Iterative aggregation of crowdsourced tasks within the belief function theory. In: European Conference on Symbolic and Quantitative Approaches to Reasoning and Uncertainty, pp. 159–168. Springer (2017)

    Google Scholar 

  20. Abassi, L., Boukhris, I.: A worker clustering-based approach of label aggregation under the belief function theory. In: Applied Intelligence, pp. 1573–7497 (2018)

    Article  Google Scholar 

  21. Florentin, S., Arnaud, M., Christophe, O.: Contradiction measures and specificity degrees of basic belief assignments. In: 14th International Conference on Information Fusion, pp. 1–8 (2011)

    Google Scholar 

  22. Kuncheva, L., et al.: Limits on the majority vote accuracy in classifier fusion. Pattern Anal. Appl. 6, 22–31 (2003)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LARODEC Laboratory, Institut Supérieur de Gestion de Tunis, University of Tunis, Tunis, Tunisia

    Lina Abassi & Imen Boukhris

Authors
  1. Lina Abassi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Imen Boukhris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lina Abassi .

Editor information

Editors and Affiliations

  1. Machine Intelligence Research Labs, Auburn, WA, USA

    Ajith Abraham

  2. School of Information Technology and Engineering, Vellore Institute of Technology, Vellore, Tamil Nadu, India

    Aswani Kumar Cherukuri

  3. Tijuana Institute of Technology, Tijuana, Mexico

    Patricia Melin

  4. Machine Intelligence Research Labs, Auburn, WA, USA

    Niketa Gandhi

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Abassi, L., Boukhris, I. (2020). Imprecise Label Aggregation Approach Under the Belief Function Theory. In: Abraham, A., Cherukuri, A., Melin, P., Gandhi, N. (eds) Intelligent Systems Design and Applications. ISDA 2018 2018. Advances in Intelligent Systems and Computing, vol 941. Springer, Cham. https://doi.org/10.1007/978-3-030-16660-1_59

Download citation

Publish with us

Policies and ethics

Search

Navigation