Gaussian Processes Regression with Multiple Annotators: When the Annotator Performance Is Not Homogeneous

  • Julián Gil GonzálezEmail author
  • Andrés Marino Álvarez
  • Álvaro Angel Orozco
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11401)


In supervised learning problems, the right label (also known as the gold standard or the ground truth) is not available because the label acquisition can be expensive or infeasible. Instead of that gold standard, we have access to some annotations provided by multiple annotators with different levels of expertise. Hence, trivial methods such as majority voting (or average in regression problems) are not suitable since they assume homogeneity between the expertise of the labelers. In this work, we introduce a regression approach based on Gaussian processes, where we consider that the expertise of the labelers is non-homogeneous across the input space–(GPR-MANH). The idea is to assume that the input space can be represented by a defined number of regions where each annotator exhibit a particular level of expertise. Experimental results show that our methodology can estimate the performance of annotators even if the gold standard is not available, defeating state-of-the-art techniques.



This work was funded by Colciencias under the project with code: 1110-744-55958. J. Gil González is funded by the program “Doctorados Nacionales - Convocatoria 785 de 2017”. A. Orozco was partially funded by Maestría en ingeniería eléctrica from the Universidad Tecnológica de Pereira.


  1. 1.
    Groot, P., Birlutiu, A., Heskes, T.: Learning from multiple annotators with Gaussian processes. In: Honkela, T., Duch, W., Girolami, M., Kaski, S. (eds.) ICANN 2011. LNCS, vol. 6792, pp. 159–164. Springer, Heidelberg (2011). Scholar
  2. 2.
    Wolley, C., Quafafou, M.: Learning from multiple annotators: when data is hard and annotators are unreliable. In: 2012 IEEE 12th International Conference on Data Mining Workshops (ICDMW), pp. 514–521. IEEE (2012)Google Scholar
  3. 3.
    Mozetič, I., Grčar, M., Smailović, J.: Multilingual Twitter sentiment classification: the role of human annotators. PloS One 11(5), e0155036 (2016)CrossRefGoogle Scholar
  4. 4.
    Rodrigues, F., Lourenco, M., Ribeiro, B., Pereira, F.C.: Learning supervised topic models for classification and regression from crowds. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2409–2422 (2017)CrossRefGoogle Scholar
  5. 5.
    González, J.G., Álvarez, M.A., Orozco, Á.A.: Automatic assessment of voice quality in the context of multiple annotations. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6236–6239. IEEE (2015)Google Scholar
  6. 6.
    Raykar, V.C., et al.: Learning from crowds. J. Mach. Learn. Res. 11, 1297–1322 (2010)MathSciNetGoogle Scholar
  7. 7.
    Rodrigues, F., Pereira, F.C., Ribeiro, B.: Gaussian process classification and active learning with multiple annotators. In: ICML, pp. 433–441 (2014)Google Scholar
  8. 8.
    Yan, Y., Rosales, R., Fung, G., Subramanian, R., Dy, J.: Learning from multiple annotators with varying expertise. Mach. Learn. 95(3), 291–327 (2014)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Xiao, H., Xiao, H., Eckert, C.: Learning from multiple observers with unknown expertise. In: Pei, J., Tseng, V.S., Cao, L., Motoda, H., Xu, G. (eds.) PAKDD 2013. LNCS (LNAI), vol. 7818, pp. 595–606. Springer, Heidelberg (2013). Scholar
  10. 10.
    Bishop, C.M.: Pattern recognition. Mach. Learn. 128, 1–58 (2006)Google Scholar
  11. 11.
    Rasmussen, C.E.: Gaussian Processes for Machine Learning. MIT Press, Cambridge (2006)zbMATHGoogle Scholar
  12. 12.
    Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315(5814), 972–976 (2007)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Julián Gil González
    • 1
    Email author
  • Andrés Marino Álvarez
    • 1
  • Álvaro Angel Orozco
    • 1
  1. 1.Faculty of EngineeringUniversidad Tecnológica de PereiraPereiraColombia

Personalised recommendations