Gaussian Processes Regression with Multiple Annotators: When the Annotator Performance Is Not Homogeneous
In supervised learning problems, the right label (also known as the gold standard or the ground truth) is not available because the label acquisition can be expensive or infeasible. Instead of that gold standard, we have access to some annotations provided by multiple annotators with different levels of expertise. Hence, trivial methods such as majority voting (or average in regression problems) are not suitable since they assume homogeneity between the expertise of the labelers. In this work, we introduce a regression approach based on Gaussian processes, where we consider that the expertise of the labelers is non-homogeneous across the input space–(GPR-MANH). The idea is to assume that the input space can be represented by a defined number of regions where each annotator exhibit a particular level of expertise. Experimental results show that our methodology can estimate the performance of annotators even if the gold standard is not available, defeating state-of-the-art techniques.
This work was funded by Colciencias under the project with code: 1110-744-55958. J. Gil González is funded by the program “Doctorados Nacionales - Convocatoria 785 de 2017”. A. Orozco was partially funded by Maestría en ingeniería eléctrica from the Universidad Tecnológica de Pereira.
- 2.Wolley, C., Quafafou, M.: Learning from multiple annotators: when data is hard and annotators are unreliable. In: 2012 IEEE 12th International Conference on Data Mining Workshops (ICDMW), pp. 514–521. IEEE (2012)Google Scholar
- 5.González, J.G., Álvarez, M.A., Orozco, Á.A.: Automatic assessment of voice quality in the context of multiple annotations. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6236–6239. IEEE (2015)Google Scholar
- 7.Rodrigues, F., Pereira, F.C., Ribeiro, B.: Gaussian process classification and active learning with multiple annotators. In: ICML, pp. 433–441 (2014)Google Scholar
- 10.Bishop, C.M.: Pattern recognition. Mach. Learn. 128, 1–58 (2006)Google Scholar