Abstract
In this work we explore how neurophysiological correlates related to attention and perception can be used to better understand the image-annotation task. We explore the nature of the highly variable labelling data often seen across annotators. Our results indicate potential issues with regard to ‘how well’ a person manually annotates images and variability across annotators. We propose such issues arise in part as a result of subjectively interpretable instructions that may fail to elicit similar labelling behaviours and decision thresholds across participants. We find instances where an individual’s annotations differ from a group consensus, even though their EEG signals indicate in fact they were likely in consensus with the group. We offer a new perspective on how EEG can be incorporated in an annotation task to reveal information not readily captured using manual annotations alone. As crowd-sourcing resources become more readily available for annotation tasks one can reconsider the quality of such annotations. Furthermore, with the availability of consumer EEG hardware, we speculate that we are approaching a point where it may be feasible to better harness an annotators time and decisions by examining neural responses as part of the process. In this regard, we examine strategies to deal with inter-annotator sources of noise and correlation that can be used to understand the relationship between annotators at a neural level.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
As we are using non-parametric (rank based) statistics in our analysis, there is no difference between averaging a subset of EEG-prediction scores or taking their sum i.e. the respective underlying ranking remains the same.
References
Ambati, V.: Active learning and crowdsourcing for machine translation in low resource scenarios (2012). aAI3528171
Deselaers, T., Keysers, D., Ney, H.: Features for image retrieval: an experimental comparison. Inf. Retrieval 11(2), 77–107 (2008)
Gerson, A.D., Parra, L.C., Sajda, P.: Cortically coupled computer vision for rapid image search. IEEE Trans. Neural Syst. Rehabil. Eng. 14(2), 174–179 (2006)
Healy, G., Smeaton, A.: Eye fixation related potentials in a target search task. In: Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE, pp. 4203–4206, August 2011
Healy, G., Gurrin, C., Smeaton, A.F.: Lifelogging and EEG: utilising neural signals for sorting lifelog image data. Quantified Self Europe Conference, 10–11 May 2014, Amsterdam, Netherlands (2014)
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding, pp. 675–678 (2014). http://doi.acm.org/10.1145/2647868.2654889
Lew, M.S., Sebe, N., Djeraba, C., Jain, R.: Content-based multimedia information retrieval: state of the art and challenges. ACM Trans. Multimedia Comput. Commun. Appl. (TOMM) 2(1), 1–19 (2006)
Mohedano, E., Healy, G., McGuinness, K., Giró-i Nieto, X., O’Connor, N., Smeaton, A.: Improving object segmentation by using EEG signals and rapid serial visual presentation. Multimedia Tools and Applications, pp. 1–23 (2015). http://dx.doi.org/10.1007/s11042-015-2805-0
Noronha, J., Hysen, E., Zhang, H., Gajos, K.Z.: Platemate: Crowdsourcing nutritional analysis from food photographs, pp. 1–12 (2011). http://doi.acm.org/10.1145/2047196.2047198
Polich, J.: Updating P300: an integrative theory of P3a and P3b. Clin. Neurophysiol. 118(10), 2128–2148 (2007)
Shenoy, P., Tan, D.: Human-aided computing: utilizing implicit human processing to classify images. In: CHI 2008 Conference on Human Factors in Computing Systems (2008)
Welinder, P., Perona, P.: Online crowdsourcing: rating annotators and obtaining cost-effective labels, pp. 25–32, June 2010
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Healy, G.F., Gurrin, C., Smeaton, A.F. (2016). Informed Perspectives on Human Annotation Using Neural Signals. In: Tian, Q., Sebe, N., Qi, GJ., Huet, B., Hong, R., Liu, X. (eds) MultiMedia Modeling. MMM 2016. Lecture Notes in Computer Science(), vol 9517. Springer, Cham. https://doi.org/10.1007/978-3-319-27674-8_28
Download citation
DOI: https://doi.org/10.1007/978-3-319-27674-8_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-27673-1
Online ISBN: 978-3-319-27674-8
eBook Packages: Computer ScienceComputer Science (R0)