Informed Perspectives on Human Annotation Using Neural Signals

  • Graham F. HealyEmail author
  • Cathal Gurrin
  • Alan F. Smeaton
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9517)


In this work we explore how neurophysiological correlates related to attention and perception can be used to better understand the image-annotation task. We explore the nature of the highly variable labelling data often seen across annotators. Our results indicate potential issues with regard to ‘how well’ a person manually annotates images and variability across annotators. We propose such issues arise in part as a result of subjectively interpretable instructions that may fail to elicit similar labelling behaviours and decision thresholds across participants. We find instances where an individual’s annotations differ from a group consensus, even though their EEG signals indicate in fact they were likely in consensus with the group. We offer a new perspective on how EEG can be incorporated in an annotation task to reveal information not readily captured using manual annotations alone. As crowd-sourcing resources become more readily available for annotation tasks one can reconsider the quality of such annotations. Furthermore, with the availability of consumer EEG hardware, we speculate that we are approaching a point where it may be feasible to better harness an annotators time and decisions by examining neural responses as part of the process. In this regard, we examine strategies to deal with inter-annotator sources of noise and correlation that can be used to understand the relationship between annotators at a neural level.


Brain-computer interface EEG Hci Information retrieval Semantic 


  1. 1.
    Ambati, V.: Active learning and crowdsourcing for machine translation in low resource scenarios (2012). aAI3528171Google Scholar
  2. 2.
    Deselaers, T., Keysers, D., Ney, H.: Features for image retrieval: an experimental comparison. Inf. Retrieval 11(2), 77–107 (2008)CrossRefGoogle Scholar
  3. 3.
    Gerson, A.D., Parra, L.C., Sajda, P.: Cortically coupled computer vision for rapid image search. IEEE Trans. Neural Syst. Rehabil. Eng. 14(2), 174–179 (2006)CrossRefGoogle Scholar
  4. 4.
    Healy, G., Smeaton, A.: Eye fixation related potentials in a target search task. In: Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE, pp. 4203–4206, August 2011Google Scholar
  5. 5.
    Healy, G., Gurrin, C., Smeaton, A.F.: Lifelogging and EEG: utilising neural signals for sorting lifelog image data. Quantified Self Europe Conference, 10–11 May 2014, Amsterdam, Netherlands (2014)Google Scholar
  6. 6.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding, pp. 675–678 (2014).
  7. 7.
    Lew, M.S., Sebe, N., Djeraba, C., Jain, R.: Content-based multimedia information retrieval: state of the art and challenges. ACM Trans. Multimedia Comput. Commun. Appl. (TOMM) 2(1), 1–19 (2006)CrossRefGoogle Scholar
  8. 8.
    Mohedano, E., Healy, G., McGuinness, K., Giró-i Nieto, X., O’Connor, N., Smeaton, A.: Improving object segmentation by using EEG signals and rapid serial visual presentation. Multimedia Tools and Applications, pp. 1–23 (2015).
  9. 9.
    Noronha, J., Hysen, E., Zhang, H., Gajos, K.Z.: Platemate: Crowdsourcing nutritional analysis from food photographs, pp. 1–12 (2011).
  10. 10.
    Polich, J.: Updating P300: an integrative theory of P3a and P3b. Clin. Neurophysiol. 118(10), 2128–2148 (2007)CrossRefGoogle Scholar
  11. 11.
    Shenoy, P., Tan, D.: Human-aided computing: utilizing implicit human processing to classify images. In: CHI 2008 Conference on Human Factors in Computing Systems (2008)Google Scholar
  12. 12.
    Welinder, P., Perona, P.: Online crowdsourcing: rating annotators and obtaining cost-effective labels, pp. 25–32, June 2010Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Graham F. Healy
    • 1
    Email author
  • Cathal Gurrin
    • 1
  • Alan F. Smeaton
    • 1
  1. 1.Insight Centre for Data AnalyticsDublin City UniversityGlasnevinIreland

Personalised recommendations