Abstract
Human vision relies on saccades to extract high quality information on small areas of the field of view, pointing the high resolution region of the retina (i.e. fovea) to the regions of interest. The eye motions are guided by top-down information provided by the task, which in our case is the search for a given object. In this work we propose a Recurrent Neural Network (RNN) model that learns from human demonstrations how to explore an image. The exploration samples are obtained from eye tracking data acquired while subjects inspect images. The proposed model extracts visual features from Convolutional Neural Networks (CNNs), which correspond to the input of the RNN. The contribution of this work is to consider the visual features along with the object label in a new model that is able to search for a given object in an image. We make a comparative study on the importance of context during object search tasks, showing that foveated images perform better than uniform image region crops.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Subjects were asked to fixate the target object when the object was found.
- 2.
For each experiment, subjects were asked to search for a particular object for two seconds and indicate if they believed the object was present or not.
References
Posner, M.: Cognitive Neuroscience of Attention. Guilford Press (2012). http://books.google.pt/books?id=8yjEjoS7EQsC
Geisler, W.S., Perry, J.S.: Real-time foveated multiresolution system for low-bandwidth video communication. In: Photonics West 1998 Electronic Imaging, pp. 294–305. International Society for Optics and Photonics (1998)
Almeida, A.F., Figueiredo, R., Bernardino, A., Santos-Victor, J.: Deep networks for human visual attention: a hybrid model using foveal vision. In: Ollero, A., Sanfeliu, A., Montano, L., Lau, N., Cardeira, C. (eds.) ROBOT 2017. AISC, vol. 694, pp. 117–128. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-70836-2_10
Zhang, L., Tong, M.H., Marks, T.K., Shan, H., Cottrell, G.W.: Sun: a Bayesian framework for saliency using natural statistics. J. Vis. 8(7), 32–32 (2008)
Jiang, M., Boix, X., Roig, G., Xu, J., Van Gool, L., Zhao, Q.: Learning to predict sequences of human visual fixations. IEEE Trans. Neural Netw. Learn. Syst. 27(6), 1241–1252 (2016)
Mnih, V., Heess, N., Graves, A., et al.: Recurrent models of visual attention. In: Advances in Neural Information Processing Systems, pp. 2204–2212 (2014)
Ngo, T., Manjunath, B.: Saccade gaze prediction using a recurrent neural network. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 3435–3439. IEEE (2017)
Nunes, A., Figueiredo, R., Moreno, P.: Learning to perform visual tasks from human demonstrations. In: Morales, A., Fierrez, J., Sánchez, J.S., Ribeiro, B. (eds.) Pattern Recognition and Image Analysis, pp. 346–358. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31321-0_30
Burt, P.J., Adelson, E.H.: The Laplacian pyramid as a compact image code. IEEE Trans. Commun. 31, 532–540 (1983)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1725–1780 (1997)
Shi, X., Chen, Z.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting (2015)
Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., Quillen, D.: Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int. J. Robot. Res. 37(4–5), 421–436 (2018)
He, K., Zhang, X.: Deep residual learning for image recognition (2015)
Shulga, D.: Word embeddings using deep neural network image classifier (2018). https://towardsdatascience.com/creating-words-embedding-using-deep-neural-network-image-classifier-ae2594d3862d
Ehinger, K.A., Hidalgo-Sotelo, B., Torralba, A., Oliva, A.: Modelling search for people in 900 scenes: a combined source model of eye guidance. Vis. Cogn. 17(6–7), 945–978 (2009)
Koehler, K., Guo, F., Zhang, S., Eckstein, M.P.: What do saliency models predict. J. Vis. 14, 14 (2014)
Buja, A., Stuetzle, W., Shen, Y.: Loss functions for binary class probability estimation and classification: Structure and applications (2005)
Acknowledgements
To the grant program “New Talents in Artificial Intelligence” from Fundação Calouste Gulbenkian, the Portuguese Science Foundation (FCT) funding [UID/EEA/50009/2019] and LARSyS - FCT Plurianual funding 2020–2023.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Nunes, A., Figueiredo, R., Moreno, P. (2020). Learning to Search for Objects in Images from Human Gaze Sequences. In: Campilho, A., Karray, F., Wang, Z. (eds) Image Analysis and Recognition. ICIAR 2020. Lecture Notes in Computer Science(), vol 12131. Springer, Cham. https://doi.org/10.1007/978-3-030-50347-5_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-50347-5_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-50346-8
Online ISBN: 978-3-030-50347-5
eBook Packages: Computer ScienceComputer Science (R0)