Visual Question Answer System Based on Bidirectional Recurrent Networks
Visual Question Answer (VQA) system is the task of automatically answering natural language questions based on the content of reference image. A commonly approach for VQA is to extract image feature and question feature by convolution neural network (CNN) and long short-term memory network (LSTM) respectively, and then combine them to infer the answer through attention mechanism such as the stacked attention networks (SAN). However, the CNN ignores the information between adjacent image regions and the LSTM just memorizes the past contextual information of the question. In this paper, we propose a model based on two bidirectional recurrent networks (BiSRU and BiLSTM) to improve the accuracy of feature extraction. The BiSRU is used to allow the adjacent local region vectors of the image to maintain information each other. The BiLSTM is used to encode the question feature, which obtains past and future contextual information meanwhile when the question is very complex. The feature of image and question obtained by bidirectional recurrent networks is used to predict the answer precisely. Experiment result shows that our model get better performance on four datasets.
KeywordsVisual question answer system BiSRU network BiLSTM network
This work was supported by Xi’an Bureau of Science and Technology Program (No. 201805040 YD18CG24 (1)).
- 3.Yao, J.: Describing the scene as a whole: joint object detection, scene classification and semantic segmentation. In: 6th IEEE Conference on Computer Vision and Pattern Recognition, pp. 702–709. IEEE, Providence, RI, USA (2012). https://doi.org/10.1109/cvpr.2012.6247739
- 4.Zhang, H.: Static correlative filter based convolutional neural network for visual question answering. In: 1th IEEE International Conference on Big Data and Smart Computing, pp. 526–529. IEEE, Shanghai (2018). https://doi.org/10.1109/bigcomp.2018.00087
- 5.Chowdhury, I.,: A cascaded long short-term memory (LSTM) driven generic visual question answering (VQA). In: 1th IEEE International Conference on Image Processing (ICIP), pp. 1842–1846. IEEE, Beijing (2017). https://doi.org/10.1109/icip.2017.8296600
- 6.Yang, Z.: Stacked attention networks for image question answering. In: 6th IEEE Conference on Computer Vision and Pattern Recognition, pp. 21–29. IEEE, Las Vegas, NV, USA (2016). https://doi.org/10.1109/cvpr.2016.10
- 8.Rohrbach, M.: Translating video content to natural language descriptions. In: 3th IEEE International Conference on Computer Vision, pp. 433–440. IEEE, Syd-ney, NSW, Australia (2013). doi: https://doi.org/10.1109/iccv.2013.61
- 9.Wu, Z.: Verbs semantics and lexical selection. In: 7th Meeting on Association for Computational Linguistics, pp. 133–138. Acl Proceedings of Annual Meeting on Association for Computational Linguistics, Sydney, Australia (1994). https://doi.org/10.1162/ling.1994.00012