Visual Question Answer System Based on Bidirectional Recurrent Networks

  • Haoyang TangEmail author
  • Meng Qian
  • Ziwei Sun
  • Cong Song
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 891)


Visual Question Answer (VQA) system is the task of automatically answering natural language questions based on the content of reference image. A commonly approach for VQA is to extract image feature and question feature by convolution neural network (CNN) and long short-term memory network (LSTM) respectively, and then combine them to infer the answer through attention mechanism such as the stacked attention networks (SAN). However, the CNN ignores the information between adjacent image regions and the LSTM just memorizes the past contextual information of the question. In this paper, we propose a model based on two bidirectional recurrent networks (BiSRU and BiLSTM) to improve the accuracy of feature extraction. The BiSRU is used to allow the adjacent local region vectors of the image to maintain information each other. The BiLSTM is used to encode the question feature, which obtains past and future contextual information meanwhile when the question is very complex. The feature of image and question obtained by bidirectional recurrent networks is used to predict the answer precisely. Experiment result shows that our model get better performance on four datasets.


Visual question answer system BiSRU network BiLSTM network 



This work was supported by Xi’an Bureau of Science and Technology Program (No. 201805040 YD18CG24 (1)).


  1. 1.
    Karpathy, A.: Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 664–676 (2015). Scholar
  2. 2.
    Kulkarni, G.: Baby talk: understanding and generating simple image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2891–2903 (2013). Scholar
  3. 3.
    Yao, J.: Describing the scene as a whole: joint object detection, scene classification and semantic segmentation. In: 6th IEEE Conference on Computer Vision and Pattern Recognition, pp. 702–709. IEEE, Providence, RI, USA (2012).
  4. 4.
    Zhang, H.: Static correlative filter based convolutional neural network for visual question answering. In: 1th IEEE International Conference on Big Data and Smart Computing, pp. 526–529. IEEE, Shanghai (2018).
  5. 5.
    Chowdhury, I.,: A cascaded long short-term memory (LSTM) driven generic visual question answering (VQA). In: 1th IEEE International Conference on Image Processing (ICIP), pp. 1842–1846. IEEE, Beijing (2017).
  6. 6.
    Yang, Z.: Stacked attention networks for image question answering. In: 6th IEEE Conference on Computer Vision and Pattern Recognition, pp. 21–29. IEEE, Las Vegas, NV, USA (2016).
  7. 7.
    LeCun, Y.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1995). Scholar
  8. 8.
    Rohrbach, M.: Translating video content to natural language descriptions. In: 3th IEEE International Conference on Computer Vision, pp. 433–440. IEEE, Syd-ney, NSW, Australia (2013). doi:
  9. 9.
    Wu, Z.: Verbs semantics and lexical selection. In: 7th Meeting on Association for Computational Linguistics, pp. 133–138. Acl Proceedings of Annual Meeting on Association for Computational Linguistics, Sydney, Australia (1994).

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Xi’an University of Posts and TelecommunicationsXi’anChina

Personalised recommendations