Skip to main content
SpringerLink
Log in
Book cover

International Conference on Pattern Recognition and Information Processing

PRIP 2019: Pattern Recognition and Information Processing pp 254–266Cite as

Temporal Convolutional and Recurrent Networks for Image Captioning

  • Conference paper
  • First Online:

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1055))

Abstract

Recently temporal convolutional networks have shown excellent qualities in sequence modeling tasks [1]. Taking this fact into account, in this paper we investigate the possibilities of replacing recurrent networks in architectures targeted specifically at image captioning. We evaluate the solution on Visual Genome dataset [2], which provides extensive set of labels and descriptions that thoroughly grounds visual concepts to natural language.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Bai, S., Colter, J.Z., Coltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)

  2. Krishna, R., Zhu, Y., Groth, O., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis. 1(123), 32–73 (2017)

    Article  MathSciNet  Google Scholar 

  3. Johnson, J., Karpathy, A., Fei-Fei, L.: DenseCap: fully convolutional localization networks for dense captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4565–4574 (2016)

    Google Scholar 

  4. Iskra, N., Iskra, V., Lukashevich M.: Neural network based image understanding with ontological approach. In: Open Semantic Technologies for Intelligent Systems (OSTIS-2019), pp. 113–122 (2019)

    Google Scholar 

  5. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  6. Kulkarni, G., et al.: Baby talk: understanding and generating image descriptions. In: Proceedings of the 24th Conference on Computer Vision and Pattern Recognition (2011)

    Google Scholar 

  7. Sun, C., Gan, C., Nevatia, R.: Automatic concept discovery from parallel text and visual corpora. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2596–2604 (2015)

    Google Scholar 

  8. Hossain, M.D., Sohel, F., Shiratuddin, M.F., Laga, H.: A comprehensive survey of deep learning for image captioning. ACM Comput. Surv. (CSUR) 6(51), 118 (2019)

    Google Scholar 

  9. Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3128–3137 (2015)

    Google Scholar 

  10. Liu, D., Hanwang, Zh., Zheng-Jun, Zh., Feng, W.: Explainability by parsing: neural module tree networks for natural language visual grounding. arXiv preprint, arXiv:1812.03299 (2018)

  11. Jiang, W., Ma, L., Jiang, Y.-G., Liu, W., Zhang, T.: Recurrent fusion network for image captioning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 510–526. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_31

    Chapter  Google Scholar 

  12. Wang, Q., Chan, A.B.: CNN + CNN: Convolutional decoders for image captioning. arXiv preprint, arXiv:1805.09019 (2018)

  13. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)

    Google Scholar 

  14. Salimans, T., Kingma, D.: Weight normalization: a simple reparameterization to accelerate training of deep neural networks. In: Advances in Neural Information Processing Systems, pp. 901–909 (2016)

    Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  16. Kingma, D., Ba., J.: Adam: a method for stochastic optimization. arXiv preprint, arXiv:1412.6980 (2014)

  17. Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (2005)

    Google Scholar 

  18. Iskra, N., Shunkevich, D.: Ontological approach to image captioning evaluation. In: Proceedings of Pattern Recognition and Information Processing, pp. 219–223 (2019)

    Google Scholar 

  19. Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Belarusian State University of Informatics and Radioelectronics, Minsk, Republic of Belarus

    Natalia Iskra

  2. Omnigon Communications LLC, New York, NY, USA

    Vitaly Iskra

Authors
  1. Natalia Iskra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vitaly Iskra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Natalia Iskra .

Editor information

Editors and Affiliations

  1. Belarusian State University, Minsk, Belarus

    Sergey V. Ablameyko

  2. Belarusian State University, Minsk, Belarus

    Viktor V. Krasnoproshin

  3. Belarusian State University of Informatics and Radioelectronics, Minsk, Belarus

    Maryna M. Lukashevich

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Iskra, N., Iskra, V. (2019). Temporal Convolutional and Recurrent Networks for Image Captioning. In: Ablameyko, S., Krasnoproshin, V., Lukashevich, M. (eds) Pattern Recognition and Information Processing. PRIP 2019. Communications in Computer and Information Science, vol 1055. Springer, Cham. https://doi.org/10.1007/978-3-030-35430-5_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-35430-5_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-35429-9

  • Online ISBN: 978-3-030-35430-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation