Template Based Approach for Augmenting Image Descriptions

  • Akshansh ChahalEmail author
  • Manshul Belani
  • Akashdeep Bansal
  • Neha Jadhav
  • Meenakshi Balakrishnan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10896)


With the increasing focus on digital learning, it has become extremely important that digital content is available with ease. However, a lot of this digital content is not generated keeping Universal Access in mind. Most of such content available is either completely inaccessible or only partially accessible to the print disabled people. One of the major gaps in accessibility of digital content, especially electronic books is the lack of alternative texts for diagrams and ineffective descriptions in cases they are present. The paper discusses the design of a template, which can help in augmenting descriptions for textbook diagrams. The template consists of various components, which are populated using the information present in the diagram or from the text surrounding the diagram in the textbook. This template provides means for generation of comprehensible diagram descriptions, which not only help the user to visualize the diagram but also create a mental model of the layout of the diagram. Observations made during the user study validate the effectiveness of these augmented descriptions.


Accessibility Blind Visually impaired eBooks Template Augmentation Image Description 


  1. 1.
    Yang, Y., Teo, C.L., Daumé III, H., Aloimonos, Y.: Corpus-guided sentence generation of natural images. In: Proceedings of Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK, pp. 444–454, July 2011Google Scholar
  2. 2.
    Kuznetsova, P., Ordonez, V., Berg, A.C., Berg, T.L., Choi, Y.: Collective generation of natural image descriptions. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Republic of Korea, pp. 359–368, July 2012Google Scholar
  3. 3.
    Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. arXiv:1502.03044v3 [cs.LG], April 2016
  4. 4.
    Elamri, C., de Planque, T.: Automated Neural Image Caption Generator for Visually Impaired People (2016)Google Scholar
  5. 5.
    Hodosh, M., Young, P., Hockenmaier, J.: Framing image description as a ranking task: data, models and evaluation metrics. J. Artif. Intell. Res. 47, 853–899 (2013)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Ordonez, V., Kulkarni, G., Berg, T.L.: Im2Text: Describing images using 1 million captioned photographs. In: Advances in Neural Information Processing Systems 24 (NIPS 2011) (2011)Google Scholar
  7. 7.
    Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.L.: BabyTalk: understanding and generating simple image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2891–2903 (2013)CrossRefGoogle Scholar
  8. 8.
    Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A., Berg, T., Daume, H.: Midge: generating image descriptions from computer vision detections. In: 13th Conference of the European Chapter of the Association for Computational Linguistics, pp. 747–756 (2012)Google Scholar
  9. 9.
    Li, S., Kulkarni, G., Berg, T.L., Berg, A.C., Choi, Y.: Composing simple image descriptions using web-scale N-grams. In: Fifteenth Conference on Computational Natural Language Learning, pp. 220–228 (2011)Google Scholar
  10. 10.
    Elliott, D., Keller, F.: Image description using visual dependency representations. In: 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1292–1302 (2013)Google Scholar
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Akshansh Chahal
    • 1
    Email author
  • Manshul Belani
    • 1
  • Akashdeep Bansal
    • 1
  • Neha Jadhav
    • 1
  • Meenakshi Balakrishnan
    • 1
  1. 1.Indian Institute of Technology - DelhiNew DelhiIndia

Personalised recommendations