Abstract
In this chapter we address multi-modality in domains where not only text but also images or, as we will see next, scientific figures and diagrams are important sources of information for the task at hand. Compared to natural images, understanding scientific figures is particularly hard for machines. However, there is a valuable source of information in scientific literature that until now remained untapped: the correspondence between a figure and its caption. In this chapter we show what can be learnt by looking at a large number of figures and reading their captions, and describe a figure-caption correspondence learning task that makes use of such observation. Training visual and language networks without supervision other than pairs of unconstrained figures and captions is shown to successfully solve this task. We also follow up on previous chapters and illustrate how transferring lexical and semantic knowledge from a knowledge graph significantly enriches the resulting features. Finally, the positive impact of such hybrid and semantically enriched features is demonstrated in two transfer learning experiments involving scientific text and figures: multi-modal classification and machine comprehension for question answering.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Gomez-Perez, J.M., Denaux, R., Garcia-Silva, A. (2020). Jointly Learning Text and Visual Information in the Scientific Domain. In: A Practical Guide to Hybrid Natural Language Processing. Springer, Cham. https://doi.org/10.1007/978-3-030-44830-1_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-44830-1_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-44829-5
Online ISBN: 978-3-030-44830-1
eBook Packages: Computer ScienceComputer Science (R0)