Combining fMRI Data and Neural Networks to Quantify Contextual Effects in the Brain

  • Nora Aguirre-CelisEmail author
  • Risto Miikkulainen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11309)


Does word meaning change according to the context? Although this hypothesis has existed for a long time, only recently it has become possible to test it based on neuroimaging. Embodiment theories of knowledge representation suggest that word meaning consist of a collection of attributes defined in terms of various neural systems. This approach represents an unlimited number of objects through weighted attributes and the weights may change in context. This paper aims at quantifying such dynamic meanings using computational modeling. A neural network is trained with backpropagation to map attribute-based representations to fMRI images of subjects reading everyday sentences. Backpropagation is then extended to the features, demonstrating how they change in different sentence contexts for the same word. Indeed, statistically significant changes occurred across similar contexts and across different subjects, quantifying for the first time how attribute weightings for the same word are modified by context. Such dynamic representations of meaning could be used in future natural language processing systems, allowing them to mirror human performance more accurately.


Context effect Concept representations fMRI data analysis Neural networks Embodied cognition 



We would like to thank Jeffery Binder (Medical College of Wisconsin), Rajeev Raizada and Andrew Anderson (University of Rochester), Mario Aguilar and Patrick Connolly (Teledyne Scientific Company) for providing this data and insight for this research. This work was supported in part by IARPA-FA8650-14-C-7357 and by NIH 1U01DC014922 grants.


  1. 1.
    Regier, T.: The Human Semantic Potential. MIT Press, Cambridge (1996)Google Scholar
  2. 2.
    Landau, B., Smith, L., Jones, S.: Object perception and object naming in early development. Trends Cogn. Sci. 27, 19–24 (1998)CrossRefGoogle Scholar
  3. 3.
    Barsalou, L.W.: Grounded cognition. Annl. Rev. Psyc. 59, 617–845 (2008)CrossRefGoogle Scholar
  4. 4.
    Binder, J.R., Desai, R.H., Graves, W.W., Conant, L.L.: Where is the semantic system? A critical review of 120 neuroimaging studies. Cereb. Cortex 19, 2767–2769 (2009)CrossRefGoogle Scholar
  5. 5.
    Binder, J.R., Desai, R.H.: The neurobiology of semantic memory. Trends Cogn. Sci. 15(11), 527–536 (2011)CrossRefGoogle Scholar
  6. 6.
    Binder, J.R., et al.: Toward a brain-based Comp. Sem. Cogn. Neuropsychol. 33(3–4), 130–174 (2016)CrossRefGoogle Scholar
  7. 7.
    Binder, J.R.: In defense of abstract conceptual representations. Psychon. Bull. Rev. 23, 1096–1108 (2016)CrossRefGoogle Scholar
  8. 8.
    Pecher, D., Zeelenberg, R., Barsalou, L.W.: Sensorimotors simulations underlie conceptual representations: modality-specific effects of prior activation. Psychon. Bull. Rev. 11, 164–167 (2004)CrossRefGoogle Scholar
  9. 9.
    Aguirre-Celis, N., Miikkulainen R.: From words to sentences & back: characterizing context-dependent meaning rep in the brain. In: Proceedings of the 39th Annual Meeting of the Cognitive Science Society, London, UK, pp. 1513–1518 (2017)Google Scholar
  10. 10.
    Glasgow, K., Roos, M., Haufler, A. J., Chevillet, M., A., Wolmetz, M.: Evaluating semantic models with word-sentence relatedness. arXiv:1603.07253 (2016)
  11. 11.
    Anderson, A.J., et al.: Perdicting Neural activity patterns associated with sentences using neurobiologically motivated model of semantic representation. Cereb. Cortex 1–17 (2016).
  12. 12.
    Burgess, C.: From simple associations to the building blocks of language: modeling meaning with HAL. Behav. Res. Methods Inst. Com. 30, 188–198 (1998)CrossRefGoogle Scholar
  13. 13.
    Landauer, T.K., Dumais, S.T.: A solution to plato’s problem: the latent semantic analysis theory. Psychol. Rev. 104, 211–240 (1997)CrossRefGoogle Scholar
  14. 14.
    Vinyals, O., Toshev, A., Bengio, S., Erham, D.: Show and tell: a new image caption generator. arXiv:1506.03134v2 (2015)
  15. 15.
    Miikkulainen, R., Dyer, M.G.: Natural language processing with modular PDP networks and distributed lexicon. Cogn. Sci. 15, 343–399 (1991)CrossRefGoogle Scholar
  16. 16.
    Estes, Z., Golonka, S., Jones, L.L.: Thematic thinking: the apprehension and consequences of thematic relations. Psychol. Learn. Motiv. 54, 249–294 (2011)CrossRefGoogle Scholar
  17. 17.
    Anderson, A.J., et al.: Multiple regions of a cortical network commonly encode the meaning of words in multiple grammatical positions of read sentences. Cereb. Cortex 1–16 (2018).
  18. 18.
    Mitchell, J., Lapata, M.: Composition in distributional models of semantics. Cogn. Sci. 38(8), 1388–1439 (2010). Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.ITESMMonterreyMexico
  2. 2.The University of Texas at AustinAustinUSA

Personalised recommendations