Skip to main content
SpringerLink
Log in

Investigating Lexical and Semantic Cognition by Using Neural Network to Encode and Decode Brain Imaging

  • Conference paper
  • First Online:
Human Brain and Artificial Intelligence (HBAI 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1072))

Included in the following conference series:

  • 654 Accesses

Abstract

The question of how the human brain represents conceptual knowledge has received significant attention in many scientific fields. Over the last decade, there has been increasing interest in the use of deep learning methods for analyzing functional magnetic resonance imaging (fMRI) data. In this paper, we report a series of experiments with neural networks for fMRI encoding and decoding. Results show that by using neural networks, both encoding and decoding accuracies are improved compared to a linear model on the same input. To evaluate the contextual information influences in cognitive modeling, we also extend the stimuli dataset from single noun to description sentences. The experiments indicate the impact of context information varies from person to person. To illustrate the strong correlation between linguistic and visual representations in the human brain, we extend the stimuli from a single word to images which were not present to the participant during fMRI data collection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Afifi, M.: 11k hands: gender recognition and biometric identification using a large dataset of hand images (2017)

    Google Scholar 

  2. Anderson, A.J., Kiela, D., Clark, S., Poesio, M.: Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns. Trans. Assoc. Comput. Linguist. 5, 17–30 (2017)

    Article  Google Scholar 

  3. Anderson, A.J., et al.: Predicting neural activity patterns associated with sentences using a neurobiologically motivated model of semantic representation. Cerebral Cortex 27(9), 4379–4395 (2017)

    Google Scholar 

  4. Anderson, A.J., Bruni, E., Lopopolo, A., Poesio, M., Baroni, M.: Reading visually embodied meaning from the brain: visually grounded computational models decode visual-object mental imagery induced by written text. NeuroImage 120, 309–322 (2015)

    Article  Google Scholar 

  5. Binder, J.R., Desai, R.H.: The neurobiology of semantic memory. Trends Cogn. Sci. 15(11), 527–536 (2011). 22001867[pmid]

    Article  Google Scholar 

  6. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)

    Article  Google Scholar 

  7. Brants, T., Franz, A., Linguistic Data Consortium: Web 1t 5-gram version 1 (2006)

    Google Scholar 

  8. ColourBox: Colourbox (2019)

    Google Scholar 

  9. Cox, D.D., Savoy, R.L.: Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage 19(2), 261–270 (2003)

    Article  Google Scholar 

  10. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-fei, L.: Imagenet: a large-scale hierarchical image database. In: CVPR (2009)

    Google Scholar 

  11. Devereux, B., Kelly, C., Korhonen, A.: Using fMRI activation to conceptual stimuli to evaluate methods for extracting conceptual representations from corpora. In: Proceedings of First Workshop On Computational Neurolinguistics, NAACL HLT pp. 70–78 (2010)

    Google Scholar 

  12. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  13. Fernandino, L., Humphries, C.J., Seidenberg, M., Gross, W., Conant, L., Binder, J.R.: Predicting brain activation patterns associated with individual lexical concepts based on five sensory-motor attributes. Neuropsychologia 76, 17–26 (2015)

    Article  Google Scholar 

  14. Handjaras, G., et al.: How concepts are encoded in the human brain: a modality independent, category-based cortical organization of semantic knowledge. NeuroImage 135, 232–242 (2016)

    Article  Google Scholar 

  15. Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P.: Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293(5539), 2425–2430 (2001)

    Article  Google Scholar 

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  17. Huth, A.G., de Heer, W.A., Griffiths, T.L., Theunissen, F.E., Gallant, J.L.: Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453 (2016)

    Article  Google Scholar 

  18. Images, G.: Google images (1998). https://images.google.com

  19. Ishai, A., Ungerleider, L.G., Martin, A., Schouten, J.L., Haxby, J.V.: Distributed representation of objects in the human ventral visual pathway. Proc. Natl. Acad. Sci. U.S.A. 96(16), 9379–9384 (1999). 10430951[pmid]

    Article  Google Scholar 

  20. Jelodar, A.B., Alizadeh, M., Khadivi, S.: Wordnet based features for predicting brain activity associated with meanings of nouns. In: Proceedings of the NAACL HLT 2010 First Workshop on Computational Neurolinguistics, CN 2010, pp. 18–26. Association for Computational Linguistics, Stroudsburg (2010)

    Google Scholar 

  21. Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification (2016)

    Google Scholar 

  22. Just, M.A., Cherkassky, V.L., Aryal, S., Mitchell, T.M.: A neurosemantic theory of concrete noun representation based on the underlying brain codes. PLoS ONE 5(1), e8622 (2010)

    Article  Google Scholar 

  23. Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, 25–29 October 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1746–1751 (2014)

    Google Scholar 

  24. Kiros, R., et al.: Skip-thought vectors. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28, pp. 3294–3302. Curran Associates, Inc. (2015). http://papers.nips.cc/paper/5950-skip-thought-vectors.pdf

  25. Kriegeskorte, N., Mur, M., Bandettini, P.: Representational similarity analysis - connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008)

    Article  Google Scholar 

  26. Louwerse, M., Hutchinson, S.: Neurological evidence linguistic processes precede perceptual simulation in conceptual processing. Front. Psychol. 3 (2012). https://doi.org/10.3389/fpsyg.2012.00385

  27. van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)

    MATH  Google Scholar 

  28. Matsuo, E., Kobayashi, I., Nishimoto, S., Nishida, S., Asoh, H.: Describing semantic representations of brain activity evoked by visual stimuli (2018)

    Google Scholar 

  29. Mikolov, T., Chen, K., Corrado, G.S., Dean, J.: Efficient estimation of word representations in vector space. CoRR abs/1301.3781 (2013)

    Google Scholar 

  30. Mitchell, T.M., et al.: Learning to decode cognitive states from brain images. Mach. Learn. 57(1–2), 145–175 (2004)

    Article  Google Scholar 

  31. Mitchell, T.M., et al.: Predicting human brain activity associated with the meanings of nouns. Science 320(5880), 1191–1195 (2008)

    Article  Google Scholar 

  32. Naselaris, T., Kay, K., Nishimoto, S., Gallant, J.: Encoding and decoding in fMRI. NeuroImage 56, 400–410 (2011)

    Article  Google Scholar 

  33. Norman, K.A., Polyn, S.M., Detre, G.J., Haxby, J.V.: Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10(9), 424–430 (2006)

    Article  Google Scholar 

  34. Palatucci, M., Pomerleau, D., Hinton, G.E., Mitchell, T.M.: Zero-shot learning with semantic output codes. In: Bengio, Y., Schuurmans, D., Lafferty, J.D., Williams, C.K.I., Culotta, A. (eds.) Advances in Neural Information Processing Systems 22, pp. 1410–1418. Curran Associates, Inc. (2009)

    Google Scholar 

  35. Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  36. Pereira, F., Botvinick, M., Detre, G.: Using Wikipedia to learn semantic feature representations of concrete concepts in neuroimaging experiments. Artif. Intell. 194, 240–252 (2013)

    Article  MathSciNet  Google Scholar 

  37. Pereira, F., Detre, G., Botvinick, M.: Generating text from functional brain images. Front. Hum. Neurosci. 5, 72 (2011)

    Article  Google Scholar 

  38. Pereira, F., Mitchell, T.M., Botvinick, M.: Machine learning classifiers and fMRI: a tutorial overview. NeuroImage 45(1 Suppl), S199–209 (2009)

    Article  Google Scholar 

  39. Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of NAACL (2018)

    Google Scholar 

  40. pinterest: pinterest (2019)

    Google Scholar 

  41. Poldrack, R.A., Mumford, J.A., Nichols, T.E.: Handbook of Functional MRI Data Analysis. Cambridge University Press, Cambridge (2011). https://doi.org/10.1017/CBO9780511895029

  42. Polyn, S.M., Natu, V.S., Cohen, J.D., Norman, K.A.: Category-specific cortical activity precedes retrieval during memory search. Science 310(5756), 1963–1966 (2005)

    Article  Google Scholar 

  43. Pulvermüller, F.: How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics. Trends Cogn. Sci. 17(9), 458–470 (2013)

    Article  Google Scholar 

  44. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533 (1986)

    Article  Google Scholar 

  45. Song, Y., Shi, S., Li, J., Zhang, H.: Directional skip-gram: explicitly distinguishing left and right context for word embeddings. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) (2018)

    Google Scholar 

  46. Wang, J., Cherkassky, V.L., Just, M.A.: Predicting the brain activation pattern associated with the propositional content of a sentence: modeling neural representations of events and states. Hum. Brain Mapping 38(10), 4865–4881 (2017)

    Article  Google Scholar 

  47. Wehbe, L., Murphy, B., Talukdar, P., Fyshe, A., Ramdas, A., Mitchell, T.: Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PLoS One 9(11), e112575 (2014). 25426840[pmid]

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Singapore University of Technology and Design, Singapore, Singapore

    Lu Cao

  2. West Lake University, Hangzhou, China

    Yue Zhang

Authors
  1. Lu Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yue Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yue Zhang .

Editor information

Editors and Affiliations

  1. Guangdong University of Technology, Guangzhou, China

    An Zeng

  2. Guangdong Construction Polytechnic, Guangzhou, China

    Dan Pan

  3. South China Normal University, Guangzhou, China

    Tianyong Hao

  4. Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Daoqiang Zhang

  5. University of Notre Dame, South Bend, IN, USA

    Yiyu Shi

  6. Simon Fraser University, Vancouver, BC, Canada

    Xiaowei Song

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cao, L., Zhang, Y. (2019). Investigating Lexical and Semantic Cognition by Using Neural Network to Encode and Decode Brain Imaging. In: Zeng, A., Pan, D., Hao, T., Zhang, D., Shi, Y., Song, X. (eds) Human Brain and Artificial Intelligence. HBAI 2019. Communications in Computer and Information Science, vol 1072. Springer, Singapore. https://doi.org/10.1007/978-981-15-1398-5_6

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-1398-5_6

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-1397-8

  • Online ISBN: 978-981-15-1398-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation