Skip to main content

Advertisement

SpringerLink
Log in

Anatomical Pattern Analysis for Decoding Visual Stimuli in Human Brains

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

A universal unanswered question in neuroscience and machine learning is whether computers can decode the patterns of the human brain. Multi-Voxel Pattern Analysis (MVPA) is a critical tool for addressing this question. However, there are two challenges in the previous MVPA methods, which include decreasing sparsity and noise in the extracted features and increasing the performance of prediction. In overcoming mentioned challenges, this paper proposes Anatomical Pattern Analysis (APA) for decoding visual stimuli in the human brain. This framework develops a novel anatomical feature extraction method and a new imbalance AdaBoost algorithm for binary classification. Further, it utilizes an Error-Correcting Output Codes (ECOC) method for multiclass prediction. APA can automatically detect active regions for each category of the visual stimuli. Moreover, it enables us to combine homogeneous datasets for applying advanced classification. Experimental studies on four visual categories (words, consonants, objects, and scrambled photos) demonstrate that the proposed approach achieves superior performance to state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Anderson M, Oates T. A critique of multi-voxel pattern analysis. In: Proceedings of the cognitive science society; 2010.

  2. Carlson TA, Schrater P, He S. Patterns of activity in the categorical representations of objects. J Cogn Neurosci 2003;15(5):704–717.

    Article  PubMed  Google Scholar 

  3. Carroll MK, Cecchi GA, Rish I, Garg R, Rao AR. Prediction and interpretation of distributed neural activity with sparse models. NeuroImage 2009;44(1):112–122.

    Article  PubMed  Google Scholar 

  4. Chen PH, Chen J, Yeshurun Y, Hasson U, Haxby J, Ramadge PJ. A reduced-dimension fMRI shared response model. In: 28Th advances in neural information processing systems (NIPS-15), p. 460–468. Advances in neural information processing systems (NIPS), december/7–12, Montral; 2015.

  5. Chen PH, Zhu X, Zhang H, Turek JS, Chen J, Willke TL, Hasson U, Ramadge PJ. A convolutional autoencoder for multi-subject fmri data aggregation. In: 29th workshop of representation learning in artificial and biological neural networks. Advances in neural information processing systems (NIPS), december/5–10, barcelona; 2016.

  6. Cohen L, Dehaene S, Naccache L, Lehéricy S, Dehaene-Lambertz G, Hénaff MA, Michel F. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain 2000;123(2):291–307.

    Article  PubMed  Google Scholar 

  7. Connolly A, Gobbini M, Haxby J. 2012. Three virtues of similarity-based multi-voxel pattern analysis.

  8. Connolly AC, Guntupalli JS, Gors J, Hanke M, Halchenko YO, Wu YC, Abdi H, Haxby JV. The representation of biological classes in the human brain. J Neurosci 2012;32(8):2608–2618.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Cox DD, Savoy RL. Functional magnetic resonance imaging (fMRI) brain reading: detecting and classifying distributed patterns of fmri activity in human visual cortex. NeuroImage 2003;19(2):261–270.

    Article  PubMed  Google Scholar 

  10. Duncan KJ, Pattamadilok C, Knierim I, Devlin JT. Consistency and variability in functional localisers. NeuroImage 2009;46(4):1018–1026.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Escalera S, Pujol O, Radeva P. Error-correcting output codes library. J Mach Learn Res 2010;11(Feb): 661–664.

    Google Scholar 

  12. Friston KJ. Statistical parametric mapping. In: Neuroscience databases. Berlin: Springer; 2003. p. 237–250.

  13. Haxby JV, Connolly AC, Guntupalli JS. Decoding neural representational spaces using multivariate pattern analysis. Ann Rev Neurosci 2014;37:435–456.

    Article  CAS  PubMed  Google Scholar 

  14. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 2001;293(5539):2425–2430.

    Article  CAS  PubMed  Google Scholar 

  15. Haxby JV, Guntupalli JS, Connolly AC, Halchenko YO, Conroy BR, Gobbini MI, Hanke M, Ramadge PJ. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron 2011;72(2):404–416.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Haynes JD, Rees G. Decoding mental states from brain activity in humans. Nat Rev Neurosci 2006;7(7): 523.

    Article  CAS  PubMed  Google Scholar 

  17. Haynes JD, Sakai K, Rees G, Gilbert S, Frith C, Passingham RE. Reading hidden intentions in the human brain. Curr Biol 2007;17(4):323–328.

    Article  CAS  PubMed  Google Scholar 

  18. Jenkinson M, Bannister P, Brady M, Smith S. Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage 2002;17(2):825–841.

    Article  PubMed  Google Scholar 

  19. Kamitani Y, Tong F. Decoding the visual and subjective contents of the human brain. Nat Neurosci 2005;8 (5):679–685.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Kanwisher N, McDermott J, Chun MM. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J Neurosci 1997;17(11):4302–4311.

    Article  CAS  PubMed  Google Scholar 

  21. Kay KN, Naselaris T, Prenger RJ, Gallant JL. Identifying natural images from human brain activity. Nature 2008;452(7185):352.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Kriegeskorte N, Mur M, Bandettini P. Representational similarity analysis–connecting the branches of systems neuroscience. Front Syst Neurosci 2008;2:1–28.

    Google Scholar 

  23. Kriegeskorte N, Simmons WK, Bellgowan PS, Baker CI. Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci 2009;12(5):535–540.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Liesegang TJ. A cortical area selective for visual processing of the human body. downing pe, 1 school of psychology, centre for cognitive neuroscience, University of Wales, bangor, ll57 2as, United Kingdom. e-mail: p. downing@ bangor. ac. uk jiang y, shuman m, kanwisher n. science 2001; 293: 2470–2473. Am J Ophthalmol 2002;133(4):598.

    Google Scholar 

  25. Liu XY, Wu J, Zhou ZH. Exploratory undersampling for class-imbalance learning. IEEE Trans Syst Man Cybern B Cybern 2009;39(2):539–550.

    Article  PubMed  Google Scholar 

  26. Malach R, Reppas J, Benson R, Kwong K, Jiang H, Kennedy W, Ledden P, Brady T, Rosen B, Tootell R. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Sciences (PNAS) 1995;92(18):8135–8139.

    Article  CAS  Google Scholar 

  27. McMenamin BW, Deason RG, Steele VR, Koutstaal W, Marsolek CJ. Separability of abstract-category and specific-exemplar visual object subsystems: Evidence from fMRI pattern analysis. Brain Cogn 2015;93:54–63.

    Article  PubMed  Google Scholar 

  28. Mitchell TM, Shinkareva SV, Carlson A, Chang KM, Malave VL, Mason RA, Just MA. Predicting human brain activity associated with the meanings of nouns. Science 2008;320(5880):1191–1195.

    Article  CAS  PubMed  Google Scholar 

  29. Miyawaki Y, Uchida H, Yamashita O, Sato Ma, Morito Y, Tanabe HC, Sadato N, Kamitani Y. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 2008;60(5):915–929.

    Article  CAS  PubMed  Google Scholar 

  30. Mohr H, Wolfensteller U, Frimmel S, Ruge H. Sparse regularization techniques provide novel insights into outcome integration processes. NeuroImage 2015;104:163–176.

    Article  PubMed  Google Scholar 

  31. Norman KA, Polyn SM, Detre GJ, Haxby JV. Beyond mind-reading: multi-voxel pattern analysis of fmri data. Trends Cogn Sci 2006;10(9):424–430.

    Article  PubMed  Google Scholar 

  32. Osher DE, Saxe RR, Koldewyn K, Gabrieli JD, Kanwisher N, Saygin ZM. Structural connectivity fingerprints predict cortical selectivity for multiple visual categories across cortex. Cereb Cortex 2015;26 (4):1668–1683.

    Article  PubMed  PubMed Central  Google Scholar 

  33. O’toole AJ, Jiang F, Abdi H, Haxby JV. Partially distributed representations of objects and faces in ventral temporal cortex. J Cogn Neurosci 2005;17(4):580–590.

    Article  PubMed  Google Scholar 

  34. Rice GE, Watson DM, Hartley T, Andrews TJ. Low-level image properties of visual objects predict patterns of neural response across category-selective regions of the ventral visual pathway. J Neurosci 2014;34(26): 8837–8844.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Varoquaux G, Gramfort A, Thirion B. Small-sample brain mapping: sparse recovery on spatially correlated designs with randomization and clustering. In: Proceedings of the 29th international conference on machine learning (ICML-12); 2012. p. 1375–1382.

  36. Wakeman DG, Henson RN. A multi-subject, multi-modal human neuroimaging dataset. Scientific Data 2015; 2:1–10.

    Article  Google Scholar 

  37. Xu J, Potenza MN, Calhoun VD. Spatial ICA reveals functional activity hidden from traditional fMRI GLM-based analyses. Front Neurosci 2013;7:1–4.

    Article  Google Scholar 

  38. Yamashita O, Sato Ma, Yoshioka T, Tong F, Kamitani Y. Sparse estimation automatically selects voxels relevant for the decoding of fmri activity patterns. NeuroImage 2008;42(4):1414– 1429.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Yousefnezhad M, Zhang D. Decoding visual stimuli in human brain by using anatomical pattern analysis on fMRI images. In: 8Th international conference on brain inspired cognitive systems (BICS’16), p. 47–57. Springer, November/28–30, Beijing; 2016.

  40. Yousefnezhad M, Zhang D. Local discriminant hyperalignment for multi-subject fmri data alignment. In: 34Th AAAI conference on artificial intelligence (AAAI-17), pp. 59–65. Association for the advancement of artificial intelligence (AAAI), february/4–9, san francisco; 2017.

  41. Yousefnezhad M, Zhang D. Multi-region neural representation: a novel model for decoding visual stimuli in human brains. In: 17Th SIAM international conference on data mininig (SDM-17), pp. 54–62. Society for industrial and applied mathematics (SIAM), april/27–29, houston; 2017.

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (61422204 and 61473149) and NUAA Fundamental Research Funds (NE2013105).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Muhammad Yousefnezhad & Daoqiang Zhang

Authors
  1. Muhammad Yousefnezhad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daoqiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Yousefnezhad.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yousefnezhad, M., Zhang, D. Anatomical Pattern Analysis for Decoding Visual Stimuli in Human Brains. Cogn Comput 10, 284–295 (2018). https://doi.org/10.1007/s12559-017-9518-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-017-9518-9

Keywords

Search

Navigation