Learning Human Cognition via fMRI Analysis Using 3D CNN and Graph Neural Network

  • Xiuyan NiEmail author
  • Tian Gao
  • Tingting Wu
  • Jin Fan
  • Chao Chen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11846)


Human cognitive control involves how mental resources are allocated when the brain processes various information. The study of such complex brain functionality is essential in understanding different neurological disorders. To investigate cognition control, various cognitive tasks have been designed and functional MRI data have been collected. In this paper, we study uncertainty representation, an important problem in human cognition study, with task-evoked fMRI data. Our goals are to learn how brain region of interests (ROIs) are activated under tasks with different uncertainty levels and how they interact with each other. We propose a novel neural network architecture to achieve the two goals simultaneously. Our architecture uses a 3D convolutional neural network (CNN) to extract a high-level representation for each ROI, and uses a graph neural network module to capture the interactions between ROIs. Empirical evaluations reveal that our method significantly outperforms the existing methods, and the derived brain network is consistent with domain knowledge.


Graph neural network Brain network learning 



This work was partially supported by NSF IIS-1855759 and CCF-1855760.


  1. 1.
    Bassett, D.S., Yang, M., Wymbs, N.F., Grafton, S.T.: Learning-induced autonomy of sensorimotor systems. Nat. Neurosci. 18(5), 744 (2015)CrossRefGoogle Scholar
  2. 2.
    Belilovsky, E., Kastner, K., Varoquaux, G., Blaschko, M.B.: Learning to discover sparse graphical models. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 440–448. JMLR. org (2017)Google Scholar
  3. 3.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)zbMATHGoogle Scholar
  4. 4.
    Castellanos, F.X., Sonuga-Barke, E.J., Milham, M.P., Tannock, R.: Characterizing cognition in ADHD: beyond executive dysfunction. Trends Cognit. Sci. 10(3), 117–123 (2006)CrossRefGoogle Scholar
  5. 5.
    Chen, P.H., et al.: A convolutional autoencoder for multi-subject FMRI data aggregation. arXiv preprint (2016). arXiv:1608.04846
  6. 6.
    Cole, M.W., Bassett, D.S., Power, J.D., Braver, T.S., Petersen, S.E.: Intrinsic and task-evoked network architectures of the human brain. Neuron 83(1), 238–251 (2014)CrossRefGoogle Scholar
  7. 7.
    Cole, M.W., Schneider, W.: The cognitive control network: integrated cortical regions with dissociable functions. Neuroimage 37(1), 343–360 (2007)CrossRefGoogle Scholar
  8. 8.
    Diamond, A., Barnett, W.S., Thomas, J., Munro, S.: Preschool program improves cognitive control. Science 318(5855), 1387 (2007)CrossRefGoogle Scholar
  9. 9.
    Elton, A., Gao, W.: Task-positive functional connectivity of the default mode network transcends task domain. J. Cognit. Neurosci. 27(12), 2369–2381 (2015)CrossRefGoogle Scholar
  10. 10.
    Hamilton, W.L., Ying, R., Leskovec, J.: Representation learning on graphs: methods and applications. IEEE Data Eng. Bull. 40(3), 52–74 (2017)Google Scholar
  11. 11.
    Hellyer, P.J., Shanahan, M., Scott, G., Wise, R.J., Sharp, D.J., Leech, R.: The control of global brain dynamics: opposing actions of frontoparietal control and default mode networks on attention. J. Neurosci. 34(2), 451–461 (2014)CrossRefGoogle Scholar
  12. 12.
    Kelly, A.C., Uddin, L.Q., Biswal, B.B., Castellanos, F.X., Milham, M.P.: Competition between functional brain networks mediates behavioral variability. Neuroimage 39(1), 527–537 (2008)CrossRefGoogle Scholar
  13. 13.
    Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint (2016). arXiv:1609.02907
  14. 14.
    Leech, R., Kamourieh, S., Beckmann, C.F., Sharp, D.J.: Fractionating the default mode network: distinct contributions of the ventral and dorsal posterior cingulate cortex to cognitive control. J. Neurosci. 31(9), 3217–3224 (2011)CrossRefGoogle Scholar
  15. 15.
    Ni, X., Yan, Z., Wu, T., Fan, J., Chen, C.: A region-of-interest-reweight 3D convolutional neural network for the analytics of brain information processing. In: Medical Image Computing and Computer Assisted Intervention - MICCAI, pp. 302–310 (2018)Google Scholar
  16. 16.
    Nie, D., Zhang, H., Adeli, E., Liu, L., Shen, D.: 3D deep learning for multi-modal imaging-guided survival time prediction of brain tumor patients. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 212–220. Springer, Cham (2016). Scholar
  17. 17.
    Nielsen, A.N., Greene, D.J., Gratton, C., Dosenbach, N.U., Petersen, S.E., Schlaggar, B.L.: Evaluating the prediction of brain maturity from functional connectivity after motion artifact denoising. Cereb. Cortex 29(6), 2455–2469 (2018)CrossRefGoogle Scholar
  18. 18.
    Power, J.D., Schlaggar, B.L., Lessov-Schlaggar, C.N., Petersen, S.E.: Evidence for hubs in human functional brain networks. Neuron 79(4), 798–813 (2013)CrossRefGoogle Scholar
  19. 19.
    Raichle, M.E., MacLeod, A.M., Snyder, A.Z., Powers, W.J., Gusnard, D.A., Shulman, G.L.: A default mode of brain function. Proc. National Acad. Sci. 98(2), 676–682 (2001)CrossRefGoogle Scholar
  20. 20.
    Ridderinkhof, K.R., Ullsperger, M., Crone, E.A., Nieuwenhuis, S.: The role of the medial frontal cortex in cognitive control. Science 306(5695), 443–447 (2004)CrossRefGoogle Scholar
  21. 21.
    Schmidt, M., Murphy, K., Fung, G., Rosales, R.: Structure learning in random fields for heart motion abnormality detection. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)Google Scholar
  22. 22.
    Shannon, C.E., Weaver, W.: The Mathematical Theory of Communication. University of Illinois Press, Champaign (1949)zbMATHGoogle Scholar
  23. 23.
    Solomon, M., Ozonoff, S.J., Ursu, S., Ravizza, S., Cummings, N., Ly, S., Carter, C.S.: The neural substrates of cognitive control deficits in autism spectrum disorders. Neuropsychologia 47(12), 2515–2526 (2009)CrossRefGoogle Scholar
  24. 24.
    Spreng, R.N., Sepulcre, J., Turner, G.R., Stevens, W.D., Schacter, D.L.: Intrinsic architecture underlying the relations among the default, dorsal attention, and frontoparietal control networks of the human brain. J. Cognit. Neurosci. 25(1), 74–86 (2013)CrossRefGoogle Scholar
  25. 25.
    Tan, M., Shi, Q., van den Hengel, A., Shen, C., Gao, J., Hu, F., Zhang, Z.: Learning graph structure for multi-label image classification via clique generation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4100–4109 (2015)Google Scholar
  26. 26.
    Van Den Heuvel, M.P., Mandl, R.C., Kahn, R.S., Hulshoff Pol, H.E.: Functionally linked resting-state networks reflect the underlying structural connectivity architecture of the human brain. Hum. Brain Mapp. 30(10), 3127–3141 (2009)CrossRefGoogle Scholar
  27. 27.
    Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. In: International Conference on Learning Representations (2018)Google Scholar
  28. 28.
    Wu, T., et al.: Hick-hyman law is mediated by the cognitive control network in the brain. Cereb. Cortex 28(7), 1–16 (2017)Google Scholar
  29. 29.
    Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? In: International Conference on Learning Representations (2019)Google Scholar
  30. 30.
    Zhang, X., He, L., Chen, K., Luo, Y., Zhou, J., Wang, F.: Multi-view graph convolutional network and its applications on neuroimage analysis for parkinson’s disease. arXiv preprint (2018). arXiv:1805.08801

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Xiuyan Ni
    • 1
    Email author
  • Tian Gao
    • 2
  • Tingting Wu
    • 3
  • Jin Fan
    • 3
  • Chao Chen
    • 1
    • 4
  1. 1.Department of Computer Science, The Graduate CenterCity University of New York (CUNY)New YorkUSA
  2. 2.IBM Thomas J. Watson Research CenterYorktown HeightsUSA
  3. 3.Department of PsychologyCUNY Queens CollegeFlushingUSA
  4. 4.Department of Biomedical InformaticsStony Brook UniversityStony BrookUSA

Personalised recommendations