A Hybrid 3DCNN and 3DC-LSTM Based Model for 4D Spatio-Temporal fMRI Data: An ABIDE Autism Classification Study

  • Ahmed El-GazzarEmail author
  • Mirjam Quaak
  • Leonardo Cerliani
  • Peter Bloem
  • Guido van Wingen
  • Rajat Mani Thomas
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11796)


Functional Magnetic Resonance Imaging (fMRI) captures the temporal dynamics of neural activity as a function of spatial location in the brain. Thus, fMRI scans are represented as 4-Dimensional (3-space + 1-time) tensors. And it is widely believed that the spatio-temporal patterns in fMRI manifests as behaviour and clinical symptoms. Because of the high dimensionality (\(\sim \)1 Million) of fMRI, and the added constraints of limited cardinality of data sets, extracting such patterns are challenging. A standard approach to overcome these hurdles is to reduce the dimensionality of the data by either summarizing activation over time or space at the expense of possible loss of useful information. Here, we introduce an end-to-end algorithm capable of extracting spatiotemporal features from the full 4-D data using 3-D CNNs and 3-D Convolutional LSTMs. We evaluate our proposed model on the publicly available ABIDE dataset to demonstrate the capability of our model to classify Autism Spectrum Disorder (ASD) from resting-state fMRI data. Our results show that the proposed model achieves state of the art results on single sites with F1-scores of 0.78 and 0.7 on NYU and UM sites, respectively.


Deep learning ASD 3D convolutions 3D convolutional-LSTM rs-fMRI 



This work was supported by the Netherlands Organization for Scientific Research (NWO; 628.011.023), Philips Research, AAA Data Science Program, and ZonMW (Vidi; 016.156.318).


  1. 1.
    Abraham, A., et al.: Deriving reproducible biomarkers from multi-site resting-state data: an autism-based example. NeuroImage 147, 736–745 (2017)CrossRefGoogle Scholar
  2. 2.
    Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)
  3. 3.
    Bengs, M., Gessert, N., Schlaefer, A.: 4D spatio-temporal deep learning with 4D fMRI data for autism spectrum disorder classification (2019). Extended Abstract MIDL
  4. 4.
    Craddock, R.C., James, G.A., Holtzheimer III, P.E., Hu, X.P., Mayberg, H.S.: A whole brain fMRI atlas generated via spatially constrained spectral clustering. Hum. Brain Mapp. 33(8), 1914–1928 (2012)CrossRefGoogle Scholar
  5. 5.
    Cristianini, N., Shawe-Taylor, J., et al.: An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, Cambridge (2000)CrossRefGoogle Scholar
  6. 6.
    Dvornek, N.C., Ventola, P., Pelphrey, K.A., Duncan, J.S.: Identifying autism from resting-state fMRI using long short-term memory networks. In: Wang, Q., Shi, Y., Suk, H.-I., Suzuki, K. (eds.) MLMI 2017. LNCS, vol. 10541, pp. 362–370. Springer, Cham (2017). Scholar
  7. 7.
    El Gazzar, A., Cerliani, L., van Wingen, G., Mani Thomas, R.: Simple 1-D convolutional networks for resting-state fMRI based classification in autism, July 2019Google Scholar
  8. 8.
    Heinsfeld, A.S., Franco, A.R., Craddock, R.C., Buchweitz, A., Meneguzzi, F.: Identification of autism spectrum disorder using deep learning and the ABIDE dataset. NeuroImage: Clin. 17, 16–23 (2018)CrossRefGoogle Scholar
  9. 9.
    Khosla, M., Jamison, K., Kuceyeski, A., Sabuncu, M.: 3D convolutional neural networks for classification of functional connectomes. arXiv preprint arXiv:1806.04209 (2018)
  10. 10.
    Li, X., et al.: 2-channel convolutional 3D deep neural network (2CC3D) for fMRI analysis: ASD classification and feature learning. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1252–1255. IEEE (2018)Google Scholar
  11. 11.
    Vieira, S., Pinaya, W.H., Mechelli, A.: Using deep learning to investigate the neuroimaging correlates of psychiatric and neurological disorders: methods and applications. Neurosci. Biobehav. Rev. 74, 58–75 (2017)CrossRefGoogle Scholar
  12. 12.
    Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.C.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in Neural Information Processing Systems, pp. 802–810 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Ahmed El-Gazzar
    • 1
    Email author
  • Mirjam Quaak
    • 1
    • 2
  • Leonardo Cerliani
    • 1
  • Peter Bloem
    • 2
  • Guido van Wingen
    • 1
  • Rajat Mani Thomas
    • 1
  1. 1.Department of PsychiatryAmsterdam UMC, University of AmsterdamAmsterdamThe Netherlands
  2. 2.Vrije Universiteit AmsterdamAmsterdamThe Netherlands

Personalised recommendations