3D Convolutional Long-Short Term Memory Network for Spatiotemporal Modeling of fMRI Data

  • Wei Suo
  • Xintao HuEmail author
  • Bowei Yan
  • Mengyang Sun
  • Lei Guo
  • Junwei Han
  • Tianming Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11846)


Complex spatiotemporal correlation and dependency embedded in functional magnetic resonance imaging (fMRI) data introduce critical challenges in related analytical methodologies. Despite remarkable successes, most of existing approaches only model spatial or temporal dependency alone and the development of a unified spatiotemporal model is still a challenge. Meanwhile, the recent emergence of deep neural networks has provided powerful models for interpreting complex spatiotemporal data. Here, we proposed a novel convolutional long-short term memory network (3DCLN) for spatiotemporal modeling of fMRI data. The proposed model is designed to decode fMRI volumes belonging to different task events by joint training a 3D convolutional neural network (CNN) for spatial dependency modeling and a long short-term memory (LSTM) network for temporal dependency modeling. We also designed a 3D deconvolution scheme for fMRI sequence reconstruction to inspect the feature learning process in the 3DCLN. The experimental results on the motor task-fMRI data from Human Connectome Project (HCP) showed that fMRI volumes can be decoded with a relatively high accuracy (76.38%). More importantly, the proposed 3DCLN can dramatically remove noises and highlights signals of interest in the reconstructed fMRI sequence and hence improve the performance of activation detection, validating the spatiotemporal feature learning in the proposed 3DCLN model.


Functional magnetic resonance imaging 3D convolutional neural network Long short-term memory network Spatiotemporal modeling 


  1. 1.
    Logothetis, N.K.: What we can do and what we cannot do with fMRI. Nature 453(7197), 869–878 (2008)CrossRefGoogle Scholar
  2. 2.
    Friston, K.J.: Transients, metastability, and neuronal dynamics. NeuroImage 5, 164–171 (1997)CrossRefGoogle Scholar
  3. 3.
    Hjelm, R.D., Calhoun, V.D., Salakhutdinov, R., Allen, E.A., Adali, T., Plis, S.M.: Restricted Boltzmann machines for neuroimaging: an application in identifying intrinsic networks. NeuroImage 96(8), 245–260 (2014)CrossRefGoogle Scholar
  4. 4.
    Hu, X., et al.: Latent source mining in FMRI via restricted Boltzmann machine. Hum. Brain Mapp. 39(6), 2368–2380 (2018)CrossRefGoogle Scholar
  5. 5.
    Xia, W., Wen, X., Li, J., Li, Y.: A new dynamic Bayesian network approach for determining effective connectivity from fMRI data. Neural Comput. Appl. 24(1), 91–97 (2014)CrossRefGoogle Scholar
  6. 6.
    Graves, A.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  7. 7.
    Wang, H., et al.: Recognizing brain states using deep sparse recurrent neural network. IEEE Trans. Med. Imag. 34, 1058–1068 (2018)Google Scholar
  8. 8.
    Huang, S., Li, X., Zhang, Z., Wu, F., Han, J.: User-ranking video summarization with multi-stage spatio-temporal representation. IEEE Trans. Image Process. PP(99), 1 (2018)Google Scholar
  9. 9.
    Barch, D.M., et al.: Function in the human connectome: task-fMRI and individual differences in behavior. NeuroImage 80(8), 169–189 (2013)CrossRefGoogle Scholar
  10. 10.
    Glasser, M.F., Sotiropoulos, S.N., Wilson, J.A., et al.: The minimal preprocessing pipelines for the Human Connectome Project. NeuroImage 80, 105–124 (2013)CrossRefGoogle Scholar
  11. 11.
    Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)
  12. 12.
    Zeiler, Matthew D., Fergus, R.: Visualizing and Understanding Convolutional Networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). Scholar
  13. 13.
    Huang, H., Hu, X., Zhao, Y., et al.: Modeling task fMRI data via deep convolutional autoencoder. IEEE Trans. Med. Imag. 37(7), 1551–1561 (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Wei Suo
    • 1
  • Xintao Hu
    • 1
    Email author
  • Bowei Yan
    • 1
  • Mengyang Sun
    • 2
  • Lei Guo
    • 1
  • Junwei Han
    • 1
  • Tianming Liu
    • 3
  1. 1.School of AutomationNorthwestern Polytechnical UniversityXi’anChina
  2. 2.School of Computer ScienceNorthwestern Polytechnical UniversityXi’anChina
  3. 3.Department of Computer Science and Bioimaging Research CenterThe University of GeorgiaAthensUSA

Personalised recommendations