Deep Learning on Brain Images in Autism: What Do Large Samples Reveal of Its Complexity?

  • Matthew LemingEmail author
  • John Suckling
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11486)


Deep learning models for image classification face two recurring problems: they are typically limited by low sample size and are abstracted by their own complexity (the “black box problem”). We address these problems with the largest functional MRI connectome dataset ever compiled, classifying it across gender and Task vs rest (no task) to ascertain its performance, and then apply the model to a cross-sectional comparison of autism vs typically developing (TD) controls that has proved difficult to characterise with inferential statistics. Employing class-balancing to build a training set, a convolutional neural network was classified fMRI connectivity with overall accuracies of 76.35% (AUROC 0.8401), 90.71% (AUROC 0.9573), and 67.65% (AUROC 0.7162) for gender, task vs rest, and autism vs TD, respectively. Salience maps demonstrated that the deep learning model is capable of distinguishing complex patterns across either wide networks or localized areas of the brain, and, by analyzing maximal activations of the hidden layers, that the deep learning model partitions data at an early stage in its classification.


Autism Big data Functional connectivity Deep learning 



This study used publicly available datasets, each with their own acknowledgements. For brevity, we have not included the full text, but recognise the contributions of the Alzheimer’s Disease Neuroimaging Initiative, International Consortium for Brain Mapping, National Database for Autism Research, NIH Pediatric MRI Data Repository, National Database for Clinical Trials, Research Domain Criteria Database, Adolescent Brain Cognitive Development Study, UK Biobank Resource, 1000 Functional Connectomes Project, ABIDE I and II, and Open fMRI. This research was co-funded by the NIHR Cambridge Biomedical Research Centre and Marmaduke Sheild. ML is supported by a Gates Cambridge Scholarship from the University of Cambridge.


  1. 1.
    Arbabshirani, M., Havlicek, M., Kiehl, K., Pearlson, G., Calhoun, V.: Functional network connectivity during rest and task conditions: a comparative study. Hum. Brain Mapp. 34, 2959–2971 (2012). Scholar
  2. 2.
    Blanken, L., et al.: A prospective study of fetal head growth, autistic traits and autism spectrum disorder. Autism Res. 11, 602–612 (2018). Scholar
  3. 3.
    Brown, C., Kawahara, J., Hamarneh, G.: Connectome priors in deep neural networks to predict autism. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) (2018).
  4. 4.
    Bruna, J., Zaremba, W., Szlam, A., LeCun, Y.: Spectral networks and locally connected networks on graphs. In: ICLR (2014)Google Scholar
  5. 5.
    Casey, B., Dale, A.: The adolescent brain cognitive development (ABCD) study: imaging acquisition across 21 sites. Dev. Cogn. Neurosci. 32, 43–54 (2018). Scholar
  6. 6.
    Cauda, F., et al.: Grey matter abnormality in autism spectrum disorder: an activation likelihood estimation meta-analysis study. J. Neurol. Neurosurg. Psychiatry 82, 1304–1313 (2011). Scholar
  7. 7.
    Chollet, F.: Keras (2015).
  8. 8.
    Courchesne, E., Carper, R., Akshoomoff, N.: Evidence of brain overgrowth in the first year of life in autism. JAMA 290, 337–344 (2003). Scholar
  9. 9.
    Defferrard, M., Bresson, P., Vandergheynst, X.: Convolutional neural networks on graphs with fast localized spectral filtering. In: NIPS, pp. 3844–3852 (2016)Google Scholar
  10. 10.
    DeRamus, T., Kana, R.: Anatomical likelihood estimation meta-analysis of grey and white matter anomalies in autism spectrum disorders author links open overlay panel. NeuroImage: Clin. 7, 525–536 (2015). Scholar
  11. 11.
    Di Martino, A., et al.: Enhancing studies of the connectome in autism using the autism brain imaging data exchange II. Sci. Data 4, 170010 (2017). Scholar
  12. 12.
    Di Martino, A., et al.: The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Mol. Psychiatry 19, 659–67 (2014). Scholar
  13. 13.
    Dinstein, I., Haar, S., Atsmon, S., Schtaerman, H.: No evidence of early head circumference enlargements in children later diagnosed with autism in Israel. Mol. Autism 8 (2018).
  14. 14.
    Dolgin, E.: This is your brain online: the functional connectomes project. Nat. Med. 16, 351 (2010). Scholar
  15. 15.
    Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. Technical report 1341, University of Montreal (2009)Google Scholar
  16. 16.
    Finn, E., et al.: Functional connectome fingerprinting: identifying individuals using patterns of brain connectivity. Nat Neurosci. 18, 1664–1671 (2015). Scholar
  17. 17.
    Haar, S., Berman, S., Behrmann, M., Dinstein, I.: Anatomical abnormalities in autism? Cereb. Cortex 26, 1440–1452 (2016). Scholar
  18. 18.
    Hall, D., Huerta, M., McAuliffe, M., Farber, G.: Sharing heterogeneous data: the national database for autism research. Neuroinformatics 10, 331–339 (2012). Scholar
  19. 19.
    Hamilton, W., Ying, R., Leskovec, J.: Representation learning on graphs: methods and applications. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering (2017)Google Scholar
  20. 20.
    Hasson, U., Nusbaum, H., Small, S.: Task-dependent organization of brain regions active during rest. PNAS 106, 10841–10846 (2009). Scholar
  21. 21.
    Hechtlinger, Y., Chakravarti, P., Qin, J.: A generalization of convolutional neural networks to graph-structured data. arXiv (2017)Google Scholar
  22. 22.
    Heinsfeld, A., Franco, A., Craddock, R., Buchweitz, A., Meneguzzia, F.: Identification of autism spectrum disorder using deep learning and the abide dataset. NeuroImage: Clin. 17, 16–23 (2018). Scholar
  23. 23.
    Kawahara, J., et al.: BrainNetCNN: convolutional neural networks for brain networks; towards predicting neurodevelopment. NeuroImage 146, 1038–1049 (2017). Scholar
  24. 24.
    Keown, C., Datko, M., Chen, C., Maximo, J., Jahedi, A., Müller, R.: Network organization is globally atypical in autism: a graph theory study of intrinsic functional connectivity. Biol. Psychiatry: Cogn. Neurosci. Neuroimaging 2, 66–75 (2017). Scholar
  25. 25.
    Khundrakpam, B., Lewis, J., Kostopoulos, P., Carbonell, F., Evans, A.: Cortical thickness abnormalities in autism spectrum disorders through late childhood, adolescence, and adulthood: a large-scale MRI study. Cereb. Cortex 27, 1721–1731 (2017). Scholar
  26. 26.
    Kipf, T., Welling, M.: Semi-supervised classification with graph convolutional neural networks. In: ICLR 2017 (2017)Google Scholar
  27. 27.
    Kotikalapudi, R., Contributors: keras-vis (2017).
  28. 28.
    Khosla, M., Jamison, K., Kuceyeski, A., Sabuncu, M.R.: 3D convolutional neural networks for classification of functional connectomes. In: Stoyanov, D., et al. (eds.) DLMIA/ML-CDS-2018. LNCS, vol. 11045, pp. 137–145. Springer, Cham (2018). Scholar
  29. 29.
    Nikolentzos, G., Meladianos, P., Tixier, A.J.-P., Skianis, K., Vazirgiannis, M.: Kernel graph convolutional neural networks. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11139, pp. 22–32. Springer, Cham (2018). Scholar
  30. 30.
    Patel, A., Bullmore, E.: A wavelet-based estimator of the degrees of freedom in denoised fmri time series for probabilistic testing of functional connectivity and brain graphs. NeuroImage 142, 14–26 (2016). Scholar
  31. 31.
    Piven, J., Arndt, S., Bailey, J., Havercamp, S., Andreasen, N., Palmer, P.: An MRI study of brain size in autism. Am. J. Psychiatry 152, 1145–1149 (1995). Scholar
  32. 32.
    Plitt, M., Barnes, K., Martin, A.: Functional connectivity classification of autism identifies highly predictive brain features but falls short of biomarker standards. NeuroImage Clin. 7, 359–66 (2015). Scholar
  33. 33.
    Poldrack, R., et al.: Toward open sharing of task-based fMRI data: the OpenfMRI project. Front. Neuroinform. 7 (2013).
  34. 34.
    Poldrack, R., Gorgolewski, K.: OpenfMRI: open sharing of task fMRI data. NeuroImage 144, 259–261 (2017). Scholar
  35. 35.
    Redcay, E., Courchesne, E.: Biol. Psychiatry 58, 1–9 (2005). Scholar
  36. 36.
    Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015). Scholar
  37. 37.
    Satterthwaite, T., et al.: Linked sex differences in cognition and functional connectivity in youth. Cereb. Cortex 25, 2383–2394 (2015). Scholar
  38. 38.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. In: Workshop at International Conference on Learning Representations (2014)Google Scholar
  39. 39.
    Tomasi, D., Volkow, N.: Gender differences in brain functional connectivity density. Hum. Brain Mapp. 33, 849–860 (2013). Scholar
  40. 40.
    Wang, W., et al.: Altered resting-state functional activity in patients with autism spectrum disorder: a quantitative meta-analysis. Front. Neurol. 9, 556 (2018). Scholar
  41. 41.
    Yang, J., Hofmann, J.: Action observation and imitation in autism spectrum disorders: an ALE meta-analysis of fMRI studies. Brain Imaging Behav. 10, 960–969 (2016). Scholar
  42. 42.
    Zhang, W., Groen, W., Mennes, M., Greven, C., Buitelaar, J., Rommelse, N.: Revisiting subcortical brain volume correlates of autism in the abide dataset: effects of age and sex. Psychol. Med. 48, 654–668 (2018). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of PsychiatryCambridge UniversityCambridgeUK

Personalised recommendations