Abstract
Integrating information from multiple biological layers is a key approach to unraveling the complexities of the human brain, with its multiple overlapping structural and functional subsystems operating at widely different temporal and spatial scales. Moreover, it has true potential to positively impact mental health patients through early diagnosis and individualized treatment. This chapter lays out a succession of approaches to synergistic fusion of multimodal brain imaging data, with a special focus on blind source separation (BSS) and deep learning (DL) methods. Firstly, a broad unified description of the BSS field is introduced, serving as a theoretical backbone for the chapter. Complementary to that, a detailed case study of three different applications of joint independent component analysis (jICA) provides both a reference guide on data fusion and a bridge into more advanced BSS methods. Various advanced BSS methods such as multiset canonical correlation analysis (mCCA), multi-way partial least squares (N-PLS), independent vector analysis (IVA) and Parallel ICA are then reviewed and discussed in terms of their strengths and limitations. Finally, DL methods are introduced, focusing on three important applications: classification utilizing strategies for multimodal data augmentation, embedding of learned representations in order to reveal disease severity spectra, and multimodal tissue segmentation.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
While here “multi-way” refers to the order of a tensor (i.e., the number of data dimensions), the term multi-way has also been used in the literature to refer to the number of modalities being fused.
- 2.
IVA-G is identical to mCCA with the GENVAR cost, except it also allows non-orthogonal W.
- 3.
The IVA cost is a sum of M separate ICAs (one per dataset) with an additional term to increase/retain the mutual information between corresponding sources across datasets.
- 4.
The MATLAB code used for this study (available at http://www.fmrib.ox.ac.uk/ukbiobank/nnpaper/ukb_NN.m) actually implements this step as \(\left [\mathbf { A}_{\mathrm {CCA},1}, {\mathbf {A}}_{\mathrm {CCA},2}\right ] = F\left ({\mathbf {R}}^{\mathbf {y}\mathbf {x}}\right )\), where F(⋅) = atanh(⋅) is the element-wise Fisher transform of the C × (N 1 + N 2) cross-correlation matrix \({\mathbf {R}}^{\mathbf {y}\mathbf {x}} = \mathrm {diag}\left ({\mathbf {Y}}_{\mathrm {CCA}}^{} {\mathbf {Y}}_{\mathrm {CCA}}^\top \right )^{-\frac {1}{2}} (\mathbf { Y}_{\mathrm {CCA}}^{} \mathbf {X}) \mathrm {diag} \left ({\mathbf {X}}^\top \mathbf { X}\right )^{-\frac {1}{2}}\) between y CCA and x ⊤, diag(B) is a diagonal matrix containing only the diagonal elements of B, and \(\mathbf {X} = \left [{\mathbf {X}}_1, {\mathbf {X}}_2\right ]\) is a matrix concatenation. Equivalence to the form indicated in the main text is claimed but not proven.
- 5.
Note that the implementation of mCCA+jICA in that work utilized simple matrix transpose instead of the pseudo-inverses indicted above, possibly presuming that the columns of \({\mathbf {Y}}_{\mathrm {CCA}}^{\top }\) and rows of \(\left [{\mathbf {Y}}_{\mathrm {jICA},1}^{}, \mathbf { Y}_{\mathrm {jICA},2}^{}\right ]\) are orthonormal due to uncorrelation and independence, respectively.
References
Adalı T, Anderson M, Fu GS (2014) Diversity in independent component and vector analyses: Identifiability, algorithms, and applications in medical imaging. IEEE Signal Process Mag 31(3):18–33. https://doi.org/10.1109/MSP.2014.2300511
Adalı T, Levin-Schwartz Y, Calhoun VD (2015) Multimodal data fusion using source separation: Application to medical imaging. Proc IEEE 103(9):1494–1506. https://doi.org/10.1109/JPROC.2015.2461601
Anderson M, Li XL, Adalı T (2010) Nonorthogonal independent vector analysis using multivariate gaussian model. In: Vigneron V, Zarzoso V, Moreau E, Gribonval R, Vincent E (eds) Proc LVA/ICA 2010, Lecture Notes in Computer Science, vol 6365. Springer, St. Malo, France, pp 354–361. https://doi.org/10.1007/978-3-642-15995-4_44
Anderson M, Adalı T, Li XL (2012) Joint blind source separation with multivariate gaussian model: Algorithms and performance analysis. IEEE Trans Signal Process 60(4):1672–1683. https://doi.org/10.1109/TSP.2011.2181836
Anderson M, Fu GS, Phlypo R, Adalı T (2013) Independent vector analysis, the Kotz distribution, and performance bounds. In: Proc IEEE ICASSP 2013, Vancouver, BC, pp 3243–3247. https://doi.org/10.1109/ICASSP.2013.6638257
Bell A, Sejnowski T (1995) An information-maximization approach to blind separation and blind deconvolution. Neural Comput 7(6):1129–1159
Belouchrani A, Abed-Meraim K, Cardoso JF, Moulines E (1993) Second-order blind separation of temporally correlated sources. In: Proc ICDSP 1993, Nicosia, Cyprus, pp 346–351
Biessmann F, Plis S, Meinecke FC, Eichele T, Muller KR (2011) Analysis of multimodal neuroimaging data. IEEE Rev Biomed Eng 4:26–58. https://doi.org/10.1109/RBME.2011.2170675
Calhoun VD, Adalı T (2009) Feature-based fusion of medical imaging data. IEEE Trans Inf Technol Biomed 13(5):711–720. https://doi.org/10.1109/TITB.2008.923773
Calhoun VD, Sui J (2016) Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness. Biol Psychiatry Cogn Neurosci Neuroimaging 1(3):230–244. https://doi.org/10.1016/j.bpsc.2015.12.005
Calhoun VD, Adalı T, Giuliani NR, Pekar JJ, Kiehl KA, Pearlson GD (2006a) Method for multimodal analysis of independent source differences in schizophrenia: Combining gray matter structural and auditory oddball functional data. Hum Brain Mapp 27(1):47–62. https://doi.org/10.1002/hbm.20166
Calhoun VD, Adalı T, Kiehl K, Astur R, Pekar J, Pearlson G (2006b) A method for multi-task fMRI data fusion applied to schizophrenia. Hum Brain Mapp 27(7):598–610. https://doi.org/10.1002/hbm.20204
Calhoun VD, Adalı T, Pearlson GD, Kiehl KA (2006c) Neuronal chronometry of target detection: Fusion of hemodynamic and event-related potential data. NeuroImage 30(2):544–553. https://doi.org/10.1016/j.neuroimage.2005.08.060
Cardoso JF (1998) Multidimensional independent component analysis. In: Proc IEEE ICASSP 1998, Seattle, WA, vol 4, pp 1941–1944. https://doi.org/10.1109/ICASSP.1998.681443
Castro E, Ulloa A, Plis SM, Turner JA, Calhoun VD (2015) Generation of synthetic structural magnetic resonance images for deep learning pre-training. In: Proc IEEE ISBI 2015, pp 1057–1060. https://doi.org/10.1109/ISBI.2015.7164053
Chen K, Reiman EM, Huan Z, Caselli RJ, Bandy D, Ayutyanont N, Alexander GE (2009) Linking functional and structural brain images with multivariate network analyses: A novel application of the partial least square method. NeuroImage 47(2):602–610. https://doi.org/10.1016/j.neuroimage.2009.04.053
Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Proc MICCAI 2016, pp 424–432. https://doi.org/10.1007/978-3-319-46723-8_49
Comon P (1994) Independent component analysis, a new concept? Signal Process 36(3):287–314. https://doi.org/10.1016/0165-1684(94)90029-9
Comon P, Jutten C (2010) Handbook of blind source separation, 1st edn. Academic Press, Oxford, UK
Correa NM, Li YO, Adalı T, Calhoun VD (2008) Canonical correlation analysis for feature-based fusion of biomedical imaging modalities and its application to detection of associative networks in schizophrenia. IEEE J Sel Topics Signal Process 2(6):998–1007. https://doi.org/10.1109/JSTSP.2008.2008265
Correa NM, Li YO, Adalı T, Calhoun VD (2009) Fusion of fMRI, sMRI, and EEG data using canonical correlation analysis. In: Proc IEEE ICASSP 2009, pp 385–388. https://doi.org/10.1109/ICASSP.2009.4959601
Correa NM, Eichele T, Adalı T, Li YO, Calhoun VD (2010) Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI. Neuroimage 50(4):1438–1445. https://doi.org/10.1016/j.neuroimage.2010.01.062
Dähne S, Bießmann F, Meinecke F, Mehnert J, Fazli S, Müller KR (2013) Integration of multivariate data streams with bandpower signals. IEEE Trans Multimedia 15(5):1001–1013. https://doi.org/10.1109/TMM.2013.2250267
Dähne S, Meinecke F, Haufe S, Höhne J, Tangermann M, Müller KR, Nikulin V (2014a) SPoC: A novel framework for relating the amplitude of neuronal oscillations to behaviorally relevant parameters. NeuroImage 86:111–122. https://doi.org/10.1016/j.neuroimage.2013.07.079
Dähne S, Nikulin V, Ramírez D, Schreier P, Müller KR, Haufe S (2014b) Finding brain oscillations with power dependencies in neuroimaging data. NeuroImage 96:334–348. https://doi.org/10.1016/j.neuroimage.2014.03.075
Dale AM, Fischl B, Sereno MI (1999) Cortical surface-based analysis: I. segmentation and surface reconstruction. NeuroImage 9(2):179–194. https://doi.org/10.1006/nimg.1998.0395
Fedorov A, Damaraju E, Calhoun V, Plis S (2017a) Almost instant brain atlas segmentation for large-scale studies. arXiv preprint URL http://arxiv.org/abs/1711.00457
Fedorov A, Johnson J, Damaraju E, Ozerin A, Calhoun V, Plis S (2017b) End-to-end learning of brain tissue segmentation from imperfect labeling. In: Proc IJCNN 2017, pp 3785–3792. https://doi.org/10.1109/IJCNN.2017.7966333
Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press. http://www.deeplearningbook.org
Haykin S (2008) Neural networks and learning machines, 3rd edn. Prentice Hall, Upper Saddle River, NJ
Hotelling H (1936) Relations between two sets of variates. Biometrika 28(3/4):321–377. https://doi.org/10.2307/2333955
Hyvärinen A, Erkki O (1997) A fast fixed-point algorithm for independent component analysis. Neural Comput 9(7):1483–1492. https://doi.org/10.1162/neco.1997.9.7.1483
Hyvärinen A, Köster U (2006) FastISA: A fast fixed-point algorithm for independent subspace analysis. In: Proc ESANN 2006, Bruges, Belgium, pp 371–376
Hyvärinen A, Karhunen J, Oja E (2002) Independent component analysis, 1st edn. Wiley, New York, NY
Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proc ICML 2015, Lille, France, vol 37, pp 448–456
Karahan E, Rojas-López PA, Bringas-Vega ML, Valdés-Hernández PA, Valdés-Sosa PA (2015) Tensor analysis and fusion of multimodal brain images. Proc IEEE 103(9):1531–1559. https://doi.org/10.1109/JPROC.2015.2455028
Kettenring J (1971) Canonical analysis of several sets of variables. Biometrika 58(3):433–451. https://doi.org/10.2307/2334380
Kim T, Eltoft T, Lee TW (2006) Independent vector analysis: An extension of ICA to multivariate components. In: Proc ICA 2006, Springer, Charleston, SC, Lecture Notes in Computer Science, vol 3889, pp 165–172. https://doi.org/10.1007/11679363_21
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Proc NIPS 2012, pp 1097–1105
Lahat D, Jutten C (2015) Joint independent subspace analysis: A quasi-Newton algorithm. In: Proc LVA/ICA 2015, Springer, Liberec, Czech Republic, Lecture Notes in Computer Science, vol 9237, pp 111–118. https://doi.org/10.1007/978-3-319-22482-4_13
Lahat D, Cardoso J, Messer H (2012) Second-order multidimensional ICA: Performance analysis. IEEE Trans Signal Process 60(9):4598–4610. https://doi.org/10.1109/TSP.2012.2199985
Lahat D, Adalı T, Jutten C (2015) Multimodal data fusion: An overview of methods, challenges, and prospects. Proc IEEE 103(9):1449–1477. https://doi.org/10.1109/JPROC.2015.2460697
Liu J, Pearlson G, Calhoun V, Windemuth A (2007) A novel approach to analyzing fMRI and SNP data via parallel independent component analysis. Proc SPIE 6511:651,113–651,113–11. https://doi.org/10.1117/12.709344
Liu J, Pearlson G, Windemuth A, Ruano G, Perrone-Bizzozero NI, Calhoun VD (2009) Combining fMRI and SNP data to investigate connections between brain function and genetics using parallel ICA. Hum Brain Mapp 30(1):241–255. https://doi.org/10.1002/hbm.20508
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proc IEEE CVPR 2015, pp 3431–3440. https://doi.org/10.1109/CVPR.2015.7298965
Lorenzi M, Simpson IJ, Mendelson AF, Vos SB, Cardoso MJ, Modat M, Schott JM, Ourselin S (2016) Multimodal image analysis in Alzheimer’s disease via statistical modelling of non-local intensity correlations. Sci Rep 6:22,161. https://doi.org/10.1038/srep22161
Maaten Lvd, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(Nov):2579–2605
Martínez-Montes E, Valdés-Sosa PA, Miwakeichi F, Goldman RI, Cohen MS (2004) Concurrent EEG/fMRI analysis by multiway partial least squares. NeuroImage 22(3):1023–1034. https://doi.org/10.1016/j.neuroimage.2004.03.038
Meda S, Narayanan B, Liu J, Perrone-Bizzozero N, Stevens M, Calhoun VD, Glahn D, Shen L, Risacher S, Saykin A, Pearlson G (2012) A large scale multivariate parallel ICA method reveals novel imaging-genetic relationships for Alzheimer’s disease in the ADNI cohort. NeuroImage 60(3):1608–1621. https://doi.org/10.1016/j.neuroimage.2011.12.076
Meier T, Wildenberg J, Liu J, Chen J, Calhoun VD, Biswal B, Meyerand M, Birn R, Prabhakaran V (2012) Parallel ICA identifies sub-components of resting state networks that covary with behavioral indices. Front Hum Neurosci 6:281. https://doi.org/10.3389/fnhum.2012.00281
Miller KL, Alfaro-Almagro F, Bangerter NK, Thomas DL, Yacoub E, Xu J, Bartsch AJ, Jbabdi S, Sotiropoulos SN, Andersson JLR, Griffanti L, Douaud G, Okell TW, Weale P, Dragonu I, Garratt S, Hudson S, Collins R, Jenkinson M, Matthews PM, Smith SM (2016) Multimodal population brain imaging in the UK Biobank prospective epidemiological study. Nat Neurosci 19(11):1523–1536. https://doi.org/10.1038/nn.4393
Mohammadi-Nejad AR, Hossein-Zadeh GA, Soltanian-Zadeh H (2017) Structured and sparse canonical correlation analysis as a brain-wide multi-modal data fusion approach. IEEE Trans Med Imaging 36(7):1438–1448. https://doi.org/10.1109/TMI.2017.2681966
Plis SM, Hjelm DR, Salakhutdinov R, Allen EA, Bockholt HJ, Long JD, Johnson HJ, Paulsen JS, Turner JA, Calhoun VD (2014) Deep learning for neuroimaging: a validation study. Front Neurosci 8:229. https://doi.org/10.3389/fnins.2014.00229
Silva RF, Plis SM, Adalı T, Calhoun VD (2014a) Multidataset independent subspace analysis. In: Proc OHBM 2014, Poster 3506
Silva RF, Plis SM, Adalı T, Calhoun VD (2014b) Multidataset independent subspace analysis extends independent vector analysis. In: Proc IEEE ICIP 2014, Paris, France, pp 2864–2868. https://doi.org/10.1109/ICIP.2014.7025579
Silva RF, Plis SM, Adalı T, Calhoun VD (2014c) A statistically motivated framework for simulation of stochastic data fusion models applied to multimodal neuroimaging. NeuroImage 102, Part 1:92–117. https://doi.org/10.1016/j.neuroimage.2014.04.035
Silva RF, Plis SM, Sui J, Pattichis MS, Adalı T, Calhoun VD (2016) Blind source separation for unimodal and multimodal brain networks: A unifying framework for subspace modeling. IEEE J Sel Topics Signal Process 10(7):1134–1149. https://doi.org/10.1109/JSTSP.2016.2594945
Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: A simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958
Sui J, Adalı T, Pearlson G, Yange H, Sponheim S, White T, Calhoun V (2010) A CCA + ICA based model for multi-task brain imaging data fusion and its application to schizophrenia. NeuroImage 51(1):123–134. https://doi.org/10.1016/j.neuroimage.2010.01.069
Sui J, Pearlson G, Caprihan A, Adalı T, Kiehl K, Liu J, Yamamoto J, Calhoun VD (2011) Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA + joint ICA model. NeuroImage 57(3):839–855. https://doi.org/10.1016/j.neuroimage.2011.05.055
Sui J, He H, Yu Q, Chen J, Rogers J, Pearlson G, Mayer A, Bustillo J, Canive J, Calhoun VD (2013) Combination of resting state fMRI, DTI and sMRI data to discriminate schizophrenia by N-way MCCA+jICA. Front Hum Neurosci 7(235). https://doi.org/10.3389/fnhum.2013.00235
Szabó Z, Póczos B, Lőrincz A (2012) Separation theorem for independent subspace analysis and its consequences. Pattern Recognit 45(4):1782–1791. https://doi.org/10.1016/j.patcog.2011.09.007
Ulloa A, Plis S, Erhardt E, Calhoun V (2015) Synthetic structural magnetic resonance image generator improves deep learning prediction of schizophrenia. In: Proc IEEE MLSP 2015, pp 1–6. https://doi.org/10.1109/MLSP.2015.7324379
Ulloa A, Plis SM, Calhoun VD (2018) Improving classification rate of schizophrenia using a multimodal multi-layer perceptron model with structural and functional MR. arXiv preprint URL http://arxiv.org/abs/1804.04591
Vergara VM, Ulloa A, Calhoun VD, Boutte D, Chen J, Liu J (2014) A three-way parallel ICA approach to analyze links among genetics, brain structure and brain function. NeuroImage 98:386–394. https://doi.org/10.1016/j.neuroimage.2014.04.060
Wang P, Chen K, Yao L, Hu B, Wu X, Zhang J, Ye Q, Guo X (2016) Multimodal classification of mild cognitive impairment based on partial least squares. J Alzheimers Dis 54(1):359–371. https://doi.org/10.3233/JAD-160102
Wold H (1966) Nonlinear estimation by iterative least squares procedures. In: David F (ed) Research papers in statistics. Festschrift for J. Neyman. Wiley, New York, NY, pp 411–444
Yeredor A (2000) Blind separation of gaussian sources via second-order statistics with asymptotically optimal weighting. IEEE Signal Process Lett 7(7):197–200. https://doi.org/10.1109/97.847367
Zhou YT, Chellappa R (1988) Computation of optical flow using a neural network. In: Proc IEEE ICNN 1988, vol 2, pp 71–78. https://doi.org/10.1109/ICNN.1988.23914
Acknowledgements
We would like to thank Dr. Vince Calhoun for the useful discussions, as well as Alvaro Ulloa and Aleksandr Fedorov for kindly providing some of the images and results presented here. This work was supported by NIH grants R01EB006841 (SP), 2R01EB005846 (RS), and R01EB020407 (RS), NSF grants IIS-1318759 (SP), 1539067 (RS), and NIH NIGMS Center of Biomedical Research Excellent (COBRE) grant 5P20RR021938/P20GM103472/P30GM122734.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Silva, R.F., Plis, S.M. (2019). How to Integrate Data from Multiple Biological Layers in Mental Health?. In: Passos, I., Mwangi, B., Kapczinski, F. (eds) Personalized Psychiatry. Springer, Cham. https://doi.org/10.1007/978-3-030-03553-2_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-03553-2_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03552-5
Online ISBN: 978-3-030-03553-2
eBook Packages: MedicineMedicine (R0)