Advertisement

How to Integrate Data from Multiple Biological Layers in Mental Health?

  • Rogers F. Silva
  • Sergey M. PlisEmail author
Chapter

Abstract

Integrating information from multiple biological layers is a key approach to unraveling the complexities of the human brain, with its multiple overlapping structural and functional subsystems operating at widely different temporal and spatial scales. Moreover, it has true potential to positively impact mental health patients through early diagnosis and individualized treatment. This chapter lays out a succession of approaches to synergistic fusion of multimodal brain imaging data, with a special focus on blind source separation (BSS) and deep learning (DL) methods. Firstly, a broad unified description of the BSS field is introduced, serving as a theoretical backbone for the chapter. Complementary to that, a detailed case study of three different applications of joint independent component analysis (jICA) provides both a reference guide on data fusion and a bridge into more advanced BSS methods. Various advanced BSS methods such as multiset canonical correlation analysis (mCCA), multi-way partial least squares (N-PLS), independent vector analysis (IVA) and Parallel ICA are then reviewed and discussed in terms of their strengths and limitations. Finally, DL methods are introduced, focusing on three important applications: classification utilizing strategies for multimodal data augmentation, embedding of learned representations in order to reveal disease severity spectra, and multimodal tissue segmentation.

Keywords

Data fusion Data reduction Deep learning Assignment matrix Biological layer Blind source separation Canonical correlation analysis Convolutional neural networks Independent component analysis Multidataset multidimensional 

Notes

Acknowledgements

We would like to thank Dr. Vince Calhoun for the useful discussions, as well as Alvaro Ulloa and Aleksandr Fedorov for kindly providing some of the images and results presented here. This work was supported by NIH grants R01EB006841 (SP), 2R01EB005846 (RS), and R01EB020407 (RS), NSF grants IIS-1318759 (SP), 1539067 (RS), and NIH NIGMS Center of Biomedical Research Excellent (COBRE) grant 5P20RR021938/P20GM103472/P30GM122734.

References

  1. Adalı T, Anderson M, Fu GS (2014) Diversity in independent component and vector analyses: Identifiability, algorithms, and applications in medical imaging. IEEE Signal Process Mag 31(3):18–33.  https://doi.org/10.1109/MSP.2014.2300511 Google Scholar
  2. Adalı T, Levin-Schwartz Y, Calhoun VD (2015) Multimodal data fusion using source separation: Application to medical imaging. Proc IEEE 103(9):1494–1506.  https://doi.org/10.1109/JPROC.2015.2461601 Google Scholar
  3. Anderson M, Li XL, Adalı T (2010) Nonorthogonal independent vector analysis using multivariate gaussian model. In: Vigneron V, Zarzoso V, Moreau E, Gribonval R, Vincent E (eds) Proc LVA/ICA 2010, Lecture Notes in Computer Science, vol 6365. Springer, St. Malo, France, pp 354–361. https://doi.org/10.1007/978-3-642-15995-4_44 Google Scholar
  4. Anderson M, Adalı T, Li XL (2012) Joint blind source separation with multivariate gaussian model: Algorithms and performance analysis. IEEE Trans Signal Process 60(4):1672–1683.  https://doi.org/10.1109/TSP.2011.2181836 Google Scholar
  5. Anderson M, Fu GS, Phlypo R, Adalı T (2013) Independent vector analysis, the Kotz distribution, and performance bounds. In: Proc IEEE ICASSP 2013, Vancouver, BC, pp 3243–3247.  https://doi.org/10.1109/ICASSP.2013.6638257 Google Scholar
  6. Bell A, Sejnowski T (1995) An information-maximization approach to blind separation and blind deconvolution. Neural Comput 7(6):1129–1159PubMedGoogle Scholar
  7. Belouchrani A, Abed-Meraim K, Cardoso JF, Moulines E (1993) Second-order blind separation of temporally correlated sources. In: Proc ICDSP 1993, Nicosia, Cyprus, pp 346–351Google Scholar
  8. Biessmann F, Plis S, Meinecke FC, Eichele T, Muller KR (2011) Analysis of multimodal neuroimaging data. IEEE Rev Biomed Eng 4:26–58.  https://doi.org/10.1109/RBME.2011.2170675 PubMedGoogle Scholar
  9. Calhoun VD, Adalı T (2009) Feature-based fusion of medical imaging data. IEEE Trans Inf Technol Biomed 13(5):711–720.  https://doi.org/10.1109/TITB.2008.923773 PubMedGoogle Scholar
  10. Calhoun VD, Sui J (2016) Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness. Biol Psychiatry Cogn Neurosci Neuroimaging 1(3):230–244. https://doi.org/10.1016/j.bpsc.2015.12.005 PubMedPubMedCentralGoogle Scholar
  11. Calhoun VD, Adalı T, Giuliani NR, Pekar JJ, Kiehl KA, Pearlson GD (2006a) Method for multimodal analysis of independent source differences in schizophrenia: Combining gray matter structural and auditory oddball functional data. Hum Brain Mapp 27(1):47–62.  https://doi.org/10.1002/hbm.20166 PubMedGoogle Scholar
  12. Calhoun VD, Adalı T, Kiehl K, Astur R, Pekar J, Pearlson G (2006b) A method for multi-task fMRI data fusion applied to schizophrenia. Hum Brain Mapp 27(7):598–610.  https://doi.org/10.1002/hbm.20204 PubMedPubMedCentralGoogle Scholar
  13. Calhoun VD, Adalı T, Pearlson GD, Kiehl KA (2006c) Neuronal chronometry of target detection: Fusion of hemodynamic and event-related potential data. NeuroImage 30(2):544–553. https://doi.org/10.1016/j.neuroimage.2005.08.060 PubMedGoogle Scholar
  14. Cardoso JF (1998) Multidimensional independent component analysis. In: Proc IEEE ICASSP 1998, Seattle, WA, vol 4, pp 1941–1944.  https://doi.org/10.1109/ICASSP.1998.681443
  15. Castro E, Ulloa A, Plis SM, Turner JA, Calhoun VD (2015) Generation of synthetic structural magnetic resonance images for deep learning pre-training. In: Proc IEEE ISBI 2015, pp 1057–1060.  https://doi.org/10.1109/ISBI.2015.7164053 Google Scholar
  16. Chen K, Reiman EM, Huan Z, Caselli RJ, Bandy D, Ayutyanont N, Alexander GE (2009) Linking functional and structural brain images with multivariate network analyses: A novel application of the partial least square method. NeuroImage 47(2):602–610. https://doi.org/10.1016/j.neuroimage.2009.04.053 PubMedPubMedCentralGoogle Scholar
  17. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Proc MICCAI 2016, pp 424–432. https://doi.org/10.1007/978-3-319-46723-8_49 Google Scholar
  18. Comon P (1994) Independent component analysis, a new concept? Signal Process 36(3):287–314. https://doi.org/10.1016/0165-1684(94)90029-9 Google Scholar
  19. Comon P, Jutten C (2010) Handbook of blind source separation, 1st edn. Academic Press, Oxford, UKGoogle Scholar
  20. Correa NM, Li YO, Adalı T, Calhoun VD (2008) Canonical correlation analysis for feature-based fusion of biomedical imaging modalities and its application to detection of associative networks in schizophrenia. IEEE J Sel Topics Signal Process 2(6):998–1007.  https://doi.org/10.1109/JSTSP.2008.2008265 Google Scholar
  21. Correa NM, Li YO, Adalı T, Calhoun VD (2009) Fusion of fMRI, sMRI, and EEG data using canonical correlation analysis. In: Proc IEEE ICASSP 2009, pp 385–388.  https://doi.org/10.1109/ICASSP.2009.4959601 Google Scholar
  22. Correa NM, Eichele T, Adalı T, Li YO, Calhoun VD (2010) Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI. Neuroimage 50(4):1438–1445. https://doi.org/10.1016/j.neuroimage.2010.01.062 PubMedPubMedCentralGoogle Scholar
  23. Dähne S, Bießmann F, Meinecke F, Mehnert J, Fazli S, Müller KR (2013) Integration of multivariate data streams with bandpower signals. IEEE Trans Multimedia 15(5):1001–1013.  https://doi.org/10.1109/TMM.2013.2250267 Google Scholar
  24. Dähne S, Meinecke F, Haufe S, Höhne J, Tangermann M, Müller KR, Nikulin V (2014a) SPoC: A novel framework for relating the amplitude of neuronal oscillations to behaviorally relevant parameters. NeuroImage 86:111–122. https://doi.org/10.1016/j.neuroimage.2013.07.079 PubMedGoogle Scholar
  25. Dähne S, Nikulin V, Ramírez D, Schreier P, Müller KR, Haufe S (2014b) Finding brain oscillations with power dependencies in neuroimaging data. NeuroImage 96:334–348. https://doi.org/10.1016/j.neuroimage.2014.03.075 PubMedGoogle Scholar
  26. Dale AM, Fischl B, Sereno MI (1999) Cortical surface-based analysis: I. segmentation and surface reconstruction. NeuroImage 9(2):179–194.  https://doi.org/10.1006/nimg.1998.0395 PubMedGoogle Scholar
  27. Fedorov A, Damaraju E, Calhoun V, Plis S (2017a) Almost instant brain atlas segmentation for large-scale studies. arXiv preprint URL http://arxiv.org/abs/1711.00457
  28. Fedorov A, Johnson J, Damaraju E, Ozerin A, Calhoun V, Plis S (2017b) End-to-end learning of brain tissue segmentation from imperfect labeling. In: Proc IJCNN 2017, pp 3785–3792.  https://doi.org/10.1109/IJCNN.2017.7966333 Google Scholar
  29. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press. http://www.deeplearningbook.org
  30. Haykin S (2008) Neural networks and learning machines, 3rd edn. Prentice Hall, Upper Saddle River, NJGoogle Scholar
  31. Hotelling H (1936) Relations between two sets of variates. Biometrika 28(3/4):321–377. https://doi.org/10.2307/2333955 Google Scholar
  32. Hyvärinen A, Erkki O (1997) A fast fixed-point algorithm for independent component analysis. Neural Comput 9(7):1483–1492.  https://doi.org/10.1162/neco.1997.9.7.1483 Google Scholar
  33. Hyvärinen A, Köster U (2006) FastISA: A fast fixed-point algorithm for independent subspace analysis. In: Proc ESANN 2006, Bruges, Belgium, pp 371–376Google Scholar
  34. Hyvärinen A, Karhunen J, Oja E (2002) Independent component analysis, 1st edn. Wiley, New York, NYGoogle Scholar
  35. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proc ICML 2015, Lille, France, vol 37, pp 448–456Google Scholar
  36. Karahan E, Rojas-López PA, Bringas-Vega ML, Valdés-Hernández PA, Valdés-Sosa PA (2015) Tensor analysis and fusion of multimodal brain images. Proc IEEE 103(9):1531–1559.  https://doi.org/10.1109/JPROC.2015.2455028 Google Scholar
  37. Kettenring J (1971) Canonical analysis of several sets of variables. Biometrika 58(3):433–451. https://doi.org/10.2307/2334380 Google Scholar
  38. Kim T, Eltoft T, Lee TW (2006) Independent vector analysis: An extension of ICA to multivariate components. In: Proc ICA 2006, Springer, Charleston, SC, Lecture Notes in Computer Science, vol 3889, pp 165–172. https://doi.org/10.1007/11679363_21 Google Scholar
  39. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Proc NIPS 2012, pp 1097–1105Google Scholar
  40. Lahat D, Jutten C (2015) Joint independent subspace analysis: A quasi-Newton algorithm. In: Proc LVA/ICA 2015, Springer, Liberec, Czech Republic, Lecture Notes in Computer Science, vol 9237, pp 111–118. https://doi.org/10.1007/978-3-319-22482-4_13 Google Scholar
  41. Lahat D, Cardoso J, Messer H (2012) Second-order multidimensional ICA: Performance analysis. IEEE Trans Signal Process 60(9):4598–4610.  https://doi.org/10.1109/TSP.2012.2199985 Google Scholar
  42. Lahat D, Adalı T, Jutten C (2015) Multimodal data fusion: An overview of methods, challenges, and prospects. Proc IEEE 103(9):1449–1477.  https://doi.org/10.1109/JPROC.2015.2460697 Google Scholar
  43. Liu J, Pearlson G, Calhoun V, Windemuth A (2007) A novel approach to analyzing fMRI and SNP data via parallel independent component analysis. Proc SPIE 6511:651,113–651,113–11. https://doi.org/10.1117/12.709344
  44. Liu J, Pearlson G, Windemuth A, Ruano G, Perrone-Bizzozero NI, Calhoun VD (2009) Combining fMRI and SNP data to investigate connections between brain function and genetics using parallel ICA. Hum Brain Mapp 30(1):241–255.  https://doi.org/10.1002/hbm.20508 PubMedPubMedCentralGoogle Scholar
  45. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proc IEEE CVPR 2015, pp 3431–3440.  https://doi.org/10.1109/CVPR.2015.7298965 Google Scholar
  46. Lorenzi M, Simpson IJ, Mendelson AF, Vos SB, Cardoso MJ, Modat M, Schott JM, Ourselin S (2016) Multimodal image analysis in Alzheimer’s disease via statistical modelling of non-local intensity correlations. Sci Rep 6:22,161.  https://doi.org/10.1038/srep22161 Google Scholar
  47. Maaten Lvd, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(Nov):2579–2605Google Scholar
  48. Martínez-Montes E, Valdés-Sosa PA, Miwakeichi F, Goldman RI, Cohen MS (2004) Concurrent EEG/fMRI analysis by multiway partial least squares. NeuroImage 22(3):1023–1034. https://doi.org/10.1016/j.neuroimage.2004.03.038 PubMedGoogle Scholar
  49. Meda S, Narayanan B, Liu J, Perrone-Bizzozero N, Stevens M, Calhoun VD, Glahn D, Shen L, Risacher S, Saykin A, Pearlson G (2012) A large scale multivariate parallel ICA method reveals novel imaging-genetic relationships for Alzheimer’s disease in the ADNI cohort. NeuroImage 60(3):1608–1621. https://doi.org/10.1016/j.neuroimage.2011.12.076 PubMedPubMedCentralGoogle Scholar
  50. Meier T, Wildenberg J, Liu J, Chen J, Calhoun VD, Biswal B, Meyerand M, Birn R, Prabhakaran V (2012) Parallel ICA identifies sub-components of resting state networks that covary with behavioral indices. Front Hum Neurosci 6:281.  https://doi.org/10.3389/fnhum.2012.00281 PubMedPubMedCentralGoogle Scholar
  51. Miller KL, Alfaro-Almagro F, Bangerter NK, Thomas DL, Yacoub E, Xu J, Bartsch AJ, Jbabdi S, Sotiropoulos SN, Andersson JLR, Griffanti L, Douaud G, Okell TW, Weale P, Dragonu I, Garratt S, Hudson S, Collins R, Jenkinson M, Matthews PM, Smith SM (2016) Multimodal population brain imaging in the UK Biobank prospective epidemiological study. Nat Neurosci 19(11):1523–1536. https://doi.org/10.1038/nn.4393 PubMedPubMedCentralGoogle Scholar
  52. Mohammadi-Nejad AR, Hossein-Zadeh GA, Soltanian-Zadeh H (2017) Structured and sparse canonical correlation analysis as a brain-wide multi-modal data fusion approach. IEEE Trans Med Imaging 36(7):1438–1448.  https://doi.org/10.1109/TMI.2017.2681966 PubMedGoogle Scholar
  53. Plis SM, Hjelm DR, Salakhutdinov R, Allen EA, Bockholt HJ, Long JD, Johnson HJ, Paulsen JS, Turner JA, Calhoun VD (2014) Deep learning for neuroimaging: a validation study. Front Neurosci 8:229.  https://doi.org/10.3389/fnins.2014.00229 PubMedPubMedCentralGoogle Scholar
  54. Silva RF, Plis SM, Adalı T, Calhoun VD (2014a) Multidataset independent subspace analysis. In: Proc OHBM 2014, Poster 3506Google Scholar
  55. Silva RF, Plis SM, Adalı T, Calhoun VD (2014b) Multidataset independent subspace analysis extends independent vector analysis. In: Proc IEEE ICIP 2014, Paris, France, pp 2864–2868.  https://doi.org/10.1109/ICIP.2014.7025579 Google Scholar
  56. Silva RF, Plis SM, Adalı T, Calhoun VD (2014c) A statistically motivated framework for simulation of stochastic data fusion models applied to multimodal neuroimaging. NeuroImage 102, Part 1:92–117. https://doi.org/10.1016/j.neuroimage.2014.04.035 Google Scholar
  57. Silva RF, Plis SM, Sui J, Pattichis MS, Adalı T, Calhoun VD (2016) Blind source separation for unimodal and multimodal brain networks: A unifying framework for subspace modeling. IEEE J Sel Topics Signal Process 10(7):1134–1149.  https://doi.org/10.1109/JSTSP.2016.2594945 Google Scholar
  58. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: A simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958Google Scholar
  59. Sui J, Adalı T, Pearlson G, Yange H, Sponheim S, White T, Calhoun V (2010) A CCA + ICA based model for multi-task brain imaging data fusion and its application to schizophrenia. NeuroImage 51(1):123–134. https://doi.org/10.1016/j.neuroimage.2010.01.069 PubMedPubMedCentralGoogle Scholar
  60. Sui J, Pearlson G, Caprihan A, Adalı T, Kiehl K, Liu J, Yamamoto J, Calhoun VD (2011) Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA + joint ICA model. NeuroImage 57(3):839–855. https://doi.org/10.1016/j.neuroimage.2011.05.055 PubMedPubMedCentralGoogle Scholar
  61. Sui J, He H, Yu Q, Chen J, Rogers J, Pearlson G, Mayer A, Bustillo J, Canive J, Calhoun VD (2013) Combination of resting state fMRI, DTI and sMRI data to discriminate schizophrenia by N-way MCCA+jICA. Front Hum Neurosci 7(235).  https://doi.org/10.3389/fnhum.2013.00235
  62. Szabó Z, Póczos B, Lőrincz A (2012) Separation theorem for independent subspace analysis and its consequences. Pattern Recognit 45(4):1782–1791. https://doi.org/10.1016/j.patcog.2011.09.007 Google Scholar
  63. Ulloa A, Plis S, Erhardt E, Calhoun V (2015) Synthetic structural magnetic resonance image generator improves deep learning prediction of schizophrenia. In: Proc IEEE MLSP 2015, pp 1–6.  https://doi.org/10.1109/MLSP.2015.7324379 Google Scholar
  64. Ulloa A, Plis SM, Calhoun VD (2018) Improving classification rate of schizophrenia using a multimodal multi-layer perceptron model with structural and functional MR. arXiv preprint URL http://arxiv.org/abs/1804.04591
  65. Vergara VM, Ulloa A, Calhoun VD, Boutte D, Chen J, Liu J (2014) A three-way parallel ICA approach to analyze links among genetics, brain structure and brain function. NeuroImage 98:386–394. https://doi.org/10.1016/j.neuroimage.2014.04.060 PubMedPubMedCentralGoogle Scholar
  66. Wang P, Chen K, Yao L, Hu B, Wu X, Zhang J, Ye Q, Guo X (2016) Multimodal classification of mild cognitive impairment based on partial least squares. J Alzheimers Dis 54(1):359–371.  https://doi.org/10.3233/JAD-160102 PubMedGoogle Scholar
  67. Wold H (1966) Nonlinear estimation by iterative least squares procedures. In: David F (ed) Research papers in statistics. Festschrift for J. Neyman. Wiley, New York, NY, pp 411–444Google Scholar
  68. Yeredor A (2000) Blind separation of gaussian sources via second-order statistics with asymptotically optimal weighting. IEEE Signal Process Lett 7(7):197–200. https://doi.org/10.1109/97.847367 Google Scholar
  69. Zhou YT, Chellappa R (1988) Computation of optical flow using a neural network. In: Proc IEEE ICNN 1988, vol 2, pp 71–78.  https://doi.org/10.1109/ICNN.1988.23914 Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.The Mind Research NetworkAlbuquerqueUSA

Personalised recommendations