Trends of Multimodal Neural Engineering Study: A Bibliometric Review


Neural engineering, an emerging interdisciplinary subject, is aimed at using engineering techniques to investigate the function and manipulate the behavior of the nervous system. The development of technology along with the advancement in Science helps to apply increasing multimodal research into the field of neural engineering, which has promoted the development of neural engineering. In this paper, a bibliometric analysis of 808 articles in Web of Science from 2003 to 2019 was conducted to determine the current status and future trends of multimodal neural engineering study. This paper conducted a five-step bibliometric analysis based on the proposed multimodal neural engineering research framework (NE-MUL). The results showed that in the past 17 years, multimodal research not only made great contributions to the development of neural engineering, but also brought this field a series of new problems (multimodal fusion, recurrent multimodal learning, multimodal convolutional network, etc.) This paper generated a map of existing research findings with their relationship and provided future researchers with meaningful suggestions and assistance.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10


  1. 1.

    Ereifej ES, Shell CE, Schofield JS, Charkhkar H, Cuberovic I, Dorval AD, Graczyk EL, Kozai TDY, Otto KJ, Tyler DJ, Welle CG, Widge AS, Zariffa J, Moritz CT, Bourbeau DJ, Marasco PD (2019) Neural engineering: the process, applications, and its role in the future of medicine. J Neural Eng 16:6–11

    Article  Google Scholar 

  2. 2.

    Lee DG, Lee SW (2019) Prediction of partially observed human activity based on pre-trained deep representation. Pattern Recogn 85:198–206

    Article  Google Scholar 

  3. 3.

    Glory HA, Vigneswaran C, Sriram VSS (2020) Unsupervised bin-wise pre-training: A fusion of information theory and hypergraph. Knowl-Based Syst 195:17

    Google Scholar 

  4. 4.

    Shi BT, Ji L, Liang YB, Duan N, Chen P, Niu ZD, Zhou M (2019) Dense procedure captioning in narrated instructional videos. Assoc Computational Linguistics-Acl, Stroudsburg

    Google Scholar 

  5. 5.

    Parra C, Iftekharuddin K, Rendon n D (2003) Wavelet based estimation of the fractal dimension in fBm images. In: Wolf LJ, Strock JL (eds), 1st International IEEE Embs Conference on Neural Engineering 2003 on Conference Proceedings. IEEE , New York, pp 533–536

  6. 6.

    Ganbat T, Chong HY, Liao PC, Wu YD (2018) A bibliometric review on risk management and building information modeling for international construction. Adv Civil Eng, p 13

  7. 7.

    Chen C, Chen Y, Hou J, Liang Y (2009) CiteSpace II: detecting and visualizing emerging trends and transient patterns in scientific literature. J China Soc Sci Tech Inf 28(3):401–421

    Google Scholar 

  8. 8.

    Blashfield RK, Kaufman L, Rousseeuw PJ (1991) Finding groups in data—an introduction to cluster-analysis. J Classif 8(2):277–279

    Google Scholar 

  9. 9.

    Newman MEJ (2006) Modularity and community structure in networks. Proc Natl Acad Sci USA 103(23):8577–8582

    Article  Google Scholar 

  10. 10.

    Baldi P (2018) Deep learning in biomedical data science. In: Altman RB, Levitt M, Palo A (eds) Annual review of biomedical data science, vol 1. Annual Reviews, pp 181–205

  11. 11.

    Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Article  Google Scholar 

  12. 12.

    Borton F, Laiwalla S, Park YM, Aceros J (2010) Listening to brain microcircuits for interfacing with external world-progress in wireless implantable microelectronic neuroengineering devices. Proc IEEE 98(3):375–388

    Article  Google Scholar 

  13. 13.

    Prochazka A (2017) Neurophysiology and neural engineering: a review. J Neurophysiol 118(2):1292–1309

    Article  Google Scholar 

  14. 14.

    Bachmann J, Munzert J, Kruger B (2018) Neural underpinnings of the perception of emotional states derived from biological human motion: a review of neuroimaging research. Front Psychol 9:12

    Google Scholar 

  15. 15.

    Mushtaq R, Porrino J, Moses SA, Perez-Carrillo GJG (2018) Neural foraminal lesions: an imaging overview. Pm&R 10(8):880–885

    Article  Google Scholar 

  16. 16.

    Wang DX, Cui P, Ou MD, Zhu WW (2015) Learning compact hash codes for multimodal representations using orthogonal deep structure. IEEE Trans Multimedia 17(9):1404–1416

    Article  Google Scholar 

  17. 17.

    Niu ZX, Zhou M, Wang L, Gao XB, Hua G (2017) Hierarchical Multimodal LSTM for dense visual-semantic embedding. In: 2017 IEEE international conference on computer vision. IEEE, New York, pp 1899–1907

  18. 18.

    Song ZC, Ni BB, Yan YC, Ren Z, Xu Y, Yang XK, Acm, (2017) Deep cross-modality alignment for multi-shot person re-identification. Assoc Computing Machinery, New York

    Google Scholar 

  19. 19.

    Jiang XY, Wu F, Zhang Y, Tang SL, Lu WM, Zhuang YT (2015) The classification of multi-modal data with hidden conditional random field. Pattern Recogn Lett 51:63–69

    Article  Google Scholar 

  20. 20.

    Jin ZW, Cao J, Guo H, Zhang YD, Luo JB (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. Assoc Computing Machinery, New York

    Google Scholar 

  21. 21.

    Liu Y, Chen X, Peng H, Wang ZF (2017) Multi-focus image fusion with a deep convolutional neural network. Inform Fusion 36:191–207

    Article  Google Scholar 

  22. 22.

    Feng T, Mao X (2017) Multimodal data fusion for SB-JPALS status prediction under antenna motion fault mode. Neurocomputing 259:46–54

    Article  Google Scholar 

  23. 23.

    Waterman E, Keliher RW, Tannous BA (2012) Single reporter for targeted multimodal in vivo imaging. J Am Chem Soc 134(11):5149–5156

    Article  Google Scholar 

  24. 24.

    Thukral A, Ershad F, Enan N, Rao Z, Yu CJ (2018) Soft ultrathin silicon electronics for soft neural interfaces a review of recent advances of soft neural interfaces based on ultrathin silicon. IEEE Nanatechnol Mag 12(1):21–34

    Article  Google Scholar 

  25. 25.

    Hu KJ, Chen C, Meng QY, Williams Z, Xu WD (2016) Scientific profile of brain-computer interfaces: Bibliometric analysis in a 10-year period. Neurosci Lett 635:61–66

    Article  Google Scholar 

  26. 26.

    Hochberg LR, Bacher D, Jarosiewicz B, Masse NY, Simeral JD, Vogel J, Haddadin S, Liu J, Cash SS, van der Smagt P, Donoghue JP (2012) Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485(7398):372-U121

    Article  Google Scholar 

  27. 27.

    Allison BZ, Wolpaw EW, Wolpaw AR (2007) Brain-computer interface systems: progress and prospects. Expert Rev Med Dev 4(4):463–474

    Article  Google Scholar 

  28. 28.

    Gu JX, Wang ZH, Kuen J, Ma LY, Shahroudy A, Shuai B, Liu T, Wang XX, Wang G, Cai JF, Chen T (2018) Recent advances in convolutional neural networks. Pattern Recogn 77:354–377

    Article  Google Scholar 

  29. 29.

    Albahar MA (2019) Recurrent neural network model based on a new regularization technique for real-time intrusion detection in SDN environments. Secur Commun Netw, p 9

  30. 30.

    Liang M, Hu XL, IEEE (2015). Recurrent convolutional neural network for object recognition. In: 2015 IEEE conference on computer vision and pattern recognition. IEEE, New York, pp 3367–3375

  31. 31.

    Costa F, Frasconi P, Lombardo V, Soda G (2003) Towards incremental parsing of natural language using recursive neural networks. Appl Intell 19(1–2):9–25

    MATH  Article  Google Scholar 

  32. 32.

    Li Q, Wijesekera O, Salas SJ, Wang JY, Zhu MX, Aprhys C, Chaichana KL, Chesler DA, Zhang H, Smith CL, Guerrero-Cazares H, Levchenko A, Quinones-Hinojosa A (2014) Mesenchymal stem cells from human fat engineered to secrete bmp4 are nononcogenic, suppress brain cancer, and prolong survival. Clin Cancer Res 20(9):2375–2387

    Article  Google Scholar 

  33. 33.

    Daadi MM, Hu SJ, Klausner J, Li ZJ, Sofilos M, Sun GH, Wu JC, Steinberg GK (2013) Imaging neural stem cell graft-induced structural repair in stroke. Cell Transp 22(5):881–892

    Article  Google Scholar 

  34. 34.

    Torres-Valencia C, Alvarez-Lopez M, Orozco-Gutierrez A (2017) SVM-based feature selection methods for emotion recognition from multimodal data. J Multim User Int 11(1):9–23

    Article  Google Scholar 

  35. 35.

    Li WJ, Chu MH, Qiao JF (2019) Design of a hierarchy modular neural network and its application in multimodal emotion recognition. Soft Comput 23(22):11817–11828

    Article  Google Scholar 

  36. 36.

    Gumaei A, Hassan MM, Alelaiwi A, Alsalman H (2019) A hybrid deep learning model for human activity recognition using multimodal body sensing data. IEEE Access 7:99152–99160

    Article  Google Scholar 

  37. 37.

    Ha S, Ahn S, Kim S, Joo Y, Chong YH, Suh YH, Chang KA (2014) In vivo imaging of human adipose-derived stem cells in Alzheimer’s disease animal model. J Biomed Opt 19(5):4

    Google Scholar 

  38. 38.

    Limnuson K, Narayan RK, Chiluwal A, Bouton C, Wang P, Li CY (2016) Development of a brain monitoring system for multimodality investigation in awake rats. In: Patton J, Barbieri R, Ji J et al. (eds), 2016 38th annual international conference of the IEEE engineering in medicine and biology society. IEEE, New York, pp 4487–4490

  39. 39.

    Sander CY, Keil B, Chonde DB, Rosen BR, Catana C, Wald LL (2015) A 31-channel MR brain array coil compatible with positron emission tomography. Magn Reson Med 73(6):2363–2375

    Article  Google Scholar 

  40. 40.

    Jaruszewski KM, Curran GL, Swaminathan SK, Rosenberg JT, Grant SC, Ramakrishnan S, Lowe VJ, Poduslo JF, Kandimalla KK (2014) Multimodal Nanoprobes to target cerebrovascular amyloid in Alzheimer’s disease brain. Biomaterials 35(6):1967–1976

    Article  Google Scholar 

  41. 41.

    Vasquez-Correa JC, Arias-Vergara T, Orozco-Arroyave JR, Eskofier B, Klucken J, Noth E (2019) Multimodal assessment of parkinson’s disease: a deep learning approach. IEEE J Biomed Health Inf 23(4):1618–1630

    Article  Google Scholar 

  42. 42.

    Li C, Sanchez RV, Zurita G, Cerrada V, Cabrera D, Vasquez RE (2015) Multimodal deep support vector classification with homologous features and its application to gearbox fault diagnosis. Neurocomputing 168:119–127

    Article  Google Scholar 

  43. 43.

    Baltrusaitis T, Ahuja C, Morency LP (2019) Multimodal Machine Learning: A Survey and Taxonomy. IEEE Trans Pattern Anal Mach Intell 41(2):423–443

    Article  Google Scholar 

  44. 44.

    Gokmen T, Vlasov Y (2016) Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations. Frontiers in Neuroscience 10:13

    Article  Google Scholar 

  45. 45.

    Al-Qaderi MK, Rad AB (2018) A brain-inspired multi-modal perceptual system for social robots: an experimental realization. IEEE Access 6:35402–35424

    Article  Google Scholar 

  46. 46.

    Zhao LY, Jia KB (2015) Deep feature learning with discrimination mechanism for brain tumor segmentation and diagnosis. IEEE, New York

    Google Scholar 

  47. 47.

    Pashaie R, Anikeeva P, Lee JH, Prakash R, Yizhar O, Prigge M, Chander D, Richner TJ, Williams J (2014) Optogenetic brain interfaces. IEEE Rev Biomed Eng 7:3–30

    Article  Google Scholar 

  48. 48.

    Jiang JM, Fares A, Zhong SH (2019) A context-supported deep learning framework for multimodal brain imaging classification. IEEE Trans Human Mach Syst 49(6):611–622

    Article  Google Scholar 

  49. 49.

    Zhang F, Li Z, Zhang BY, Dua HS, Wang BJ, Zhang XH (2019) Multi-modal deep learning model for auxiliary diagnosis of Alzheimer’s disease. Neurocomputing 361:185–195

    Article  Google Scholar 

  50. 50.

    Yasoubi E, Sedighizadeh M, Siadatan A (2017) Coordinated design of PSS and TCSC controllers using colonal selection algorithm for stability enhancement of dynamical power system. IEEE, New York

    Google Scholar 

  51. 51.

    Hossain MM, Prybutok VR (2016) Towards developing a business performance management model using causal latent semantic analysis. Int J Business Perf Manage 17(2):161–183

    Google Scholar 

  52. 52.

    Waltman L, van Eck NJ, Noyons ECM (2010) A unified approach to mapping and clustering of bibliometric networks. J Inform 4(4):629–635

    Article  Google Scholar 

  53. 53.

    Mahmud M, Kaiser MS, Hussain A, Vassanelli S (2018) Applications of deep learning and reinforcement learning to biological data. IEEE Trans Neural Netw Learn Syst 29(6):2063–2079

    MathSciNet  Article  Google Scholar 

  54. 54.

    Chauveau F, Cho TH, Perez M, Guichardant M, Riou A, Aguettaz P, Picq M, Lagarde M, Berthezene Y, Nighoghossian N, Wiart M (2011) Brain-targeting form of docosahexaenoic acid for experimental stroke treatment: MRI evaluation and anti-oxidant impact. Current Neurovascular Res 8(2):95–102

    Article  Google Scholar 

  55. 55.

    Bondi ML, Di Gesu R, Craparo EF (2012) Lipid nanoparticles for drug targeting to the brain. In: Duzgunes N (ed) Nanomedicine: cancer, diabetes, and cardiovascular, central nervous system, pulmonary and inflammatory diseases. Elsevier Academic Press Inc, San Diego, vol 508, pp 229–251

  56. 56.

    Jiang YY, Li W, Hossain MS, Chen M, Alelaiwi A, Al-Hammadi M (2020) A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition. Inform Fusion 53:209–221

    Article  Google Scholar 

  57. 57.

    Vajari DA, Vomero M, Erhardt JB, Sadr A, Ordonez JS, Coenen VA, Stieglitz T (2018) Integrity assessment of a hybrid DBS probe that enables neurotransmitter detection simultaneously to electrical stimulation and recording. Micromachines 9(10):15

    Google Scholar 

  58. 58.

    Kanade T (1983) IEEE international-conference on computer vision and pattern-recognition. IEEE Commun Mag 21(9):52–52

    Article  Google Scholar 

  59. 59.

    Kurata G, Sethy A, Ramabhadran B, Saon G, A International Speech Communication (2017) Empirical exploration of novel architectures and objectives for language models. In: 18th annual conference of the international speech communication association. Baixas, Isca-Int Speech Communication Assoc, pp 279–283

  60. 60.

    Nurmikko AV, Donoghue JP, Hochberg LR, Patterson WR, Song YK, Bull CW, Shang DA, Zhang HX, Zhu L, Sun JD (2019) Adversarial cross-modal retrieval based on dictionary learning. Neurocomputing 355:93–104

    Article  Google Scholar 

  61. 61.

    Ma C, Luo GN, Wang KQ (2018) Concatenated and connected random forests with multiscale patch driven active contour model for automated brain tumor segmentation of MR images. IEEE Trans Med Imag 37(8):1943–1954

    Article  Google Scholar 

  62. 62.

    Wu F, Lu XY, Song J, Yan SC, Zhang Z, Rui Y, Zhuang YT (2016) Learning of multimodal representations with random walks on the click graph. IEEE Trans Image Process 25(2):630–642

    MathSciNet  MATH  Article  Google Scholar 

  63. 63.

    Yao T, Pan YW, Li YH, Mei T, IEEE (2017) Incorporating copying mechanism in image captioning for learning novel objects. In: 30th IEEE conference on computer vision and pattern recognition. IEEE, New York, pp 5263–5271

  64. 64.

    Xia YJ, Zhang LM, Liu ZG, Nie LQ, Li XL (2017) Weakly supervised multimodal kernel for categorizing aerial photographs. IEEE Trans Image Process 26(8):3748–3758

    MathSciNet  MATH  Article  Google Scholar 

  65. 65.

    Li HB, Sun J, Xu ZB, Chen LM (2017) Multimodal 2D+3D facial expression recognition with deep fusion convolutional neural network. IEEE Trans Multimedia 19(12):2816–2831

    Article  Google Scholar 

  66. 66.

    Zhang H, Wang T, Dai G (2020) Semi-supervised cross-modal common representation learning with vector-valued manifold regularization. Pattern Recogn Lett 130:335–344

    Article  Google Scholar 

Download references


We thank for all the experts who participated in this research and this research is supported by the Natural Science Foundation of China (Grant Number: 51878382).

Author information



Corresponding author

Correspondence to Pin-Chao Liao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Cheng, R. & Liao, PC. Trends of Multimodal Neural Engineering Study: A Bibliometric Review. Arch Computat Methods Eng (2021).

Download citation


  • Neural engineering
  • Multimodal
  • Bibliometric
  • Brain