Advertisement

A review of modularization techniques in artificial neural networks

  • Mohammed AmerEmail author
  • Tomás Maul
Article

Abstract

Artificial neural networks (ANNs) have achieved significant success in tackling classical and modern machine learning problems. As learning problems grow in scale and complexity, and expand into multi-disciplinary territory, a more modular approach for scaling ANNs will be needed. Modular neural networks (MNNs) are neural networks that embody the concepts and principles of modularity. MNNs adopt a large number of different techniques for achieving modularization. Previous surveys of modularization techniques are relatively scarce in their systematic analysis of MNNs, focusing mostly on empirical comparisons and lacking an extensive taxonomical framework. In this review, we aim to establish a solid taxonomy that captures the essential properties and relationships of the different variants of MNNs. Based on an investigation of the different levels at which modularization techniques act, we attempt to provide a universal and systematic framework for theorists studying MNNs, also trying along the way to emphasise the strengths and weaknesses of different modularization approaches in order to highlight good practices for neural network practitioners.

Keywords

Artificial neural network Modularity Architecture Topology Problem decomposition Taxonomy 

Notes

References

  1. Achard S, Bullmore E (2007) Efficiency and cost of economical brain functional networks. PLoS Comput Biol 3(2):0174–0183.  https://doi.org/10.1371/journal.pcbi.0030017 Google Scholar
  2. Aguirre C, Huerta R, Corbacho F, Pascual P (2002) Analysis of biologically inspired small-world networks. In: International conference on artificial neural networks. Springer, pp 27–32Google Scholar
  3. Allen F, Almasi G, Andreoni W, Beece D, Berne BJ, Bright A, Brunheroto J, Cascaval C, Castanos J, Coteus P, Crumley P, Curioni A, Denneau M, Donath W, Eleftheriou M, Flitch B, Fleischer B, Georgiou CJ, Germain R, Giampapa M, Gresh D, Gupta M, Haring R, Ho H, Hochschild P, Hummel S, Jonas T, Lieber D, Martyna G, Maturu K, Moreira J, Newns D, Newton M, Philhower R, Picunko T, Pitera J, Pitman M, Rand R, Royyuru A, Salapura V, Sanomiya A, Shah R, Sham Y, Singh S, Snir M, Suits F, Swetz R, Swope WC, Vishnumurthy N, Ward TJC, Warren H, Zhou R (2001) Blue Gene: a vision for protein science using a petaflop supercomputer. IBM Syst J 40(2):310–327.  https://doi.org/10.1147/sj.402.0310 Google Scholar
  4. Almasri MN, Kaluarachchi JJ (2005) Modular neural networks to predict the nitrate distribution in ground water using the on-ground nitrogen loading and recharge data. Environ Model Softw 20(7):851–871Google Scholar
  5. Aminian M, Aminian F (2007) A modular fault-diagnostic system for analog electronic circuits using neural networks with wavelet transform as a preprocessor. IEEE Trans Instrum Meas 56(5):1546–1554Google Scholar
  6. Anand R, Mehrotra K, Mohan C, Ranka S (1995) Efficient classification for multiclass problems using modular neural networks. IEEE Trans Neural Netw 6(1):117–124.  https://doi.org/10.1109/72.363444 Google Scholar
  7. Anderson A, Shaffer K, Yankov A, Corley CD, Hodas NO (2016) Beyond fine tuning: a modular approach to learning on small data. arXiv:1611.01714v1
  8. Andreas J, Rohrbach M, Darrell T, Klein D (2016a) Neural module networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 39–48Google Scholar
  9. Andreas J, Rohrbach M, Darrell T, Klein D (2016b) Learning to compose neural networks for question answering. arXiv:1601.01705
  10. Angelucci a, Clascá F, Bricolo E, Cramer KS, Sur M (1997) Experimentally induced retinal projections to the ferret auditory thalamus: development of clustered eye-specific patterns in a novel target. J Neurosci Off J Soc Neurosci 17(6):2040–2055Google Scholar
  11. Auda G, Kamel M (1998) Modular neural network classifiers: a comparative study. J Intell Robot Syst 21:117–129.  https://doi.org/10.1023/A:1007925203918 Google Scholar
  12. Auda G, Kamel M (1999) Modular neural networks: a survey. Int J Neural Syst 9(2):129–51Google Scholar
  13. Azam F (2000) Biologically inspired modular neural networks. https://vtechworks.lib.vt.edu/handle/10919/27998. Accessed 23 Dec 2018
  14. Ba J, Caruana R (2014) Do deep nets really need to be deep? In: Advances in neural information processing systems. pp 2654–2662Google Scholar
  15. Babaei S, Geranmayeh A, Seyyedsalehi SA (2010) Protein secondary structure prediction using modular reciprocal bidirectional recurrent neural networks. Comput Methods Programs Biomed 100(3):237–247.  https://doi.org/10.1016/j.cmpb.2010.04.005 Google Scholar
  16. Battaglia PW, Hamrick JB, Bapst V, Sanchez-Gonzalez A, Zambaldi V, Malinowski M, Tacchetti A, Raposo D, Santoro A, Faulkner R, Gulcehre C, Song F, Ballard A, Gilmer J, Dahl G, Vaswani A, Allen K, Nash C, Langston V, Dyer C, Heess N, Wierstra D, Kohli P, Botvinick M, Vinyals O, Li Y, Pascanu R (2018) Relational inductive biases, deep learning, and graph networks. arXiv:1806.01261
  17. Bender G, Kindermans PJ, Zoph B, Vasudevan V, Le Q (2018) Understanding and simplifying one-shot architecture search. http://proceedings.mlr.press/v80/bender18a. Accessed 5 Dec 2018
  18. Bengio Y, Louradour J, Collobert R, Weston J (2009) Curriculum learning. In: Proceedings of the 26th annual international conference on machine learning—ICML ’09. ACM Press, New York, New York, USA, pp 1–8.  https://doi.org/10.1145/1553374.1553380, http://portal.acm.org/citation.cfm?doid=1553374.1553380
  19. Bengio S, Vinyals O, Jaitly N, Shazeer N (2015) Scheduled sampling for sequence prediction with recurrent neural networks. http://papers.nips.cc/paper/5956-scheduled-sampling-for-sequence-prediction-with-recurrent-neural-networks. Accessed 12 Mar 2018
  20. Bhende C, Mishra S, Panigrahi B (2008) Detection and classification of power quality disturbances using S-transform and modular neural network. Electr Power Syst Res 78(1):122–128.  https://doi.org/10.1016/j.epsr.2006.12.011 Google Scholar
  21. Blundell C, Cornebise J, Kavukcuoglu K, Wierstra D (2015) Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424
  22. Bohland JW, Minai AA (2001) Efficient associative memory using small-world architecture. Neurocomputing 38:489–496.  https://doi.org/10.1016/S0925-2312(01)00378-2 Google Scholar
  23. Brandes U, Delling D, Gaertler M, Gorke R, Hoefer M, Nikoloski Z, Wagner D (2008) On modularity clustering. IEEE Trans Knowl Data Eng 20(2):172–188.  https://doi.org/10.1109/TKDE.2007.190689 zbMATHGoogle Scholar
  24. Braylan A, Hollenbeck M, Meyerson E, Miikkulainen R (2015) Reuse of neural modules for general video game playing. arXiv:1512.01537
  25. Bullmore ET, Bassett DS (2011) Brain graphs: graphical models of the human brain connectome. Annu Rev Clin Psychol 7(1):113–140.  https://doi.org/10.1146/annurev-clinpsy-040510-143934 Google Scholar
  26. Bullmore E, Sporns O (2009) Complex brain networks: graph theoretical analysis of structural and functional systems. Nat Rev Neurosci 10(3):186–198.  https://doi.org/10.1038/nrn2575 Google Scholar
  27. Buxhoeveden DP (2002) The minicolumn hypothesis in neuroscience. Brain 125(5):935–951.  https://doi.org/10.1093/brain/awf110 Google Scholar
  28. Caelli T, Guan L, Wen W (1999) Modularity in neural computing. Proc IEEE 87(9):1497–1518.  https://doi.org/10.1109/5.784227 Google Scholar
  29. Calabretta R, Nolfi S, Parisi D, Wagner GP (2000) Duplication of modules facilitates the evolution of functional specialization. Artif Life 6(1):69–84Google Scholar
  30. Chen ZJ, He Y, Rosa-Neto P, Germann J, Evans AC (2008) Revealing modular architecture of human brain structural networks by using cortical thickness from MRI. Cereb Cortex 18(10):2374–2381.  https://doi.org/10.1093/cercor/bhn003 Google Scholar
  31. Chiang CC, Fu HC (1994) A divide-and-conquer methodology for modular supervised neural network design. In: Neural networks, 1994. IEEE world congress on computational intelligence, 1994 IEEE international conference on. IEEE, vol 1, pp 119–124Google Scholar
  32. Chihaoui M, Elkefi A, Bellil W, Ben Amar C (2016) A survey of 2D face recognition techniques. Computers 5(4):21.  https://doi.org/10.3390/computers5040021 Google Scholar
  33. Chollet F (2016) Xception: deep learning with depthwise separable convolutions. arXiv:1610.02357
  34. Chris Tseng H, Almogahed B (2009) Modular neural networks with applications to pattern profiling problems. Neurocomputing 72(10–12):2093–2100.  https://doi.org/10.1016/J.NEUCOM.2008.10.020 Google Scholar
  35. Ciregan D, Meier U, Schmidhuber J (2012) Multi-column deep neural networks for image classification. In: Computer vision and pattern recognition (CVPR), 2012 IEEE conference on. IEEE, pp 3642–3649Google Scholar
  36. Clune J, Mouret JB, Lipson H (2013) The evolutionary origins of modularity. Proc Biol Sci R Soc 280(1755):20122863.  https://doi.org/10.1098/rspb.2012.2863. arXiv:1207.2743v1
  37. de Nardi R, Togelius J, Holland O, Lucas S (2006) Evolution of neural networks for helicopter control: Why modularity matters. In: 2006 IEEE international conference on evolutionary computation. IEEE, pp 1799–1806.  https://doi.org/10.1109/CEC.2006.1688525
  38. Di Ferdinando A, Calabretta R, Parisi D (2001) Evolving modular architectures for neural networks. Proc Sixth Neural Comput Psychol Workshop Evol Learn Dev 12(5):253–262Google Scholar
  39. Douglas RJ, Martin KAC (2007) Recurrent neuronal circuits in the neocortex. Curr Biol CB 17(13):R496–500.  https://doi.org/10.1016/j.cub.2007.04.024 Google Scholar
  40. Eppel S (2017) Hierarchical semantic segmentation using modular convolutional neural networks. arXiv:1710.05126v1
  41. Eyben F, Weninger F, Squartini S, Schuller B (2013) Real-life voice activity detection with LSTM recurrent neural networks and an application to Hollywood movies. In: ICASSP, IEEE international conference on acoustics, speech and signal processing—proceedings, pp 483–487.  https://doi.org/10.1109/ICASSP.2013.6637694
  42. Fernando C, Banarse D, Blundell C, Zwols Y, Ha D, Rusu AA, Pritzel A, Wierstra D (2017) PathNet: evolution channels gradient descent in super neural networks. arXiv:1701.08734
  43. Ferreira MD, Corrêa DC, Nonato LG, de Mello RF (2018) Designing architectures of convolutional neural networks to solve practical problems. Expert Syst Appl 94:205–217.  https://doi.org/10.1016/J.ESWA.2017.10.052 Google Scholar
  44. Franco L, Cannas SA (2001) Generalization properties of modular networks: implementing the parity function. IEEE Trans Neural Netw 12(6):1306–1313.  https://doi.org/10.1109/72.963767 Google Scholar
  45. Freddolino PL, Liu F, Gruebele M, Schulten K (2008) Ten-microsecond molecular dynamics simulation of a fast-folding WW domain. Biophys J 94(10):L75–L77.  https://doi.org/10.1529/biophysj.108.131565 Google Scholar
  46. Fritsch J (1996) Modular neural networks for speech recognition (No. CMU-CS-96-203). Carnegie-Mellon Univ Pittsburgh PA Dept of Computer ScienceGoogle Scholar
  47. Fu HC, Lee YP, Chiang CC, Pao HT (2001) Divide-and-conquer learning and modular perceptron networks. IEEE Trans Neural Netw 12(2):250–263.  https://doi.org/10.1109/72.914522 Google Scholar
  48. Fukushima K, Miyake S, Ito T (1983) Neocognitron: a neural network model for a mechanism of visual pattern recognition. IEEE Trans Syst Man Cybern SMC–13(5):826–834.  https://doi.org/10.1109/TSMC.1983.6313076
  49. Garcia-Pedrajas N, Hervas-Martinez C, Munoz-Perez J (2003) COVNET: a cooperative coevolutionary model for evolving artificial neural networks. IEEE Trans Neural Netw 14(3):575–596.  https://doi.org/10.1109/TNN.2003.810618 Google Scholar
  50. Gilmer J, Schoenholz SS, Riley PF, Vinyals O, Dahl GE (2017) Neural message passing for quantum chemistry. arXiv:1704.01212
  51. Gollisch T, Meister M (2010) Eye smarter than scientists believed: neural computations in circuits of the retina.  https://doi.org/10.1016/j.neuron.2009.12.009
  52. Goltsev A, Gritsenko V (2015) Modular neural networks with radial neural columnar architecture. Biol Inspir Cognit Archit 13:63–74.  https://doi.org/10.1016/J.BICA.2015.06.001 Google Scholar
  53. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. http://papers.nips.cc/paper/5423-generative-adversarial-nets. Accessed 23 Dec 2018
  54. Gradojevic N, Gençay R, Kukolj D (2009) Option pricing with modular neural networks. IEEE Trans Neural Netw Publ IEEE Neural Netw Council 20(4):626–637.  https://doi.org/10.1109/TNN.2008.2011130 Google Scholar
  55. Guan SU, Li S (2002) Parallel growing and training of neural networks using output parallelism. IEEE Trans Neural Netw 13(3):542–550Google Scholar
  56. Happel BLM, Murre JMJ (1994) Design and evolution of modular neural network architectures. Neural Netw 7(6–7):985–1004.  https://doi.org/10.1016/S0893-6080(05)80155-8 Google Scholar
  57. Haykin S (1994) Neural networks: a comprehensive foundation. Prentice Hall PTR, Upper Saddle RiverzbMATHGoogle Scholar
  58. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 770–778Google Scholar
  59. Hidalgo D, Castillo O, Melin P (2009) Type-1 and type-2 fuzzy inference systems as integration methods in modular neural networks for multimodal biometry and its optimization with genetic algorithms. Inf Sci 179(13):2123–2145Google Scholar
  60. Hochreiter S, Urgen Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780.  https://doi.org/10.1162/neco.1997.9.8.1735 Google Scholar
  61. Hu R, Rohrbach M, Andreas J, Darrell T, Saenko K (2016) Modeling relationships in referential expressions with compositional modular networks. arXiv:1611.09978
  62. Huang G-B (2003) Learning capability and storage capacity of two-hidden-layer feedforward networks. IEEE Trans Neural Netw 14(2):274–281.  https://doi.org/10.1109/TNN.2003.809401 Google Scholar
  63. Huang G, Sun Y, Liu Z, Sedra D, Weinberger KQ (2016) Deep networks with stochastic depth. In: European conference on computer vision. Springer, pp 646–661Google Scholar
  64. Huizinga J, Mouret JB, Clune J (2014) Evolving neural networks that are both modular and regular: HyperNeat plus the connection cost technique. Gecco, pp 697–704,  https://doi.org/10.1145/2576768.2598232
  65. Hüsken M, Igel C, Toussaint M (2002) Task-dependent evolution of modularity in neural networks. Connect Sci 14(3):219–229Google Scholar
  66. Jacobs RA, Jordan MI, Nowlan SJ, Hinton GE (1991) Adaptive mixtures of local experts. Neural Comput 3(1):79–87.  https://doi.org/10.1162/neco.1991.3.1.79 Google Scholar
  67. Wei Jiang, Kong Seong G (2007) Block-based neural networks for personalized ECG signal classification. IEEE Trans Neural Netw 18(6):1750–1761.  https://doi.org/10.1109/TNN.2007.900239 Google Scholar
  68. Kacprzyk J, Pedrycz W (2015) Springer handbook of computational intelligence. Springer, BerlinzbMATHGoogle Scholar
  69. Kaiser M, Hilgetag CC (2010) Optimal hierarchical modular topologies for producing limited sustained activation of neural networks. Front Neuroinform 4:8Google Scholar
  70. Karami M, Safabakhsh R, Rahmati M (2013) Modular cellular neural network structure for wave-computing-based image processing. ETRI J 35(2):207–217.  https://doi.org/10.4218/etrij.13.0112.0107 Google Scholar
  71. Kashtan N, Alon U (2005) Spontaneous evolution of modularity and network motifs. Proc Natl Acad Sci USA 102(39):13773–8.  https://doi.org/10.1073/pnas.0503610102 Google Scholar
  72. Kastellakis G, Cai DJ, Mednick SC, Silva AJ, Poirazi P (2015) Synaptic clustering within dendrites: an emerging theory of memory formation.  https://doi.org/10.1016/j.pneurobio.2014.12.002
  73. Kim T, Cha M, Kim H, Lee JK, Kim J (2017) Learning to discover cross-domain relations with generative adversarial networks. arXiv:1703.05192
  74. Larsson G, Maire M, Shakhnarovich G (2016) FractalNet: ultra-deep neural networks without residuals. arXiv:1605.07648
  75. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324.  https://doi.org/10.1109/5.726791 Google Scholar
  76. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436Google Scholar
  77. Lin M, Chen Q, Yan S (2013) Network in network. arXiv preprint p 10,  https://doi.org/10.1109/ASRU.2015.7404828, arXiv:1312.4400
  78. Liu H, Simonyan K, Vinyals O, Fernando C, Kavukcuoglu K (2017) Hierarchical representations for efficient architecture search. arXiv:1711.00436
  79. Lodato S, Arlotta P (2015) Generating neuronal diversity in the mammalian cerebral cortex. Annu Rev Cell Dev Biol 31(1):699–720.  https://doi.org/10.1146/annurev-cellbio-100814-125353
  80. López-Muñoz F, Boya J, Alamo C (2006) Neuron theory, the cornerstone of neuroscience, on the centenary of the Nobel Prize award to Santiago Ramón y Cajal. Brain Res Bull 70(4–6):391–405.  https://doi.org/10.1016/j.brainresbull.2006.07.010 Google Scholar
  81. Melin P, Mancilla A, Lopez M, Mendoza O (2007) A hybrid modular neural network architecture with fuzzy sugeno integration for time series forecasting. Appl Soft Comput 7(4):1217–1226Google Scholar
  82. Melin P, Mendoza O, Castillo O (2011) Face recognition with an improved interval type-2 fuzzy logic sugeno integral and modular neural networks. IEEE Trans Syst Man Cybern Part A Syst Hum 41(5):1001–1012Google Scholar
  83. Mendoza O, Melin P, Licea G (2009a) A hybrid approach for image recognition combining type-2 fuzzy logic, modular neural networks and the Sugeno integral. Inf Sci 179(13):2078–2101.  https://doi.org/10.1016/j.ins.2008.11.018 Google Scholar
  84. Mendoza O, Melín P, Castillo O (2009b) Interval type-2 fuzzy logic and modular neural networks for face recognition applications. Appl Soft Comput 9(4):1377–1387.  https://doi.org/10.1016/j.asoc.2009.06.007 Google Scholar
  85. Meunier D, Lambiotte R, Bullmore ET (2010) Modular and hierarchically modular organization of brain networks. Front Neurosci.  https://doi.org/10.3389/fnins.2010.00200 Google Scholar
  86. Miikkulainen R, Liang J, Meyerson E, Rawal A, Fink D, Francon O, Raju B, Shahrzad H, Navruzyan A, Duffy N, Hodjat B (2017) Evolving deep neural networks. arXiv:1703.00548
  87. Montufar GF, Pascanu R, Cho K, Bengio Y (2014) On the number of linear regions of deep neural networks. http://papers.nips.cc/paper/5422-on-the-number-of-linear-regions-of-deep-neural-networks. Accessed 24 Dec 2018
  88. Moon S-W, Kong S-G (2001) Block-based neural networks. IEEE Trans Neural Netw 12(2):307–317.  https://doi.org/10.1109/72.914525 Google Scholar
  89. Mountcastle VB (1997) The columnar organization of the neocortex. Brain J Neurol.  https://doi.org/10.1093/brain/120.4.701 Google Scholar
  90. Mouret JB, Doncieux S (2009) Evolving modular neural-networks through exaptation. In: 2009 IEEE congress on evolutionary computation, CEC 2009. pp 1570–1577.  https://doi.org/10.1109/CEC.2009.4983129
  91. Mouret JB, Doncieux S (2008) MENNAG: a modular, regular and hierarchical encoding for neural-networks based on attribute grammars. Evolut Intell 1(3):187–207.  https://doi.org/10.1007/s12065-008-0015-7 Google Scholar
  92. Newman MEJ (2004) Detecting community structure in networks. Eur Phys J B 38:321–330.  https://doi.org/10.1140/epjb/e2004-00124-y Google Scholar
  93. Newman MEJ (2006) Modularity and community structure in networks. Proc Natl Acad Sci USA 103(23):8577–82.  https://doi.org/10.1073/pnas.0601602103 Google Scholar
  94. Newman MEJ (2016) Community detection in networks: modularity optimization and maximum likelihood are equivalent. 1:1–8.  https://doi.org/10.1103/PhysRevE.94.052315, arXiv:1606.02319
  95. Oh IS, Suen CY (2002) A class-modular feedforward neural network for handwriting recognition. Pattern Recognit 35(1):229–244.  https://doi.org/10.1016/S0031-3203(00)00181-3 zbMATHGoogle Scholar
  96. Ortín S, Gutiérrez J, Pesquera L, Vasquez H (2005) Nonlinear dynamics extraction for time-delay systems using modular neural networks synchronization and prediction. Physica A Stat Mech Appl 351(1):133–141.  https://doi.org/10.1016/j.physa.2004.12.015 Google Scholar
  97. Ou G, Murphey YL (2007) Multi-class pattern classification using neural networks. Pattern Recognit 40(1):4–18.  https://doi.org/10.1016/j.patcog.2006.04.041 zbMATHGoogle Scholar
  98. Pan P, Xu Z, Yang Y, Wu F, Zhuang Y (2016) Hierarchical recurrent neural encoder for video representation with application to captioning. In: The IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  99. Phan KT, Maul TH, Tuong TV (2015) A parallel circuit approach for improving the speed and generalization properties of neural networks. In: 2015 11th international conference on natural computation (ICNC). IEEE, pp 1–7.  https://doi.org/10.1109/ICNC.2015.7377956
  100. Phan KT, Maul TH, Vu TT, Lai WK (2016) Improving neural network generalization by combining parallel circuits with dropout. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 9949 LNCS, pp 572–580.  https://doi.org/10.1007/978-3-319-46675-0_63, arXiv:1612.04970
  101. Phan KT, Maul TH, Vu TT, Lai WK (2017) Dropcircuit: a modular regularizer for parallel circuit networks. Neural Process Lett 47:1–18Google Scholar
  102. Phaye SSR, Sikka A, Dhall A, Bathula D (2018) Dense and diverse capsule networks: making the capsules learn better. arXiv:1805.04001
  103. Radicchi F, Castellano C, Cecconi F, Loreto V, Parisi D (2004) Defining and identifying communities in networks. Proc Natl Acad Sci USA 101(9):2658–2663Google Scholar
  104. Reisinger J, Stanley KO, Miikkulainen R (2004) Evolving reusable neural modules. In: Genetic and evolutionary computation conference. Springer, pp 69–81Google Scholar
  105. Ronco E, Gawthrop P (1995) Modular neural networks: a state of the art. Rapport Technique CSC-95026, center of system and control, University of Glasgow. http://www.mech.gla.ac.uk/control/report.html
  106. Ronen M, Shabtai Y, Guterman H (2002) Hybrid model building methodology using unsupervised fuzzy clustering and supervised neural networks. Biotechnol Bioeng 77(4):420–429Google Scholar
  107. Rudasi L, Zahorian S (1991) Text-independent talker identification with neural networks. In: [Proceedings] ICASSP 91: 1991 international conference on acoustics, speech, and signal processing. IEEE, vol 1, pp 389–392.  https://doi.org/10.1109/ICASSP.1991.150358
  108. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. http://papers.nips.cc/paper/6975-dynamic-routing-between-capsules. Accessed 28 Feb 2018
  109. San PP, Ling SH, Nguyen HT (2011) Block based neural network for hypoglycemia detection. In: 2011 annual international conference of the IEEE engineering in medicine and biology society. IEEE, pp 5666–5669.  https://doi.org/10.1109/IEMBS.2011.6091371
  110. Santoro A, Raposo D, Barrett DGT, Malinowski M, Pascanu R, Battaglia P, Lillicrap T (2017) A simple neural network module for relational reasoning. arXiv:1706.01427
  111. Schwarz AJ, Gozzi A, Bifone A (2008) Community structure and modularity in networks of correlated brain activity. Magn Reson Imaging 26(7):914–920.  https://doi.org/10.1016/j.mri.2008.01.048
  112. Serban IV, Sordoni A, Bengio Y, Courville A, Pineau J (2016) Building end-to-end dialogue systems using generative hierarchical neural network models. AAAI p 8.  https://doi.org/10.1017/CBO9781107415324.004, arXiv:1507.04808
  113. Sharkey AJC (1996) On combining artificial neural nets. Connect Sci 8(3–4):299–313.  https://doi.org/10.1080/095400996116785 Google Scholar
  114. Shetty R, Laaksonen J (2015) Video captioning with recurrent networks based on frame- and video-level features and visual content classification. arXiv:1512.02949
  115. Singh S, Hoiem D, Forsyth D (2016) Swapout: Learning an ensemble of deep architectures. In: Advances in neural information processing systems. pp 28–36Google Scholar
  116. Song L, Zhang Y, Wang Z, Gildea D (2018) A graph-to-sequence model for AMR-to-text generation. arXiv:1805.02473
  117. Soutner D, Müller L (2013) Application of lstm neural networks in language modelling. In: International conference on text, speech and dialogue. Springer, pp 105–112Google Scholar
  118. Sporns O (2011) The human connectome: a complex network. Ann N Y Acad Sci.  https://doi.org/10.1111/j.1749-6632.2010.05888.x Google Scholar
  119. Sporns O, Zwi JD (2004) The small world of the cerebral cortex. Neuroinformatics 2(2):145–162.  https://doi.org/10.1385/NI:2:2:145 Google Scholar
  120. Srivastava RK, Masci J, Kazerounian S, Gomez F, Schmidhuber J (2013) Compete to compute. Nips pp 2310–2318Google Scholar
  121. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958.  https://doi.org/10.1214/12-AOS1000. arXiv:1102.4807 MathSciNetzbMATHGoogle Scholar
  122. Srivastava RK, Greff K, Schmidhuber J (2015) Highway networks. arXiv:1505.00387 [cs]
  123. Stanley KO, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evolut Comput 10(2):99–127.  https://doi.org/10.1162/106365602320169811 Google Scholar
  124. Stanley KO, D’Ambrosio DB, Gauci J (2009) A hypercube-based encoding for evolving large-scale neural networks. Artif Life 15(2):185–212.  https://doi.org/10.1162/artl.2009.15.2.15202 Google Scholar
  125. Stollenga MF, Byeon W, Liwicki M, Schmidhuber J (2015) Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems, vol 28. Curran Associates Inc, Red Hook, pp 2998–3006Google Scholar
  126. Subirats JL, Jerez JM, Gómez I, Franco L (2010) Multiclass pattern recognition extension for the new C-Mantec constructive neural network algorithm. Cognit Comput 2(4):285–290.  https://doi.org/10.1007/s12559-010-9051-6 Google Scholar
  127. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. vol 07-12-June, pp 1–9.  https://doi.org/10.1109/CVPR.2015.7298594, arXiv:1409.4842
  128. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 2818–2826Google Scholar
  129. Terekhov AV, Montone G, O’Regan JK (2015) Knowledge transfer in deep block-modular neural networks. Springer, Cham, pp 268–279.  https://doi.org/10.1007/978-3-319-22979-9_27
  130. Tyler JR, Wilkinson DM, Huberman BA (2005) E-Mail as spectroscopy: automated discovery of community structure within organizations. Inf Soc 21(2):143–153.  https://doi.org/10.1080/01972240590925348
  131. Veit A, Wilber MJ, Belongie S (2016) Residual networks behave like ensembles of relatively shallow networks. In: Advances in neural information processing systems. pp 550–558Google Scholar
  132. Verbancsics P, Stanley KO (2011) Constraining connectivity to encourage modularity in HyperNEAT. In: Proceedings of the 13th annual conference on Genetic and evolutionary computation—GECCO ’11. p 1483.  https://doi.org/10.1145/2001576.2001776
  133. Vlahogianni EI, Karlaftis MG, Golias JC (2007) Spatio-temporal short-term urban traffic volume forecasting using genetically optimized modular networks. Comput Aided Civ Infrastruct Eng 22(5):317–325Google Scholar
  134. Waibel A (1989) Modular construction of time-delay neural networks for speech recognition. Neural Comput 1(1):39–46.  https://doi.org/10.1162/neco.1989.1.1.39 Google Scholar
  135. Wang M (2015) Multi-path convolutional neural networks for complex image classification. arXiv:1506.04701
  136. Wang SJ, Hilgetag CC, Zhou C (2011) Sustained activity in hierarchical modular neural networks: self-organized criticality and oscillations. Front Comput Neurosci 5:30Google Scholar
  137. Wang T, Wu DJ, Coates A, Ng AY (2012) End-to-end text recognition with convolutional neural networks. In: Pattern recognition (ICPR), 2012 21st international conference on. IEEE, pp 3304–3308Google Scholar
  138. Watanabe C, Hiramatsu K, Kashino K (2018) Modular representation of layered neural networks. Neural Netw 97:62–73.  https://doi.org/10.1016/J.NEUNET.2017.09.017 Google Scholar
  139. Watts DJ (1999) Networks, dynamics, and the smallworld phenomenon. Am J Sociol 105(2):493–527.  https://doi.org/10.1086/210318
  140. Weston J, Chopra S, Bordes A (2014) Memory networks. arXiv:1410.3916
  141. Xie S, Girshick R, Dollár P, Tu Z, He K (2016) Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431
  142. Xu L, Krzyzak A, Suen C (1992) Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Trans Syst Man Cybern 22(3):418–435.  https://doi.org/10.1109/21.155943 Google Scholar
  143. Yu L, Lin Z, Shen X, Yang J, Lu X, Bansal M, Berg TL (2018) MAttNet: modular attention network for referring expression comprehension. arXiv:1801.08186v2
  144. Yu H, Wang J, Huang Z, Yang Y, Xu W (2016) Video paragraph captioning using hierarchical recurrent neural networks. In: The IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  145. Zhang N, Donahue J, Girshick R, Darrell T (2014) Part-based R-CNNs for fine-grained category detection. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). LNCS, vol 8689 pp 834–849.  https://doi.org/10.1007/978-3-319-10590-1_54, arXiv:1407.3867
  146. Zhang F, Leitner J, Milford M, Corke P (2016) Modular deep Q networks for sim-to-real transfer of visuo-motor policies. arXiv:1610.06781v4
  147. Zhang C, Ren M, Urtasun R (2018) Graph HyperNetworks for neural architecture search. arXiv:1810.05749
  148. Zheng W, Lee DH, Shi Q (2006) Short-term freeway traffic flow prediction: Bayesian combined neural network approach. J Transp Eng 132(2):114–121.  https://doi.org/10.1061/(ASCE)0733-947X(2006)132:2(114)

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.School of Computer ScienceUniversity of Nottingham Malaysia CampusSemenyihMalaysia

Personalised recommendations