Adaptive learning rule for hardware-based deep neural networks using electronic synapse devices
- 601 Downloads
In this paper, we propose a learning rule based on a back-propagation (BP) algorithm that can be applied to a hardware-based deep neural network using electronic devices that exhibit discrete and limited conductance characteristics. This adaptive learning rule, which enables forward, backward propagation, as well as weight updates in hardware, is helpful during the implementation of power-efficient and high-speed deep neural networks. In simulations using a three-layer perceptron network, we evaluate the learning performance according to various conductance responses of electronic synapse devices and weight-updating methods. It is shown that the learning accuracy is comparable to that obtained when using a software-based BP algorithm when the electronic synapse device has a linear conductance response with a high dynamic range. Furthermore, the proposed unidirectional weight-updating method is suitable for electronic synapse devices which have nonlinear and finite conductance responses. Because this weight-updating method can compensate the demerit of asymmetric weight updates, we can obtain better accuracy compared to other methods. This adaptive learning rule, which can be applied to full hardware implementation, can also compensate the degradation of learning accuracy due to the probable device-to-device variation in an actual electronic synapse device.
KeywordsDeep neural networks (DNNs) Back-propagation Neuromorphic Synapse device Hardware-based deep neural networks (HW-DNNs) Classification accuracy
This work was supported by the Korea Institute of Science and Technology (KIST) Institutional Program (Project No. 2E27810-18-P040) and the Brain Korea 21 Plus Project in 2018.
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
- 2.Suri M, Bichler O, Querlioz D, Cueto O, Perniola L, Sousa V, Vuillaume D, Gamrat C, DeSalvo B (2011) Phase change memory as synapse for ultra-dense neuromorphic systems: application to complex visual pattern extraction. IEEE Electron Devices Meeting. https://doi.org/10.1109/IEDM.2011.6131488 CrossRefGoogle Scholar
- 13.Bi G-q, Poo M-m (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci 18:10464. https://www.ncbi.nlm.nih.gov/pubmed/9852584
- 15.Burr GW, Shelby RM, di Nolfo C, Jang J-W, Shenoy RS, Narayanan P, Virwani K, Giacometti EU, Kurdi B, Hwang H (2014) Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element. IEEE Electron Devices Meeting. https://doi.org/10.1109/iedm.2014.7047135 CrossRefGoogle Scholar
- 19.Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. In: Parallel distributed processing: explorations in macrostructure of cognition, vol I. Badford, Cambridge. https://dl.acm.org/citation.cfm?id=104293
- 20.Krizhevsky A, Sutskever I, and Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1–9. https://dl.acm.org/citation.cfm?id=2999257
- 21.He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. CoRR arXiv:abs/1512.03385
- 22.Cho K, van Merrienboer B, Gülçehre Ç, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. arXiv:1406.1078
- 35.Binas J, Neil D, Indiveri G, Liu S-C, Pfeiffer M (2016) Precise deep neural network computation on imprecise low-power analog hardware. arXiv preprint. arXiv:1606.07786
- 36.Schiffmann W, Joost M, Werner R (1994) Optimization of the backpropagation algorithm for training multilayer perceptrons. Technical report, University of Koblenz, Institute of Physics, Rheinau. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.53.6869&rep=rep1&type=pdf
- 38.MATLAB and Statistics Toolbox Release 2016a. The MathWorks, Inc., NatickGoogle Scholar
- 42.Bengio Y, Lamblin P, Popovici D, Larochelle H (2006) Greedy layer-wise training of deep networks. Adv Neural Inf Process Syst 153–160. https://dl.acm.org/citation.cfm?id=2976476
- 43.Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. The J Mach Learn Res 15:1929–1958. http://dl.acm.org/citation.cfm?id=2670313