Neuromorphic Systems Engineering pp 339-365 | Cite as

# Neuromorphic Synapses for Artificial Dendrites

## Abstract

In the past few years several researchers have developed general-purpose spiking silicon neurons, or neuromorphs, which attempt to capture various functional behaviors of biological neurons [19, 23, 24]. These CMOS analog VLSI implementations are intended to be the computational elements linking sensory input to motor output in artificial nervous systems. Systems consisting of thousands or millions of neuromorphs may be required to realize certain basic behaviors. However, until appropriate methods are found to specify and adapt network parameters (*e.g.*, connectivity, synapse weights, dynamics, morphology), these systems will remain beyond our grasp. Though various Hebbian or correlative adaptation rules have been applied to perceptron based networks (*e.g.*, [77, 11, 141]) and single-compartment model neurons [12, 26], these rules are virtually untested with multi-compartment spiking neuromorphs that model the spatially extensive dendrites found in biological systems. This paper discusses the VLSI design and performance of just one of the adaptive network parameters: variable synaptic conductances that mimic the fast chemical synapses found on dendrites throughout high-order nervous systems.

## Keywords

Synaptic Weight Hebbian Learning Synaptic Conductance Analog VLSI Synapse Activation## Preview

Unable to display preview. Download preview PDF.

## References

- [1]C. Acar and M. S. Ghausi. Fully integrated active-rc filters using MOS and non-balanced structure.
*Int. J. of Circuit Theory and Applications*, 15:105–121, 1987.CrossRefGoogle Scholar - [2]C. Allen and C. F. Stevens. An evaluation of causes for unreliability of synapt ic transmission. In
*Proc. Natl. Acad. Sci. USA*, volume 91, pages 10380–10383, 1994.CrossRefGoogle Scholar - [3]P. O. Anderson. Properties of hippocampal synapses of importance for integration and memory. In G. M. Edelman, W. E. Gall, and W. M. Cowan, editors,
*Synaptic Function*, pages 403–430. John Wiley & Sons, 1987.Google Scholar - [4]Y. Arima, M. Murasaki, T. Yamada, A. Maeda, and H. Shinohara. A refreshable analog VLSI neural network chip with 400 neurons and 40k synapses.
*IEEE J. of Solid State Circuits*, 27:1854–1861, 1992.CrossRefGoogle Scholar - [5]M. Banu and Y. Tsividis. Floating voltage-controlled resistors in CMOS technology.
*Electron. Lett.*, 18:678–679, 1982.CrossRefGoogle Scholar - [6]J. N. Barrett and W. E. Crill. Specific membrane properties of cat motoneurones.
*J. of Physiol.*, 239:301–324, 74.Google Scholar - [7]M. Bawdry and J. L. Davis.
*Long-Term Potentiation*, volume 2. The MIT Press, Cambridge, MA, 1994.Google Scholar - [8]J. M. Bekkers. Quantal analysis of synaptic transmission in the central nervous system.
*Current Opinion in Neurobiology*, 4:360–365, 1994.CrossRefGoogle Scholar - [9]T. V. P. Bliss and G. L. Collingridge. A synaptic model of memory: Long-term potentiation in the hippocampus.
*Nature*, 361:31–39, 1993.CrossRefGoogle Scholar - [10]K. A. Boahen and A. G. Andreou. A contrast sensitive silicon retina with reciprocal synapses.
*Advances in Neural Information Processing Systems*, 4:764–772, 1992.Google Scholar - [11]T. H. Brown, E. W. Kairiss, and C. L. Keenan. Hebbian synapses: Biophysical mechanisms and algorithms.
*Annu. Rev. of Neurosci.*, 13:475–511, 1990.CrossRefGoogle Scholar - [12]D. V. Buonomano and M. M. Merzenich. Temporal information transformed into a spatial code by a neural network with realistic properties.
*Science*, 267:1028–1030, 1995.CrossRefGoogle Scholar - [13]H. A. Castro, S. M. Tam, and M. A. Holler. Implementation and preformance of an analog nonvolatile neural network.
*Analog Integrated Circuits and Signal Processing*, 4:97–113, 1993.CrossRefGoogle Scholar - [14]J. D. Clements and S. J. Redman. Cable properties of cat spinal motoneurons measured by combining voltage clamp, current clamp and intracellular staining.
*J. of Physiol.*, 409:63–87, 1989.Google Scholar - [15]M. H. Cohen and A. G. Andreou. Current-node subthreshold MOS implementation of the herault-jutten autoadaptive network.
*IEEE J. of Solid State Circuits*, 27:714–727, 1992.CrossRefGoogle Scholar - [16]G. L. Collingridge and T. V. P. Bliss. Memories of nmda receptors and ltp.
*Trends in Neurosci.*, 18:54–56, 1995.CrossRefGoogle Scholar - [17]T. Delbrück. Silicon retina with correlation-based velocity-tuned pixels.
*IEEE Transactions on Neural Networks*, 4(3):529–541, May 1993.CrossRefGoogle Scholar - [18]J. Van der Spiegel, P. Mueller, D. Blackman, P. Chance, C. Donham, R. Etienne-Cummings, and P. Kinget. An analog neural computer with modular architecture for real-time dynamic computations.
*IEEE J. of Solid-State Circuits*, 27:82–92, 1992.CrossRefGoogle Scholar - [19]R. Douglas and M. Mahowald. A constructor set for silicon neurons. In S. F. Zornetzer, J. Davis, and T. McKenna, editors,
*An Introduction to Neural and Electronic Networks*. Academic Press, 1994.Google Scholar - [20]S. Eberhardt, T. Duong, and A. Thakoor. Design of parallel hardware neural network systems from custom analog VLSI ‘building blocks’ chips. In
*Int. Jt. Conf. Neural Network*, volume 2, pages 183–190, 1988.Google Scholar - [21]G. M. Edelman, W. E. Gall, and W. M. Cowan.
*Synaptic Function*. John Wiley & Sons, 1987.Google Scholar - [22]F. A. Edwards. Ltp-a structural model to explain the inconsistencies.
*Trends in Neurosci.*, 18:250–255, 1995.CrossRefGoogle Scholar - [23]J. G. Elias. Artificial dendritic trees.
*Neural Computation*, 5:648–663, 1993.Google Scholar - [24]J. G. Elias and D. P. M. Northmore. Switched-capacitor neuromorphs with wide-range variable dynamics.
*IEEE Trans. on Neural Networks*, 6:1542–1548, 1995.CrossRefGoogle Scholar - [25]J. G. Elias, D. P. M. Northmore, and W. Westerman. An analog memory device for spiking silicon neurons.
*Neural Computation*, 9:419–440, 1997.CrossRefGoogle Scholar - [26]W. Gerstner, R. Ritz, and J. Leo van Hemmen. Why spikes? hebbian learning and retrieval of time-resolved excitation patterns.
*Biol. Cybern.*, 69:503–515, 1993.MATHGoogle Scholar - [27]H. P. Graf, E. Sackinger, and L. D. Jackel. Recent developments of electronic neural nets in north america.
*J. of VLSI Signal Processing*, 5:19–31, 1993.CrossRefGoogle Scholar - [28]P. Hasler, C. Diorio, B. A. Minch, and C. Mead. Single transistor learning synapses with long term storage. In
*IEEE Intl. Symp. on Circuits and Systems*, volume 3, pages 1660–1663, 1995.Google Scholar - [29]N. A. Hessler, A. M. Shirke, and R. Malinow. The probability of transmitter release at a mammalian central synapse.
*Nature*, 366:569–573, 1993.CrossRefGoogle Scholar - [30]B. Hille.
*Ionic Channels of Excitable Membranes*. Sinauer Associates Inc., 2 edition, 1992.Google Scholar - [31]Y. Hirai. Recent VLSI neural networks in japan.
*J. of VLSI Signal Processing*, 6:7–18, 1993.CrossRefGoogle Scholar - [32]A. J. Holmes, R. A. G. Gibson, J. Hajto, A. F. Murray, A. E. Owen, M. J. Rose, and A. J. Snell. Use of a-si:h memory devices for non-volatile weight storage in artificial neural networks.
*J. of Non-Crystalline Solids*, 164–166:817–820, 1993.CrossRefGoogle Scholar - [33]T. M. Jessell and E. R. Kandel. Synaptic transmission: A bidirectional and self-modifiable form of cell-cell comunication.
*Cell*, 72:1–30, 1993. Neuron Vol. 10 (Suppl.) pp. 1–30.CrossRefGoogle Scholar - [34]E. R. Kandel. Part III: Elementary interactions between neurons: Synaptic transmission. In J. H. Schwartz and T. M. Jessell, editors,
*Principles of Neural Science*, pages 123–260. Appleton and Lange, Norwalk, Connecticut, 3 edition, 1991.Google Scholar - [35]E. R. Kandel, M. Klein, B. Hochner, M. Shuster, S. A. Siegelbaum, R. D. Hawkins, D. L. Glanzman, V. F. Castellucci, and T. W. Abrams. Synaptic modulation and learning: New insights into synaptic transmission from the study of behavior. In G. M. Edelman, W. E. Gall, and W. M. Cowan, editors,
*Synaptic Function*, pages 471–518. John Wiley & Sons, 1987.Google Scholar - [36]C. Koch and T. Poggio. The biophysical properties of spines as a basis for their electrical function: A comment on kawato and tsakahara (1983).
*J. Theor. Biol.*, 113:225–230, 1985.CrossRefGoogle Scholar - [37]C. Koch and T. Poggio. Biophysics of computation: Neurons, synapses, and membranes. In G. M. Edelman, W. E. Gall, and W. M. Cowan, editors,
*Synaptic Function*, pages 637–698. ohn Wiley & Sons, 1987.Google Scholar - [38]C. Koch and T. Poggio. Multiplying with synapses and neurons. In T. McKenna, J. Davis, and S. F. Zornetzer, editors,
*Single Neuron Computation*, pages 315–346. Academic Press, Inc., 1992.Google Scholar - [39]H. Korn and D. S. Faber. Regulation and significance of probablistic release mechanisms at central synapses. In G. M. Edelman, W. E. Gall, and W. M. Cowan, editors,
*Synaptic Function*, pages 57–108. John Wiley & Sons, 1987.Google Scholar - [40]H. Korn and A. Mallet. Transformation of binomial input by the postsynaptic membrane at a central synapse.
*Science*, 225:1157–1159, 1984.CrossRefGoogle Scholar - [41]D. M. Kullmann and R. A. Nicoll. Long-term potentiation is associated with increases in quantal content and quantal amplitude.
*Nature*, 357:240–244, 1992.CrossRefGoogle Scholar - [42]A. Larkman, T. Hannay, K. Stratford, and J. Jack. Presynaptic release probability influences the locus of long-term potentiation.
*Nature*, 360:70–73, 1992.CrossRefGoogle Scholar - [43]J. P. Lazzaro and C. Mead. Circuit models of sensory transduction in the cochlea. In Mead and Ismail, editors,
*Analog VLSI Implementation of Neural Systems*, pages 85–101. Kluwer Academic Publishers, Norwell, MA, 1989.Google Scholar - [44]B. W. Lee, B. J. Sheu, and H. Yang. Analog floating-gate synapses for general-purpose VLSI neural computation.
*IEEE Trans. on Circuits and Systems*, 38:654–658, 1991.CrossRefGoogle Scholar - [45]B. W. Lee and B. J. Sheu. General-purpose neural chips with electrically programmable synapses and gain-adjustable neurons.
*IEEE J. of Solid-State Circuits*, 27:1299–1302, 1992.CrossRefGoogle Scholar - [46]D. Liao, N. A. Hessler, and R. Malinow. Activation of post synaptically silent synapses during pairing-induced ltp in ca1 region of hippocampal slice.
*Nature*, 375:400–404, 1995.CrossRefGoogle Scholar - [47]B. Linares-Barranco, E. Sánchez-Sinencio, A. Rodriguez-Vazquez, and J. L. Huertas. A CMOS implementation of fitzhugh-nagumo neuron model.
*IEEE Solid-State Circuits*, 26(7):956–965, 1991.CrossRefGoogle Scholar - [48]B. Linares-Barranco, E. Sánchez-Sinencio, A. Rodriguez-Vazquez, and J. L. Huertas. A CMOS analog adaptive bam with on-chip learning and weight refreshing.
*IEEE Trans. on Neural Networks*, 4:445–457, 1993.CrossRefGoogle Scholar - [49]P. J. Mackenzie, M. Umemiya, and T. H. Murphy. a2+ imaging of cns axons in culture indicates reliable coupling between single action potentials and distal functional release sites.
*Neuron*, 16:783–795, 1996.CrossRefGoogle Scholar - [50]K. L. Magleby. Short-term changes in synaptic efficacy. In G. M. Edelman, W. E. Gall, and W. M. Cowan, editors,
*Synaptic Function*, pages 21–56. John Wiley & Sons, 1987.Google Scholar - [51]M. Mahowald.
*VLSI Analogs of Neuronal Visual Processing: a Synthesis of Form and Function*. Computation and neural systems, California Institute of Technology, 1992.Google Scholar - [52]M. Mahowald and R. Douglas. A silicon neuron.
*Nature*, 354:515–518, 1991.CrossRefGoogle Scholar - [53]G. Major, A. U. Larkman, P. Jones, B. Sakmann, and J. J. B. Jack. Detailed passive cable models of whole-cell recorded ca3 pyramidal neurons in rat hippocampal slices.
*J. of Neurosci.*, 14(8):4613–4638, 1994.Google Scholar - [54]R. C. Malenka, J. A. Kauer, D. J. Perkel, M. D. Mauk, P. T. Kelly, R. A. Nicoll, and M. N. Waxham. An essential role for postsynaptic calmodulin and protein kinase activity in long-term potentiation.
*Nature*, 340:554–557, 1989.CrossRefGoogle Scholar - [55]R. C. Malenka, J. A. Kauer, R. S. Zucker, and R. A. Nicoll. Postsynaptic calcium is sufficient for potentiation of hippocampal synaptic transmission.
*Science*, 242:81–84, 1988.CrossRefGoogle Scholar - [56]A. Malgaroli, A. E. Ting, B. Wendland, A. Bergamaschi, A. Villa, R.W. Tsien, and R.H. Scheller. Presynaptic component of long-term potentiation visualized at individual hippocampal synapses.
*Science*, 268:1624–1628, 1995.CrossRefGoogle Scholar - [57]J. L. Martinez and B. E. Derrick. Long-term potentiation and learning.
*Annu. Rev. of Psychology*, 47:173–203, 1996.CrossRefGoogle Scholar - [58]C. A. Mead.
*Analog VLSI and Neural Systems*. Addison-Wesley, Reading, MA, 1989.MATHGoogle Scholar - [59]C. A. Mead and M. A. Mahowald. Silicon model of early visual processing.
*Neural Networks*, 1:91–97, 1988.CrossRefGoogle Scholar - [60]B. W. Mel. Information processing in dendritic trees.
*Neural Computation*, 6:1031–1085, 1994.MATHGoogle Scholar - [61]G. Moon, M. E. Zaghloul, and R. W. Newcomb. An enhancement-mode MOS voltage-controlled linear resistor with large dynamic range.
*IEEE Trans. on Circuits and Systems*, 37:1284–1288, 1990.CrossRefGoogle Scholar - [62]G. Moon, M. E. Zaghloul, and R. W. Newcomb. VLSI implementation of synaptic weighting and summing in pulse coded neural-type cells.
*EEE Trans. Neural Networks*, 3:394–403, 1992.CrossRefGoogle Scholar - [63]A. Mortara and E. A. Vittoz. A communication architecture tailored for analog VLSI neural networks: Intrinsic performance and limitations.
*IEEE Transactions on Neural Networks*, TNN-5(3):459–466, May 1994.CrossRefGoogle Scholar - [64]P. Mueller, J. V. D. Spiegel, D. Blackman, T. Chiu, T. Clare, J. Dao, C. Donham, T. Hsieh, and M. Loinaz. A general purpose analog neural computer. In
*Int. Jt. Conf. Neural Network*, volume 2, pages 177–182, 1989.CrossRefGoogle Scholar - [65]T. H. Murphy, J. M. Baraban, W. G. Weir, and L. A. Blatter. Visualization of quantal synaptic transmission by dendritic calcium imaging.
*Science*, 263:529–532, 1994.CrossRefGoogle Scholar - [66]A. F. Murray and P. J. Edwards. Enhanced mlp performance and fault tolerance resulting from synaptic weight noise during training.
*IEEE Trans. on Neural Networks*, 5:792–802, 1994.CrossRefGoogle Scholar - [67]A. F. Murray and L. Tarassenko.
*Analogue Neural VLSI: A Pulse Stream Approach*. Chapman and Hall, London, England, 1994.Google Scholar - [68]A.F. Murray, D. Del Corso, and L. Tarassenko. Pulse-stream VLSI neural networks mixing analog and digital techniques.
*IEEE Transactions on Neural Networks*, 2:193–204, 1991.CrossRefGoogle Scholar - [69]D. P. M. Northmore and J. G. Elias. Spike train processing by a silicon neuromorph: The role of sublinear summation in dendrites.
*Neural Computation*, 8:1245–1265, 1996.CrossRefGoogle Scholar - [70]N. Otmakhov, A. M. Shrike, and R. Malinow. Measuring the impact of probabilistic transmission on neuronal output.
*Neuron*, 10:1101–1111, 1993.CrossRefGoogle Scholar - [71]T. Poggio and V. Torre. A new approach to synaptic interactions. In H. Heim and G. Palm, editors,
*Lecture Notes in Biomathematics. Theorectical Approaches to Computer Systems*, volume 2, pages 89–115. Heidelberg, New York, 1977.Google Scholar - [72]W. Rall. Theoretical significance of dendritic trees for neuronal input-output relations. In R. F. Reiss, editor,
*Neural Theory and Modeling*. Stanford University Press, Palo Alto, 1964.Google Scholar - [73]W. Rall. Distinguishing theoretical synaptic potentials computed for different soma-dendritic distributions of synaptic inputs.
*J. of Neurophys.*, 30:1138–1168, 1967.Google Scholar - [74]M. Rapp, I. Segev, and Y. Yarom. Physiology, morphology and detailed passived models of cerebellar purkinje cell.
*J. of Physiol*, 474:101–118, 1994.Google Scholar - [75]M. J. Rose, J. Hajto, P. G. Lecomber, S. M. Gage, W. K. Choi, A. J. Snell, and A. E. Owen. Amorphous silicon analogue memory devices.
*J. of Non-Crystalline Solids*, 115:168–170, 1989.CrossRefGoogle Scholar - [76]D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. In D. E. Rumelhart, J. L. McClelland, and the PDP Research Group, editors,
*Parallel Distributed Processing: Explorations in the Microstructures of Cognition*, volume I: Foundations. MIT Press/Bradford Books, Cambridge, MA, 1986.Google Scholar - [77]I. Segev, J. W. Fleshman, and R. E. Burke. Compartmental models of complex neurons. In C. Koch and I. Segev, editors,
*Methods in Neuronal Modeling: From Synapses to Networks*, pages 63–96. The MIT Press, 1989.Google Scholar - [78]D. P. Shelton. Membrane resistivity estimated for the purkinje neuron by means of a passive computer model.
*Neuroscience*, 14:111–131, 1985.CrossRefGoogle Scholar - [79]G. M. Shepherd. Canonical neurons and their computational organization. In J. Davis and S. F. Zornetzer, editors,
*Single Neuron Computation*, pages 27–60. Academic Press, Inc., 1992.Google Scholar - [80]G. M. Shepherd and C. Koch. Introduction to synaptic circuits. In G.M. Shepherd, editor,
*The Synaptic Organization of the Brain*, pages 3–31. Oxford University Press, New York, 1990.Google Scholar - [81]T. Shibata and T. Ohmi. A functional MOS transistor featuring gate-level weighted sum and threshold operations.
*IEEE Trans. on Electron Devices*, 39:1444–1455, 1992.CrossRefGoogle Scholar - [82]T. Shima, T. Kimura, Y. Kamatani, T. Itakura, Y. Fujita, and T. Iida. Neuro chips with on-chip back-propagation and/or hebbian learning.
*IEEE J. of Solid-State Circuits*, 27(12):1868–1875, 1992.CrossRefGoogle Scholar - [83]C. F. Stevens and Y. Wang. Changes in reliability of synaptic function as a mec hanism for plasticity.
*Nature*, 371:704–707, 1994.CrossRefGoogle Scholar - [84]K. J. Stratford, A. J. R. Mason, A. U. Larkman, G. Major, and J. J. B. Jack. The modelling of pyramidal neurones in the visual cortex. In R. Durbing, C. Miall, and G. Mitchison, editors,
*The computing neurone*, pages 296–321. Addison-Wesley, 1989.Google Scholar - [85]D. Thurbon, A. Field, and S. J. Redman. Electrotonic profiles of interneurones in stratum pyramidale of the ca1 region of rat hippocampus.
*J. of Neurophys.*, 71:1948–1958, 1994.Google Scholar - [86]J. Tomberg. Synchronous pulse density modulation in neural network implementation. In M. E. Zaghloul, J. L. Meador, and R. W. Newcomb, editors,
*Silicon Implementation of Pulse Coded Neural Networks*, pages 165–198. Kluwer Academic Publishers, 1994.Google Scholar - [87]K. Y. Tsai, N. T. Carnevale, and T. H. Brown. Hebbian learning is jointly controlled by electronic and input structure.
*Network*, 5:1–19, 1994.CrossRefGoogle Scholar - [88]Y. P. Tsividis and D. Anastassiou. Switched-capacitor neural networks.
*Electron. Lett.*, 23:958–959, 1988.CrossRefGoogle Scholar - [89]Z. Wang. Novel electronically-controlled floating resistors using MOS transistors operating in saturation.
*Electron. Lett.*, 27:188–189, 1991.CrossRefGoogle Scholar - [90]L. Watts, D. Kerns, R. F. Lyon, and C. Mead. Improved implementation of the silicon cochlea.
*IEEE Journal Solid-State Circuits*, 27(5):692–700, May 1992.CrossRefGoogle Scholar - [91]A. Zador, C. Koch, and T. H. Brown. Biophysical model of a hebbian synapse. In
*Proc. Natl. Acad. Sci. U.S.A.*, pages 6718–6722, 1990.Google Scholar - [92]M. E. Zaghloul.
*Silicon Implementation of Pulse Coded Neural Networks*. Kluwer Academic Publishers, 1994.Google Scholar