Abstract
The human brain constantly learns via mutiple different learning strategies. It can learn by simply having stimuli being presented to its sensory organs which is considered unsupervised learning. In addition, it can learn associations between inputs and outputs when a teacher provides the output which is considered as supervised learning. Most importantly, it can learn very efficiently if correct behaviour is followed by reward and/or incorrect behaviour is followed by punishment which is considered reinforcement learning. So far, most artificial neural architectures implement only one of the three learning mechanisms — even though the brain integrates all three. Here, we have implemented unsupervised, supervised, and reinforcement learning within a network of spiking neurons. In order to achieve this ambitious goal, the existing learning rule called spike-timing-dependent plasticity had to be extended such that it is modulated by the reward signal dopamine.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Hebb, D.: The Organization of Behavior: A Neuropsychological Theory. New York, NY: Wiley (1949).
Dayan, P., Abbott, L.F.: Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems. Cambridge, MA: MIT press (2001).
Izhikevich, E.M.: Simple model of spiking neurons. IEEE. Trans. Neural. Netw. 14(6) (2003) 1569–1572.
Markram, H., Lubke, J., Frotscher, M., Sakmann, B.: Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 275(5297) (1997) 213–215.
Bi, G.Q., Poo, M.M.: Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18(24) (1998) 10464–10472.
Izhikevich, E.M., Gally, J.A., Edelman, G.M.: Spike-timing dynamics of neuronal groups. Cerebral. Cortex. 14 (2004) 933–944.
Yuille, A.L., Geiger, D.: Winner-Take-All Networks. In Arbib, M.A., ed.: The Handbook of Brain Therory and Neural Networks, 2nd edn. Cambridge, MA: MIT Press (2003) 1228–1231.
Handrich, S., Herzog, A., Wolf, A., Herrmann, C.S.: A Biologically Plausible Winner-Takesall Architecture. In: International Conference on Intelligent Computing (ICIC). (in press).
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press (1998).
Izhikevich, E.M.: Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb. Cortex. 17(10) (2007) 2443–2452.
Handrich, S., Herzog, A., Wolf, A., Herrmann, C.S.: Prerequisites for integrating unsupervised and reinforcement learning in a single network of spiking neurons. In: International Joint Conference on Neural Networks (IJCNN). (in press).
Desimone, R.: Neural mechanisms for visual memory and their role in attention. Proc. Natl. Acad. Sci. USA. 93(24) (1996) 13494–13499.
Suzuki,W.A., Miller, E.K., Desimone, R.: Object and place memory in the macaque entorhinal cortex. J. Neurophysiol. 78(2) (1997) 1062–1081.
Redgrave, P., Gurney, K.: The short-latency dopamine signal: a role in discovering novel actions? Nat. Rev. Neurosci. 7(12) (2006) 967–975.
Shen, W., Flajolet, M., Greengard, P., Surmeier, D.J.: Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321(5890) (2008) 848–851.
Gurney, K., Redgrave, P.: A model of sensory reinforced corticostriatal plasticity in the anaethetised rat. In: Society for Neuroscience Abstracts. (2008).
Herzog, A., Kube, K., Michaelis, B., de Lima, A., Baltz, Tand Voigt, T.: Contribution of the GABA shift to the transition from structural initializationto working stage in biologically realistic networks. Neurocomputing (2008) 1134–1142.
Morrison, A., Diesmann, M., Gerstner, W.: Phenomenological models of synaptic plasticity based on spike timing. Biol. Cybern. 98(6) (2008) 459–478.
Abbott, L., Nelson, S.: Synaptic plasticity: taming the beast. Nat. Neurosci. 3(Suppl) (2000) 1178–1183.
Turrigiano, G.: The self-tuning neuron: synaptic scaling of excitatory synapses. Cell 135(3) (2008) 422–435.
Wright, J.J., Liley, D.T.: Simulation of electrocortical waves. Biol. Cybern. 72(4) (1995) 347–356.
David, O., Friston, K.J.: A neural mass model for MEG/EEG: coupling and neuronal dynamics. Neuroimage 20(3) (2003) 1743–1755.
Freeman,W.J.: A field-theoretic approach to understanding scale-free neocortical dynamics. Biol.Cybern. 92(6) (2005) 350–359.
beim Graben, P., Kurths, J.: Simulation of global properties of electroencephalograms with minimal random neural networks. Neurocomputing 71 (2008) 999–1007.
Fründ, I., Herrmann, C.S.: Simulating evoked gamma oscillations of human EEG in a network of spiking neurons reveals an early mechanism of memory matching. In Descalzi, O., Rosso, O., Larrondo, H., eds.: Nonequilibrium Statistical Mechanics and Nonlinear Physics, AIP Conference Proceedings 913(1), American Institute of Physics (2007) 215–221.
Schultz, W.: Dopamine neurons and their role in reward mechanisms. Curr. Opin. Neurobiol. 7(2) (1997) 191–197.
Ungless, M., Magill, P., Bolam, J.: Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 303(5666) (2004) 2040–2042.
Acknowledgments
This work was supported by the DFG HE 3353/6-1 Forschungsverbund “Electrophysiological Correlates of Memory and their Generation”, SFB 779, and by BMBF Bernstein-group “Components of cognition: small networks to flexible rules”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media B.V.
About this paper
Cite this paper
Handrich, S., Herzog, A., Wolf, A., Herrmann, C.S. (2011). Combining Supervised, Unsupervised, and Reinforcement Learning in a Network of Spiking Neurons. In: Wang, R., Gu, F. (eds) Advances in Cognitive Neurodynamics (II). Springer, Dordrecht. https://doi.org/10.1007/978-90-481-9695-1_26
Download citation
DOI: https://doi.org/10.1007/978-90-481-9695-1_26
Published:
Publisher Name: Springer, Dordrecht
Print ISBN: 978-90-481-9694-4
Online ISBN: 978-90-481-9695-1
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)