A Model of Neuronal Specialization Using Hebbian Policy-Gradient with “Slow” Noise
We study a model of neuronal specialization using a policy gradient reinforcement approach. (1) The neurons stochastically fire according to their synaptic input plus a noise term; (2) The environment is a closed-loop system composed of a rotating eye and a visual punctual target; (3) The network is composed of a foveated retina, a primary layer and a motoneuron layer; (4) The reward depends on the distance between the subjective target position and the fovea and (5) the weight update depends on a Hebbian trace defined according to a policy gradient principle. In order to take into account the mismatch between neuronal and environmental integration times, we distort the firing probability with a “pink noise” term whose autocorrelation is of the order of 100 ms, so that the firing probability is overestimated (or underestimated) for about 100 ms periods. The rewards occuring meanwhile assess the “value” of those elementary shifts, and modify the firing probability accordingly. Every motoneuron being associated to a particular angular direction, we test at the end of the learning process the preferred output of the visual cells. We find that accordingly with the observed final behavior, the visual cells preferentially excite the motoneurons heading in the opposite angular direction.
KeywordsNeuronal Specialization Reinforcement Learning Synaptic Input Motor Command Firing Probability
Unable to display preview. Download preview PDF.
- 3.Bartlett, P.L., Baxter, J.: Hebbian synaptic modifications in spiking neurons that learn. Technical report, Research School of Information Sciences and Engineering, Australian National University (1999)Google Scholar
- 7.Florian, R.V.: A reinforcement learning algorithm for spiking neural networks. In: Seventh International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC 2005), pp. 299–306 (2005)Google Scholar
- 9.Sutton, R.S.: Learning to predict by the method of temporal differences. Machine learning 3, 9–44 (1988)Google Scholar
- 10.Hebb, D.: The Organization of behavior. Wiley, New York (1949)Google Scholar
- 11.Softky, W., Koch, C.: The highly irregular firing of cortical cells is inconsistent with temporal integration of random epsps. J. of Neuroscience 13(1), 334–450 (1993)Google Scholar