Abstract
The field of Reinforcement Learning (RL) in machine learning relates significantly to the domains of classical and instrumental conditioning in psychology, which give an understanding of biology’s approach to RL. In recent years, there has been a thrust to correlate some machine learning RL algorithms with brain structure and function, a benefit to both fields. Our focus has been on one such structure, the striatum, from which we have built a general model. In machine learning terms, this model is equivalent to a value-function approximator (VFA) that learns according to Temporal Difference error. In keeping with a biological approach to RL, the present work seeks to evaluate the robustness of this striatum-based VFA using biological criteria. We selected five classical conditioning tests to expose the learning accuracy and efficiency of the VFA for simple state-value associations. Manually setting the VFA’s many parameters to reasonable values, we characterize it by varying each parameter independently and repeatedly running the tests. The results show that this VFA is both capable of performing the selected tests and is quite robust to changes in parameters. Test results also reveal aspects of how this VFA encodes reward value.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Schultz, W.: Predictive Reward Signal of Dopamine Neurons. J. Neurophysiol. 80(1), 1–27 (1998)
Lubow, R.E.: Latent inhibition. Psychological Bulletin 79, 398–407 (1973)
Matzel, L.D., Schachtman, T.R., Miller, R.R.: Recovery of an overshadowed association achieved by extinction of the overshadowing stimulus. Learning and Motivation 16(4), 398–412 (1985)
Connor, P.C., Trappenberg, T.: Classical conditioning through a lateral inhibitory model of the striatum (2011) (in preparation)
Wilson, C.J.: Basal Ganglia, 5th edn., pp. 361–413. Oxford University Press, Inc., Oxford (2004)
Wickens, J.R., Begg, A.J., Arbuthnott, G.W.: Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience 70, 1–5 (1996)
Hori, Y., Minamimoto, T., Kimura, M.: Neuronal encoding of reward value and direction of actions in the primate putamen. Journal of Neurophysiology 102(6), 3530–3543 (2009)
Lau, B., Glimcher, P.W.: Value representations in the primate striatum during matching behavior. Neuron 58(3), 451–463 (2008)
Samejima, K.: Representation of Action-Specific reward values in the striatum. Science 310(5752), 1337–1340 (2005)
Bromberg-Martin, E.S., Hikosaka, O., Nakamura, K.: Coding of task reward value in the dorsal raphe nucleus. Journal of Neuroscience 30(18), 6262–6272 (2010)
Gottfried, J.A.: Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science 301(5636), 1104–1107 (2003)
Roesch, M.R.: Neuronal activity related to reward value and motivation in primate frontal cortex. Science 304(5668), 307–310 (2004)
Wickens, J.R., Arbuthnott, G.W., Shindou, T.: Simulation of GABA function in the basal ganglia: computational models of GABAergic mechanisms in basal ganglia function. In: Progress in Brain Research, vol. 160, pp. 313–329. Elsevier, Amsterdam (2007)
Rescorla, R.A., Wagner, A.R.: A theory of pavlovian conditioning: Variations in the effectiveness of reinforcement and non-reinforcement. In: Black, A.H., Prokasy, W.F. (eds.) Classical Conditioning II. Appleton-Century-Crofts, New York (1972)
Rabinovich, M.I., Huerta, R., Volkovskii, A., Abarbanel, H.D.I., Stopfer, M., Laurent, G.: Dynamical coding of sensory information with competitive networks. J. Physiol. (Paris) 94, 465–471 (2000)
Sutton, R.S.: Generalization in reinforcement learning: Successful examples using sparse coarse coding. In: Advances in Neural Information Processing Systems, vol. 8, pp. 1038–1044. MIT Press, Cambridge (1996)
Houk, J., Adams, J., Barto, A.: A Model of How the Basal Ganglia Generate and Use Neural Signals that Predict Reinforcement, pp. 249–270. The MIT Press, Cambridge (1995)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Connor, P., Trappenberg, T. (2011). Characterizing a Brain-Based Value-Function Approximator. In: Butz, C., Lingras, P. (eds) Advances in Artificial Intelligence. Canadian AI 2011. Lecture Notes in Computer Science(), vol 6657. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-21043-3_12
Download citation
DOI: https://doi.org/10.1007/978-3-642-21043-3_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-21042-6
Online ISBN: 978-3-642-21043-3
eBook Packages: Computer ScienceComputer Science (R0)