The Role of the Basal Ganglia in Exploratory Behavior in a Model Based on Reinforcement Learning

  • Sridharan Devarajan
  • P. S. Prashanth
  • V. S. Chakravarthy
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3316)


We present a model of basal ganglia as a key player in exploratory behavior. The model describes exploration of a virtual rat in a simulated “water pool” experiment. The virtual rat is trained using a reward-based or reinforcement learning paradigm which requires units with stochastic behavior for exploration of the system’s state space. We model the STN-GPe system as a pair of neuronal layers with oscillatory dynamics, exhibiting a variety of dynamic regimes like chaos, traveling waves and clustering. Invoking the property of chaotic systems to explore a state space, we suggest that the complex “exploratory” dynamics of STN-GPe system in conjunction with dopamine-based reward signaling present the two key ingredients of a reinforcement learning system.


Basal Ganglion Output Layer Exploratory Behavior Effective Dimension Water Pool 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Redgrave, P., Prescott, T.J., Gurney, K.: The basal ganglia: a vertebrate solution to the selection problem? Neuroscience 89, 1009–1023 (1999)CrossRefGoogle Scholar
  2. 2.
    Berns, G.S., Sejnowski, T.J.: A computational model of how the Basal Ganglia produce sequences. Journal of Cognitive Neuroscience 10(1), 108–121 (1998)CrossRefGoogle Scholar
  3. 3.
    Houk, J.C., Davis, J.L., Beiser, D.G.: Models of Information Processing in the Basal Ganglia. MIT Press, Cambridge (1995)Google Scholar
  4. 4.
    Bevan, M.D., Magill, P.J., Terman, D., Bolam, J.P., Wilson, C.J.: Move To The Rhythm: Oscillations In The Subthalamic Nucleus-External Globus Pallidus Network. In: Trends in Neuroscience (2003) (In press)Google Scholar
  5. 5.
    Chirikov, B.: A universal instability of many-dimensional oscillator systems. Phys. Rev. 52, 263–379 (1979)MathSciNetGoogle Scholar
  6. 6.
    Chakravarthy, V.S., Thomas, S.T., Nair, N.: A model for scheduling motor unit recruitment in skeletal muscle. In: International Conference on Theoretical Neurobiology, National Brain Research Center, Gurgoan, February, 24-26 (2003)Google Scholar
  7. 7.
    Gillies, A., Willshaw, D., Li, Z.: Subthalamic-pallidal interactions are critical in determining normal and abnormal functioning of the basal ganglia. In: Proc R Soc Lond B Biol Sci., March 22, vol. 269(1491), pp. 545–551 (2002)Google Scholar
  8. 8.
    Terman, D., Rubin, J.E., Yew, A.C., Wilson, C.J.: Activity Patterns in a Model for the Subthalamopallidal Network of the Basal Ganglia. In: J Neurosci., April 1, vol. 22(7), pp. 2963–2976 (2002)Google Scholar
  9. 9.
    Harner, A.M.: An Introduction to Basal Ganglia Function. Boston University, Boston (1997)Google Scholar
  10. 10.
    Obeso, J.A., Rodriguez-Oroz, M.C., Rodriguez, M., Arbizu, J., Gimenez-Amaya, J.M.: The Basal Ganglia and Disorders of Movement: Pathophysiological Mechanisms. News Physiol Sci. 17, 51–55 (2002)Google Scholar
  11. 11.
    Montague, Dayan, Sejnowski: A Framework for Mesencephalic Dopamine Systems Based on Predictive Hebbian Learning. The Journal of Neuroscience 16(5), 1936–1947 (1996)Google Scholar
  12. 12.
    Sridharan, D.: Human Factors in Aviation: Willed action and its disorders, MTech Thesis, Department of Aerospace Engineering, IIT, Madras, India (2004)Google Scholar
  13. 13.
    Barto, A.G.: Reinforcement Learning. In: Arbib, M.A. (ed.) The Handbook of Brain Theory and Neural Networks (1st Edition), MIT Press, Cambridge (1999)Google Scholar
  14. 14.
    Morris, R.G.M., Garrud, P., Rawlins, J.N.P.: Place navigation impaired in rats with hippocampal lesions. Nature 297, 681–683 (1982)CrossRefGoogle Scholar
  15. 15.
    Skarda, C.A., Freeman, W.J.: How brain makes chaos in order to make sense of the world. Behavioral and Brain Sciences 10, 161–195 (1987)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Sridharan Devarajan
    • 1
  • P. S. Prashanth
    • 1
  • V. S. Chakravarthy
    • 1
  1. 1.Department of Aerospace Engineering and Department of Electrical EngineeringIndian Institute of TechnologyMadrasIndia

Personalised recommendations