Abstract
Inspired by biological spike information processing and the multiple brain region coordination mechanism, we propose an autonomous spiking neural network model for decision making. The proposed model is an expansion of the basal ganglia circuitry with automatic environment perception. It automatically constructs environmental states from image inputs. Contributions of this investigation can be summarized as the following: (1) In our model, the simplified Hodgkin-Huxley computing model is developed to achieve calculation efficiency closed to the LIF model and is used to obtain and test the ionic level properties in cognition. (2) A spike based motion perception mechanism is proposed to extract key elements for learning process from raw pixels without large amount of training. We apply our model in the “flappy bird” game and after dozens of training times, it can automatically generate rules to play well in the game. Besides, our model simulates cognitive defects when blocking some of sodium or potassium ion channels in the Hodgkin-Huxley model and this can be considered as a computational exploration on the mechanisms of cognition deep into ionic level.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Chris Eliasmith gave an invited keynote talk at the 2014 Computational Neuroscience Meeting (CNS 2014) on the effect of sodium channel for learning in SPAUN. This talk inspired our investigation introduced in this subsection.
References
Stewart, T.C., Bekolay, T., Eliasmith, C.: Learning to select actions with spiking neurons in the Basal Ganglia. Front. Neurosci. 6(2), 1–14 (2012)
Bekolay, T., Eliasmith, C.: A general error-modulated STDP learning rule applied to reinforcement learning in the Basal Ganglia. In: Computational and Systems Neuroscience Conference, Salt Lake City, Utah, pp. 24–27 (2011)
Eliasmith, C.: How to Build a Brain, pp. 121–171. Oxford, New York (2013). Reprint edition
Chakravarthy, V.S., Joseph, D., Bapi, R.S.: What do the Basal Ganglia do? Model. Perspect. Biol. Cybern. 103(3), 237–253 (2010)
Frank, M.J.: Dynamic dopamine modulation in the Basal Ganglia: a neuro computational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J. Cogn. Neurosci. 17(1), 51–72 (2005)
Utter, A.A., Basso, M.A.: The Basal Ganglia: an overview of circuits and function. Neurosci. Biobehav. Rev. 32(3), 333–342 (2008)
Redgrave, P., Rodriguez, M., Smith, Y., Rodriguez-Oroz, M.C., et al.: Goal-directed and habitual control in the Basal Ganglia: implications for Parkinson’s disease. Nat. Rev. Neurosci. 11, 760–772 (2011)
Stewart, T.C., Choo, X., Eliasmith, C.: Dynamic behavior of a spiking model of action selection in the Basal Ganglia. In: Proceedings of the 10th International Conference on Cognitive Modeling, pp. 5–8 (2010)
Frank, M.J.: Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw. 19(8), 1120–1136 (2006)
Gurney, K., Prescott, T.J., Redgrave, P.: A computational model of action selection in the Basal Ganglia. Biol. Cybern. 84(6), 401–410 (2001)
Stewart, T.C., Eliasmith, C.: Large-scale synthesis of functional spiking neural circuits. Proc. IEEE 102(5), 881–898 (2014)
MacNeil, D., Eliasmith, C.: Fine-tuning and the stability of recurrent neural networks. Public Lib. Sci. (PLoS One) 6(9), 1–16 (2011)
Gurney, K., Prescott, T.J., Wickens, J.R., Redgrave, P.: Computational models of the Basal Ganglia: from robots to membranes. Trends Neurosci. 27(8), 453–459 (2004)
Albin, R.L., Young, A.B., Penney, J.B.: The functional anatomy of Basal Ganglia disorders. Trends Neurosci. 12(10), 366–375 (1989)
Bar-Gad, I., Bergman, H.: Stepping out of the box: information processing in the neural networks of the Basal Ganglia. Curr. Opin. Neurobiol. 11(6), 689–695 (2011)
Iqarashi, J., Shouno, O., Fukai, T., Tsujino, H.: Real-time simulation of a spiking neural network model of the Basal Ganglia circuitry using general purpose computing on graphics processing units. Neural Netw. 24(9), 950–960 (2011)
Cessac, B., Paugam-Moisy, H., Viéville, T.: Overview of facts and issues about neural coding by spikes. J. Physiol. Paris 104(1), 5–18 (2010)
Dayan, P., Abbott, L.F.: Computational and Mathematical Modeling of Neural Systems: Model Neurons I: Neuroelectronic. MIT Press, Cambridge (2003)
Izhikevich, E.M.: Dynamical Systems in Neuroscience: The Geometry of Excitability and Bursting. MIT Press, Cambridge (2004)
Hodgkin, A.L., Huxley, A.F.: A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Physiol. 117(4), 500–544 (1952)
Nelson, M.E.: Electrophysiological models. In: Databasing the Brain: From Data to Knowledge. Wiley, New York (2004)
Gerstner, W., Kistler, W.M.: Spiking Neuron Models. Single Neurons, Populations, Plasticity. Cambridge University Press, Cambridge (2002)
Wells, R.B.: Introduction to biological signal processing and computational neuroscience. Moscow (2010)
Long, L.N., Fang, G.L.: A review of biologically plausible neuron models for spiking neural networks. In: AIAA InfoTech Aerospace Conference, Atlanta, 20–22 April 2010
Weber, C., Elshaw, M., Wermter, S., Triesch, J., Willmot, C.: Reinforcement Learning: Theory and Applications: Reinforcement Learning Embedded in Brains and Robots. Austria (2008)
Bohte, S.M., Poutre, H.L., Kok, J.N.: Unsupervised clustering with spiking neurons by sparse temporal coding and multilayer RBF networks. IEEE Trans. Neural Netw. 13(2), 426–435 (2002)
Acknowledgements
This study was funded by the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB02060007), and Beijing Municipal Commission of Science and Technology (Z151100000915070, Z161100000216124).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Wang, G., Zeng, Y., Xu, B. (2016). A Spiking Neural Network Based Autonomous Reinforcement Learning Model and Its Application in Decision Making. In: Liu, CL., Hussain, A., Luo, B., Tan, K., Zeng, Y., Zhang, Z. (eds) Advances in Brain Inspired Cognitive Systems. BICS 2016. Lecture Notes in Computer Science(), vol 10023. Springer, Cham. https://doi.org/10.1007/978-3-319-49685-6_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-49685-6_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-49684-9
Online ISBN: 978-3-319-49685-6
eBook Packages: Computer ScienceComputer Science (R0)