Modulation of Dopamine for Adaptive Learning: a Neurocomputational Model

Abstract

There have been many proposals that learning rates in the brain are adaptive, in the sense that they increase or decrease depending on environmental conditions. The majority of these models are abstract and make no attempt to describe the neural circuitry that implements the proposed computations. This article describes a biologically detailed computational model that overcomes this shortcoming. Specifically, we propose a neural circuit that implements adaptive learning rates by modulating the gain on the dopamine response to reward prediction errors, and we model activity within this circuit at the level of spiking neurons. The model generates a dopamine signal that depends on the size of the tonically active dopamine neuron population and the phasic spike rate. The model was tested successfully against results from two single-neuron recording studies and a fast-scan cyclic voltammetry study. We conclude by discussing the general applicability of the model to dopamine-mediated tasks that transcend the experimental phenomena it was initially designed to address.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Notes

  1. 1.

    However, note that in Izhikevich (2007) and Ashby (2018), the β parameter controls the rate of tonic spiking. Each region in our model has a different tonic firing rate; therefore, β = 0 in NAcc, β = 20 in VP, and β = 62 in VTA.

  2. 2.

    However, for Figs. 345 (left and center), and 6, the PPTN square wave lasted 1000 ms and the LH square wave lasted a maximum of 1000 ms. This was done to ensure a sufficiently long interval to extract accurate measurements of firing rate and active population size. Figures showing dopamine output used the parameters described in the text.

References

  1. Aminoff, E.M., Kveraga, K., & Bar, M. (2013). The role of the parahippocampal cortex in cognition. Trends in Cognitive Sciences, 17(8), 379–390.

    PubMed  PubMed Central  Google Scholar 

  2. Ashby, F.G. (2018). Computational cognitive neuroscience. In Batchelder, W., Colonius, H., Dzhafarov, E., & Myung, J. (Eds.) New handbook of mathematical psychology, vol. 2 (pp. 223–270). New York. New York: Cambridge University Press.

  3. Ashby, F.G., & Crossley, M.J. (2011). A computational model of how cholinergic interneurons protect striatal-dependent learning. Journal of Cognitive Neuroscience, 23(6), 1549–1566.

    PubMed  Google Scholar 

  4. Ashby, F.G., & Ennis, J.M. (2006). The role of the basal ganglia in category learning. Psychology of Learning and Motivation, 46, 1–36.

    Google Scholar 

  5. Ashby, F.G., & Helie, S. (2011). A tutorial on computational cognitive neuroscience: modeling the neurodynamics of cognition. Journal of Mathematical Psychology, 55(4), 273–289.

    PubMed  PubMed Central  Google Scholar 

  6. Ashby, F.G., Isen, A.M., & Turken, A. (1999). A neuropsychological theory of positive affect and its influence on cognition. Psychological Review, 106(3), 529–550.

    PubMed  Google Scholar 

  7. Ashby, F.G., Valentin, V.V., & Turken, A.U. (2002). The effects of positive affect and arousal and working memory and executive attention: neurobiology and computational models. In Moore, S., & Oaksford, M. (Eds.) Emotional cognition: from brain to behaviour (pp. 245–287). Amsterdam: John Benjamins Publishing Company.

  8. Ashby, F.G., & Vucovich, L.E. (2016). The role of feedback contingency in perceptual category learning. Journal of Experimental Psychology: Learning. Memory, and Cognition, 42(11), 1731.

    Google Scholar 

  9. Bayer, H.M., & Glimcher, P.W. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47(1), 129–141.

    PubMed  PubMed Central  Google Scholar 

  10. Bayer, H.M., Lau, B., & Glimcher, P.W. (2007). Statistics of midbrain dopamine neuron spike trains in the awake primate. Journal of Neurophysiology, 98(3), 1428–1439.

    PubMed  Google Scholar 

  11. Behrens, T.E., Woolrich, M.W., Walton, M.E., & Rushworth, M.F. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10(9), 1214–1221.

    PubMed  Google Scholar 

  12. Belin, D., & Everitt, B.J. (2008). Cocaine seeking habits depend upon dopamine-dependent serial connectivity linking the ventral with the dorsal striatum. Neuron, 57(3), 432–441.

    PubMed  Google Scholar 

  13. Berke, J.D. (2018). What does dopamine mean? Nature Neuroscience, 21(6), 787–793.

    PubMed  PubMed Central  Google Scholar 

  14. Bernacchia, A., Seo, H., Lee, D., & Wang, X. -J. (2011). A reservoir of time constants for memory traces in cortical neurons. Nature Neuroscience, 14(3), 366–372.

    PubMed  PubMed Central  Google Scholar 

  15. Berridge, K.C. (2000). Reward learning: reinforcement, incentives, and expectations. In Medin, D. (Ed.) Psychology of learning and motivation, (Vol. 40 pp. 223–278): Elsevier.

  16. Bland, A.R., & Schaefer, A. (2012). Different varieties of uncertainty in human decision-making. Frontiers in Neuroscience, 6, 85.

    PubMed  PubMed Central  Google Scholar 

  17. Bortz, D.M., Gazo, K.L., & Grace, A.A. (2019). The medial septum enhances reversal learning via opposing actions on ventral tegmental area and substantia nigra dopamine neurons. Neuropsychopharmacology, 1–9.

  18. Bortz, D.M., & Grace, A.A. (2018). Medial septum differentially regulates dopamine neuron activity in the rat ventral tegmental area and substantia nigra via distinct pathways. Neuropsychopharmacology, 43, 2093–2100.

    PubMed  PubMed Central  Google Scholar 

  19. Braganza, O., & Beck, H. (2018). The circuit motif as a conceptual tool for multilevel neuroscience. Trends in Neurosciences, 41(3), 128–136.

    PubMed  Google Scholar 

  20. Bromberg-Martin, E.S., Matsumoto, M., & Hikosaka, O. (2010). Dopamine in motivational control: rewarding, aversive, and alerting. Neuron, 68(5), 815–834.

    PubMed  PubMed Central  Google Scholar 

  21. Brown, J., Bullock, D., & Grossberg, S. (1999). How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. Journal of Neuroscience, 19(23), 10502–10511.

    PubMed  Google Scholar 

  22. Bush, R.R., & Mosteller, F. (1951). A model for stimulus generalization and discrimination. Psychological Review, 58(6), 413– 423.

    PubMed  Google Scholar 

  23. Cantwell, G., Riesenhuber, M., Roeder, J.L., & Ashby, F.G. (2017). Perceptual category learning and visual processing: an exercise in computational cognitive neuroscience. Neural Networks, 89, 31–38.

    PubMed  PubMed Central  Google Scholar 

  24. Christopoulos, G.I., Tobler, P.N., Bossaerts, P., Dolan, R.J., & Schultz, W. (2009). Neural correlates of value, risk, and risk aversion contributing to decision making under risk. Journal of Neuroscience, 29 (40), 12574–12583.

    PubMed  Google Scholar 

  25. Cohen, J.Y., Haesler, S., Vong, L., Lowell, B.B., & Uchida, N. (2012). Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature, 482(7383), 85–88.

    PubMed  PubMed Central  Google Scholar 

  26. Contreras-Vidal, J.L., & Schultz, W. (1999). A predictive reinforcement model of dopamine neurons for learning approach behavior. Journal of Computational Neuroscience, 6(3), 191–214.

    PubMed  Google Scholar 

  27. Cools, R. (2006). Dopaminergic modulation of cognitive function-implications for L-DOPA treatment in Parkinson’s disease. Neuroscience and Biobehavioral Reviews, 30(1), 1–23.

    PubMed  Google Scholar 

  28. Cools, R., Clark, L., Owen, A.M., & Robbins, T.W. (2002). Defining the neural mechanisms of probabilistic reversal learning using event-related functional magnetic resonance imaging. Journal of Neuroscience, 22(11), 4563–4567.

    PubMed  Google Scholar 

  29. Cools, R., & D’Esposito, M. (2011). Inverted U-shaped dopamine actions on human working memory and cognitive control. Biological Psychiatry, 69(12), e113–e125.

    PubMed  PubMed Central  Google Scholar 

  30. Cools, R., & Robbins, T.W. (2004). Chemistry of the adaptive mind. Philosophical Transactions of the Royal Society of London. Series A: Mathematical. Physical and Engineering Sciences, 362(1825), 2871–2888.

    Google Scholar 

  31. Cornwall, J., & Phillipson, O. (1988). Afferent projections to the parafascicular thalamic nucleus of the rat, as shown by the retrograde transport of wheat germ agglutinin. Brain Research Bulletin, 20(2), 139–150.

    PubMed  Google Scholar 

  32. Crossley, M.J., Ashby, F.G., & Maddox, W.T. (2013). Erasing the engram: the unlearning of procedural skills. Journal of Experimental Psychology: General, 142(3), 710–741.

    Google Scholar 

  33. Daw, N.D., & O’Doherty, J.P. (2014). Multiple systems for value learning. In P. W. Glimcher, & E. Fehr (Eds.) Neuroeconomics: decision making and the brain, Second edition (pp. 393–410). Amsterdam: Elsevier.

  34. Dayan, P., & Abbott, L.F. (2001). Theoretical neuroscience: computational and mathematical modeling of neural systems. Cambridge: MIT Press.

    Google Scholar 

  35. Dayan, P., Kakade, S., & Montague, P.R. (2000). Learning and selective attention. Nature Neuroscience, 3(11s), 1218–1223.

    PubMed  Google Scholar 

  36. Dayan, P., & Long, T. (1998). Statistical models of conditioning. In Jordan, M. I., Kearns, M. J., & Solla, S. A. (Eds.) Advances in neural information processing systems: Proceedings of the 1997 Conference (pp. 117–123). Cambridge, MA: MIT Press.

  37. Dayan, P., & Yu, A.J. (2003). Expected and unexpected uncertainty: ACh and NE in the neocortex. In Becker, S., Thrun, S., & Obermayer, K. (Eds.) Advances in neural information processing systems: Proceedings of the 2002 Conference (pp. 173–180). Cambridge, MA: MIT Press.

  38. Deng, P., Zhang, Y., & Xu, Z.C. (2007). Involvement of Ih in dopamine modulation of tonic firing in striatal cholinergic interneurons. Journal of Neuroscience, 27(12), 3148–3156.

    PubMed  Google Scholar 

  39. Ding, J.B., Guzman, J.N., Peterson, J.D., Goldberg, J.A., & Surmeier, D.J. (2010). Thalamic gating of corticostriatal signaling by cholinergic interneurons. Neuron, 67(2), 294–307c.

    PubMed  PubMed Central  Google Scholar 

  40. Doig, N.M., Magill, P.J., Apicella, P., Bolam, J.P., & Sharott, A. (2014). Cortical and thalamic excitation mediate the multiphasic responses of striatal cholinergic interneurons to motivationally salient stimuli. Journal of Neuroscience, 34(8), 3101–3117.

    PubMed  Google Scholar 

  41. Ermentrout, G.B. (1996). Type I membranes, phase resetting curves, and synchrony. Neural Computation, 8 (5), 979–1001.

    PubMed  Google Scholar 

  42. Eshel, N., Bukwich, M., Rao, V., Hemmelder, V., Tian, J., & Uchida, N. (2015). Arithmetic and local circuitry underlying dopamine prediction errors. Nature, 525(7568), 243–246.

    PubMed  PubMed Central  Google Scholar 

  43. Fabbricatore, A.T., Ghitza, U.E., Prokopenko, V.F., & West, M.O. (2009). Electrophysiological evidence of mediolateral functional dichotomy in the rat accumbens during cocaine self-administration: tonic firing patterns. European Journal of Neuroscience, 30(12), 2387–2400.

    Google Scholar 

  44. Faget, L., Osakada, F., Duan, J., Ressler, R., Johnson, A.B., Proudfoot, J.A., & Hnasko, T. S. (2016). Afferent inputs to neurotransmitter-defined cell types in the ventral tegmental area. Cell reports, 15(12), 2796–2808.

    PubMed  PubMed Central  Google Scholar 

  45. Fanselow, M.S., & Dong, H.W. (2010). Are the dorsal and ventral hippocampus functionally distinct structures? Neuron, 65(1), 7–19.

    PubMed  PubMed Central  Google Scholar 

  46. Farashahi, S., Donahue, C.H., Khorsand, P., Seo, H., Lee, D., & Soltani, A. (2017). Metaplasticity as a neural substrate for adaptive learning and choice under uncertainty. Neuron, 94(2), 401–414.

    PubMed  PubMed Central  Google Scholar 

  47. Franklin, N.T., & Frank, M.J. (2015). A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning. Elife, 4.

  48. Friston, K.J., Shiner, T., FitzGerald, T., Galea, J.M., Adams, R., Brown, H., & Bestmann, S. (2012). Dopamine, affordance and active inference. PLoS Computational Biology, 8(1).

  49. Gloor, P. (1997). The temporal lobe and limbic system. New York: Oxford University Press.

    Google Scholar 

  50. Grace, A.A. (2010). Dopamine system dysregulation by the ventral subiculum as the common pathophysiological basis for schizophrenia psychosis, psychostimulant abuse, and stress. Neurotoxicity Research, 18(3-4), 367–376.

    PubMed  PubMed Central  Google Scholar 

  51. Grace, A.A., & Bunney, B.S. (1983). Intracellular and extracellular electrophysiology of nigral dopaminergic neurons-1. Identification and characterization. Neuroscience, 10(2), 301–315.

    PubMed  PubMed Central  Google Scholar 

  52. Grace, A.A., Floresco, S.B., Goto, Y., & Lodge, D.J. (2007). Regulation of firing of dopaminergic neurons and control of goal-directed behaviors. Trends in Neurosciences, 30(5), 220–227.

    PubMed  Google Scholar 

  53. Haber, S.N. (2016). Corticostriatal circuitry. Dialogues in Clinical Neuroscience, 18(1), 7.

    PubMed  PubMed Central  Google Scholar 

  54. Haber, S.N., Fudge, J.L., & McFarland, N.R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. Journal of Neuroscience, 20(6), 2369–2382.

    PubMed  Google Scholar 

  55. Harrison, L.M., Duggins, A., & Friston, K.J. (2006). Encoding uncertainty in the hippocampus. Neural Networks, 19(5), 535–546.

    PubMed  PubMed Central  Google Scholar 

  56. Hart, A.S., Rutledge, R.B., Glimcher, P.W., & Phillips, P.E. (2014). Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. Journal of Neuroscience, 34(3), 698–704.

    PubMed  Google Scholar 

  57. Hazy, T.E., & Frank, M.J. (2010). O’Reilly, R. C Neural mechanisms of acquired phasic dopamine responses in learning. Neuroscience and Biobehavioral Reviews, 34(5), 701–720.

    PubMed  Google Scholar 

  58. Hong, S., & Hikosaka, O. (2014). Pedunculopontine tegmental nucleus neurons provide reward, sensorimotor, and alerting signals to midbrain dopamine neurons. Neuroscience, 282, 139–155.

    PubMed  PubMed Central  Google Scholar 

  59. Hong, S., Jhou, T.C., Smith, M., Saleem, K.S., & Hikosaka, O. (2011). Negative reward signals from the lateral habenula to dopamine neurons are mediated by rostromedial tegmental nucleus in primates. Journal of Neuroscience, 31(32), 11457–11471.

    PubMed  Google Scholar 

  60. Horvitz, J.C. (2002). Dopamine gating of glutamatergic sensorimotor and incentive motivational input signals to the striatum. Behavioural Brain Research, 137(1-2), 65–74.

    PubMed  Google Scholar 

  61. Houk, J., Adams, J., & Barto, A. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In Davis, J.L., Beiser, D.G., & Houk J.C. (Eds.) Models of information processing in the basal ganglia (pp. 249–270). Cambridge: MIT Press.

  62. Huettel, S.A., Song, A.W., & McCarthy, G. (2005). Decisions under uncertainty: probabilistic context influences activation of prefrontal and parietal cortices. Journal of Neuroscience, 25(13), 3304–3311.

    PubMed  Google Scholar 

  63. Humphries, M.D., & Prescott, T.J. (2010). The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Progress in Neurobiology, 90(4), 385–417.

    PubMed  Google Scholar 

  64. Iigaya, K. (2016). Adaptive learning and decision-making under uncertainty by metaplastic synapses guided by a surprise detection system. Elife, 5, e18073.

    PubMed  PubMed Central  Google Scholar 

  65. Insausti, R., Amaral, D., & Cowan, W. (1987). The entorhinal cortex of the monkey: II. Cortical afferents. Journal of Comparative Neurology, 264(3), 356–395.

    Google Scholar 

  66. Izhikevich, E.M. (2003). Simple model of spiking neurons. IEEE Transactions on Neural Networks, 14(6), 1569–1572.

    PubMed  Google Scholar 

  67. Izhikevich, E.M. (2007). Dynamical systems in neuroscience. Cambridge, CA: MIT Press.

    Google Scholar 

  68. Jacobs, J., Kahana, M.J., Ekstrom, A.D., Mollison, M.V., & Fried, I. (2010). A sense of direction in human entorhinal cortex. Proceedings of the National Academy of Sciences, 107(14), 6487–6492.

    Google Scholar 

  69. Jhou, T.C., Fields, H.L., Baxter, M.G., Saper, C.B., & Holland, P.C. (2009). The rostromedial tegmental nucleus (RMTg), a GABAergic afferent to midbrain dopamine neurons, encodes aversive stimuli and inhibits motor responses. Neuron, 61(5), 786–800.

    PubMed  PubMed Central  Google Scholar 

  70. Jo, S., & Jung, M.W. (2016). Differential coding of uncertain reward in rat insular and orbitofrontal cortex. Scientific Reports, 6, 24085.

    PubMed  PubMed Central  Google Scholar 

  71. Joel, D., Niv, Y., & Ruppin, E. (2002). Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks, 15(4-6), 535–547.

    Google Scholar 

  72. Jones, B.F., & Witter, M.P. (2007). Cingulate cortex projections to the parahippocampal region and hippocampal formation in the rat. Hippocampus, 17(10), 957–976.

    PubMed  Google Scholar 

  73. Kawato, M., & Samejima, K. (2007). Efficient reinforcement learning: computational theories, neuroscience and robotics. Current Opinion in Neurobiology, 17(2), 205–212.

    PubMed  Google Scholar 

  74. Keiflin, R., Pribut, H.J., Shah, N.B., & Janak, P.H. (2019). Ventral tegmental dopamine neurons participate in reward identity predictions. Current Biology, 29(1), 93–103.

    PubMed  Google Scholar 

  75. Kerr, K.M., Agster, K.L., Furtak, S.C., & Burwell, R.D. (2007). Functional neuroanatomy of the parahippocampal region: the lateral and medial entorhinal areas. Hippocampus, 17(9), 697–708.

    PubMed  Google Scholar 

  76. Kobayashi, Y., & Okada, K. (2007). Reward prediction error computation in the pedunculopontine tegmental nucleus neurons. Annals of the New York Academy of Sciences, 1104(1), 310–323.

    PubMed  Google Scholar 

  77. Kumaran, D., & Maguire, E.A. (2006). An unexpected sequence of events: mismatch detection in the human hippocampus. PLoS Biology, 4(12), e424.

    PubMed  PubMed Central  Google Scholar 

  78. Lipski, W.J., & Grace, A.A. (2013). Activation and inhibition of neurons in the hippocampal ventral subiculum by norepinephrine and locus coeruleus stimulation. Neuropsychopharmacology, 38(2), 285.

    PubMed  Google Scholar 

  79. Liu, X., Hairston, J., Schrier, M., & Fan, J. (2011). Common and distinct networks underlying reward valence and processing stages: a meta-analysis of functional neuroimaging studies. Neuroscience and Biobehavioral Reviews, 35(5), 1219–1236.

    PubMed  PubMed Central  Google Scholar 

  80. Lodge, D.J., & Grace, A.A. (2006). The hippocampus modulates dopamine neuron responsivity by regulating the intensity of phasic neuron activation. Neuropsychopharmacology, 31(7), 1356–1361.

    PubMed  Google Scholar 

  81. Maia, T.V. (2009). Reinforcement learning, conditioning, and the brain: successes and challenges. Cognitive, Affective, and Behavioral Neuroscience, 9(4), 343–364.

    Google Scholar 

  82. Marr, D. (1982). Vision: a computational investigation into the human representation and processing of visual information. New York: Freeman.

    Google Scholar 

  83. Mathys, C., Daunizeau, J., Friston, K.J., & Stephan, K.E. (2011). A Bayesian foundation for individual learning under uncertainty. Frontiers in Human Neuroscience, 5, 39.

    PubMed  PubMed Central  Google Scholar 

  84. Matsumoto, M., & Hikosaka, O. (2007). Lateral habenula as a source of negative reward signals in dopamine neurons. Nature, 447(7148), 1111–1115.

    PubMed  Google Scholar 

  85. Matsumoto, M., & Hikosaka, O. (2009). Representation of negative motivational value in the primate lateral habenula. Nature Neuroscience, 12(1), 77–84.

    PubMed  Google Scholar 

  86. Matsumoto, N., Minamimoto, T., Graybiel, A.M., & Kimura, M. (2001). Neurons in the thalamic CM-Pf complex supply striatal neurons with information about behaviorally significant sensory events. Journal of Neurophysiology, 85(2), 960–976.

    PubMed  Google Scholar 

  87. Mishkin, M., Malamut, B., & Bachevalier, J. (1984). Memories and habits: two neural systems. In Lynch, G., McGaugh, J. L., & Weinberger, N. M. (Eds.) Neurobiology of human learning and memory (pp. 65–77). New York: Guilford Press.

  88. Monosov, I.E. (2017). Anterior cingulate is a source of valence-specific information about value and uncertainty. Nature Communications, 8(1), 134.

    PubMed  PubMed Central  Google Scholar 

  89. Montague, P.R., Dayan, P., & Sejnowski, T.J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience, 16(5), 1936–1947.

    PubMed  Google Scholar 

  90. Morita, K., Morishima, M., Sakai, K., & Kawaguchi, Y. (2012). Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways. Trends in Neurosciences, 35(8), 457–467.

    PubMed  Google Scholar 

  91. Morita, K., Morishima, M., Sakai, K., & Kawaguchi, Y. (2013). Dopaminergic control of motivation and reinforcement learning: a closed-circuit account for reward-oriented behavior. Journal of Neuroscience, 33(20), 8866–8890.

    PubMed  Google Scholar 

  92. Niv, Y., Daw, N.D., Joel, D., & Dayan, P. (2007). Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology, 191(3), 507–520.

    PubMed  Google Scholar 

  93. Okada, K., & Kobayashi, Y. (2013). Reward prediction-related increases and decreases in tonic neuronal activity of the pedunculopontine tegmental nucleus. Frontiers in Integrative Neuroscience, 7, 36.

    PubMed  PubMed Central  Google Scholar 

  94. Neill, M., & Schultz, W. (2010). Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value. Neuron, 68(4), 789–800.

    Google Scholar 

  95. O’Reilly, R.C., Frank, M.J., Hazy, T.E., & Watz, B. (2007). PVLV: the primary value and learned value Pavlovian learning algorithm. Behavioral Neuroscience, 121(1), 31–49.

    PubMed  Google Scholar 

  96. Payzan-LeNestour, E., & Bossaerts, P. (2011). Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Computational Biology, 7(1), e1001048.

    PubMed  PubMed Central  Google Scholar 

  97. Payzan-LeNestour, E., Dunne, S., Bossaerts, P., & O’Doherty, J.P. (2013). The neural representation of unexpected uncertainty during value-based decision making. Neuron, 79(1), 191–201.

    PubMed  PubMed Central  Google Scholar 

  98. Pickering, A.D., & Pesola, F. (2014). Modeling dopaminergic and other processes involved in learning from reward prediction error: contributions from an individual differences perspective. Frontiers in Human Neuroscience, 8, 740.

    PubMed  PubMed Central  Google Scholar 

  99. Preuschoff, K., & Bossaerts, P. (2007). Adding prediction risk to the theory of reward learning. Annals of the New York Academy of Sciences, 1104(1), 135–146.

    PubMed  Google Scholar 

  100. Preuschoff, K., Quartz, S.R., & Bossaerts, P. (2008). Human insula activation reflects risk prediction errors as well as risk. Journal of Neuroscience, 28(11), 2745–2752.

    PubMed  Google Scholar 

  101. Quintero, E., Diaz, E., Vargas, J.P., de la Casa, G., & Lopez, J.C. (2011). Ventral subiculum involvement in latent inhibition context specificity. Physiology and Behavior, 102(3-4), 414–420.

    PubMed  Google Scholar 

  102. Rall, W. (1967). Distinguishing theoretical synaptic potentials computed for different soma-dendritic distributions of synaptic input. Journal of Neurophysiology, 30(5), 1138–1168.

    PubMed  Google Scholar 

  103. Rescorla, R.A., & Wagner, A.R. (1972). A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In Black, A. H., & Prokasy, W. F. (Eds.) Classical conditioning II: current research and theory (pp. 64–99). New York: Appleton-Century-Crofts.

  104. Riceberg, J.S., & Shapiro, M.L. (2012). Reward stability determines the contribution of orbitofrontal cortex to adaptive behavior. Journal of Neuroscience, 32(46), 16402–16409.

    PubMed  Google Scholar 

  105. Root, D.H., Fabbricatore, A.T., Pawlak, A.P., Barker, D.J., Ma, S., & West, M.O. (2012). Slow phasic and tonic activity of ventral pallidal neurons during cocaine self-administration. Synapse, 66(2), 106–127.

    PubMed  Google Scholar 

  106. Rushworth, M.F., & Behrens, T.E. (2008). Choice, uncertainty and value in prefrontal and cingulate cortex. Nature Neuroscience, 11(4), 389–397.

    PubMed  Google Scholar 

  107. Rutishauser, U., Mamelak, A.N., & Schuman, E.M. (2006). Single-trial learning of novel stimuli by individual neurons of the human hippocampus-amygdala complex. Neuron, 49(6), 805– 813.

    PubMed  Google Scholar 

  108. Sadikot, A., Parent, A., & Francois, C. (1992). Efferent connections of the centromedian and parafascicular thalamic nuclei in the squirrel monkey: a PHA-L study of subcortical projections. Journal of Comparative Neurology, 315(2), 137–159.

    Google Scholar 

  109. Salum, C., da Silva, A.R., & Pickering, A. (1999). Striatal dopamine in attentional learning: a computational model. Neurocomputing, 26, 845–854.

    Google Scholar 

  110. Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80(1), 1–27.

    PubMed  Google Scholar 

  111. Schultz, W., Dayan, P., & Montague, P.R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599.

    PubMed  PubMed Central  Google Scholar 

  112. Seamans, J.K., & Robbins, T.W. (2010). Dopamine modulation of the prefrontal cortex and cognitive function. In Neve, K. A. (Ed.) The dopamine receptors. 2nd edn. (pp. 373–398). New York: Springer.-.

  113. Sesack, S.R., & Grace, A.A. (2010). Cortico-basal ganglia reward network: microcircuitry. Neuropsychopharmacology, 35(1), 27–47.

    PubMed  Google Scholar 

  114. Smith, Y., & Kieval, J.Z. (2000). Anatomy of the dopamine system in the basal ganglia. Trends in Neurosciences, 23, S28–S33.

    PubMed  Google Scholar 

  115. Soltani, A., & Izquierdo, A. (2019). Adaptive learning under expected and unexpected uncertainty. Nature Reviews Neuroscience, 20(10), 635–644.

    PubMed  PubMed Central  Google Scholar 

  116. Strange, B.A., Duggins, A., Penny, W., Dolan, R.J., & Friston, K.J. (2005). Information theory, novelty and hippocampal responses: unpredicted or unpredictable? Neural Networks, 18(3), 225–230.

    PubMed  Google Scholar 

  117. Stuber, G.D., Klanker, M., De Ridder, B., Bowers, M.S., Joosten, R.N., Feenstra, M.G., & Bonci, A. (2008). Reward predictive cues enhance excitatory synaptic strength onto midbrain dopamine neurons. Science, 321(5896), 1690–1692.

    PubMed  PubMed Central  Google Scholar 

  118. Sutton, R.S. (1992). Adapting bias by gradient descent: an incremental version of delta-bar-delta. In Proceedings of the tenth national conference on artificial intelligence (pp. 171–176). Cambridge: MIT Press.

  119. Sutton, R.S., & Barto, A.G. (1998). Reinforcement learning: an introduction Cambridge. MA: MIT Press.

    Google Scholar 

  120. Takahashi, Y.K., Langdon, A.J., Niv, Y., & Schoenbaum, G. (2016). Temporal specificity of reward prediction errors signaled by putative dopamine neurons in rat VTA depends on ventral striatum. Neuron, 91(1), 182–193.

    PubMed  PubMed Central  Google Scholar 

  121. Takahashi, Y.K., Schoenbaum, G., & Niv, Y. (2008). Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Frontiers in neuroscience, 2, 14.

    Google Scholar 

  122. Tan, C.O., & Bullock, D. (2008). A local circuit model of learned striatal and dopamine cell responses under probabilistic schedules of reward. Journal of Neuroscience, 28(40), 10062–10074.

    PubMed  Google Scholar 

  123. Taswell, C.A., Costa, V.D., Murray, E.A., & Averbeck, B.B. (2018). Ventral striatum’s role in learning from gains and losses. Proceedings of the National Academy of Sciences, 115(52), E12398–E12406.

    Google Scholar 

  124. Tian, J., & Uchida, N. (2015). Habenula lesions reveal that multiple mechanisms underlie dopamine prediction errors. Neuron, 87(6), 1304–1316.

    PubMed  PubMed Central  Google Scholar 

  125. Vanni-Mercier, G., Mauguiere, F., Isnard, J., & Dreher, J.-C. (2009). The hippocampus codes the uncertainty of cue-outcome associations: an intracranial electrophysiological study in humans. Journal of Neuroscience, 29(16), 5287–5294.

    PubMed  Google Scholar 

  126. Van Rossum, G., & Drake, F.L. (2011). The Python language reference manual. Network Theory Ltd.

  127. Vitay, J., & Hamker, F.H. (2014). Timing and expectation of reward: a neuro-computational model of the afferents to the ventral tegmental area. Frontiers in Neurorobotics, 8, 4.

    PubMed  PubMed Central  Google Scholar 

  128. Watabe-Uchida, M., Zhu, L., Ogawa, S.K., Vamanrao, A., & Uchida, N. (2012). Whole-brain mapping of direct inputs to midbrain dopamine neurons. Neuron, 74(5), 858–873.

    PubMed  Google Scholar 

  129. Willingham, D.B. (1998). A neuropsychological theory of motor skill learning. Psychological Review, 105, 558–584.

    PubMed  Google Scholar 

  130. Yu, A.J., & Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron, 46(4), 681–692.

    PubMed  Google Scholar 

Download references

Funding

This research was supported by NIH Grant 2R01MH063760.

Author information

Affiliations

Authors

Corresponding author

Correspondence to F. Gregory Ashby.

Ethics declarations

Competing Interests

The authors declare no that they no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Inglis, J.B., Valentin, V.V. & Ashby, F.G. Modulation of Dopamine for Adaptive Learning: a Neurocomputational Model. Comput Brain Behav 4, 34–52 (2021). https://doi.org/10.1007/s42113-020-00083-x

Download citation

Keywords

  • Dopamine
  • Adaptive learning rate
  • Computational cognitive neuroscience
  • Ventral subiculum