Skip to main content

Advertisement

SpringerLink
Log in

Modulation of Dopamine for Adaptive Learning: a Neurocomputational Model

  • Original Paper
  • Published:
Computational Brain & Behavior Aims and scope Submit manuscript

Abstract

There have been many proposals that learning rates in the brain are adaptive, in the sense that they increase or decrease depending on environmental conditions. The majority of these models are abstract and make no attempt to describe the neural circuitry that implements the proposed computations. This article describes a biologically detailed computational model that overcomes this shortcoming. Specifically, we propose a neural circuit that implements adaptive learning rates by modulating the gain on the dopamine response to reward prediction errors, and we model activity within this circuit at the level of spiking neurons. The model generates a dopamine signal that depends on the size of the tonically active dopamine neuron population and the phasic spike rate. The model was tested successfully against results from two single-neuron recording studies and a fast-scan cyclic voltammetry study. We conclude by discussing the general applicability of the model to dopamine-mediated tasks that transcend the experimental phenomena it was initially designed to address.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. However, note that in Izhikevich (2007) and Ashby (2018), the β parameter controls the rate of tonic spiking. Each region in our model has a different tonic firing rate; therefore, β = 0 in NAcc, β = 20 in VP, and β = 62 in VTA.

  2. However, for Figs. 3, 4, 5 (left and center), and 6, the PPTN square wave lasted 1000 ms and the LH square wave lasted a maximum of 1000 ms. This was done to ensure a sufficiently long interval to extract accurate measurements of firing rate and active population size. Figures showing dopamine output used the parameters described in the text.

References

  • Aminoff, E.M., Kveraga, K., & Bar, M. (2013). The role of the parahippocampal cortex in cognition. Trends in Cognitive Sciences, 17(8), 379–390.

    Article  PubMed  PubMed Central  Google Scholar 

  • Ashby, F.G. (2018). Computational cognitive neuroscience. In Batchelder, W., Colonius, H., Dzhafarov, E., & Myung, J. (Eds.) New handbook of mathematical psychology, vol. 2 (pp. 223–270). New York. New York: Cambridge University Press.

  • Ashby, F.G., & Crossley, M.J. (2011). A computational model of how cholinergic interneurons protect striatal-dependent learning. Journal of Cognitive Neuroscience, 23(6), 1549–1566.

    Article  PubMed  Google Scholar 

  • Ashby, F.G., & Ennis, J.M. (2006). The role of the basal ganglia in category learning. Psychology of Learning and Motivation, 46, 1–36.

    Article  Google Scholar 

  • Ashby, F.G., & Helie, S. (2011). A tutorial on computational cognitive neuroscience: modeling the neurodynamics of cognition. Journal of Mathematical Psychology, 55(4), 273–289.

    Article  PubMed  PubMed Central  Google Scholar 

  • Ashby, F.G., Isen, A.M., & Turken, A. (1999). A neuropsychological theory of positive affect and its influence on cognition. Psychological Review, 106(3), 529–550.

    Article  PubMed  Google Scholar 

  • Ashby, F.G., Valentin, V.V., & Turken, A.U. (2002). The effects of positive affect and arousal and working memory and executive attention: neurobiology and computational models. In Moore, S., & Oaksford, M. (Eds.) Emotional cognition: from brain to behaviour (pp. 245–287). Amsterdam: John Benjamins Publishing Company.

  • Ashby, F.G., & Vucovich, L.E. (2016). The role of feedback contingency in perceptual category learning. Journal of Experimental Psychology: Learning. Memory, and Cognition, 42(11), 1731.

    Article  Google Scholar 

  • Bayer, H.M., & Glimcher, P.W. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47(1), 129–141.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bayer, H.M., Lau, B., & Glimcher, P.W. (2007). Statistics of midbrain dopamine neuron spike trains in the awake primate. Journal of Neurophysiology, 98(3), 1428–1439.

    Article  PubMed  Google Scholar 

  • Behrens, T.E., Woolrich, M.W., Walton, M.E., & Rushworth, M.F. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10(9), 1214–1221.

    Article  PubMed  Google Scholar 

  • Belin, D., & Everitt, B.J. (2008). Cocaine seeking habits depend upon dopamine-dependent serial connectivity linking the ventral with the dorsal striatum. Neuron, 57(3), 432–441.

    Article  PubMed  Google Scholar 

  • Berke, J.D. (2018). What does dopamine mean? Nature Neuroscience, 21(6), 787–793.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bernacchia, A., Seo, H., Lee, D., & Wang, X. -J. (2011). A reservoir of time constants for memory traces in cortical neurons. Nature Neuroscience, 14(3), 366–372.

    Article  PubMed  PubMed Central  Google Scholar 

  • Berridge, K.C. (2000). Reward learning: reinforcement, incentives, and expectations. In Medin, D. (Ed.) Psychology of learning and motivation, (Vol. 40 pp. 223–278): Elsevier.

  • Bland, A.R., & Schaefer, A. (2012). Different varieties of uncertainty in human decision-making. Frontiers in Neuroscience, 6, 85.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bortz, D.M., Gazo, K.L., & Grace, A.A. (2019). The medial septum enhances reversal learning via opposing actions on ventral tegmental area and substantia nigra dopamine neurons. Neuropsychopharmacology, 1–9.

  • Bortz, D.M., & Grace, A.A. (2018). Medial septum differentially regulates dopamine neuron activity in the rat ventral tegmental area and substantia nigra via distinct pathways. Neuropsychopharmacology, 43, 2093–2100.

    Article  PubMed  PubMed Central  Google Scholar 

  • Braganza, O., & Beck, H. (2018). The circuit motif as a conceptual tool for multilevel neuroscience. Trends in Neurosciences, 41(3), 128–136.

    Article  PubMed  Google Scholar 

  • Bromberg-Martin, E.S., Matsumoto, M., & Hikosaka, O. (2010). Dopamine in motivational control: rewarding, aversive, and alerting. Neuron, 68(5), 815–834.

    Article  PubMed  PubMed Central  Google Scholar 

  • Brown, J., Bullock, D., & Grossberg, S. (1999). How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. Journal of Neuroscience, 19(23), 10502–10511.

    Article  PubMed  Google Scholar 

  • Bush, R.R., & Mosteller, F. (1951). A model for stimulus generalization and discrimination. Psychological Review, 58(6), 413– 423.

    Article  PubMed  Google Scholar 

  • Cantwell, G., Riesenhuber, M., Roeder, J.L., & Ashby, F.G. (2017). Perceptual category learning and visual processing: an exercise in computational cognitive neuroscience. Neural Networks, 89, 31–38.

    Article  PubMed  PubMed Central  Google Scholar 

  • Christopoulos, G.I., Tobler, P.N., Bossaerts, P., Dolan, R.J., & Schultz, W. (2009). Neural correlates of value, risk, and risk aversion contributing to decision making under risk. Journal of Neuroscience, 29 (40), 12574–12583.

    Article  PubMed  Google Scholar 

  • Cohen, J.Y., Haesler, S., Vong, L., Lowell, B.B., & Uchida, N. (2012). Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature, 482(7383), 85–88.

    Article  PubMed  PubMed Central  Google Scholar 

  • Contreras-Vidal, J.L., & Schultz, W. (1999). A predictive reinforcement model of dopamine neurons for learning approach behavior. Journal of Computational Neuroscience, 6(3), 191–214.

    Article  PubMed  Google Scholar 

  • Cools, R. (2006). Dopaminergic modulation of cognitive function-implications for L-DOPA treatment in Parkinson’s disease. Neuroscience and Biobehavioral Reviews, 30(1), 1–23.

    Article  PubMed  Google Scholar 

  • Cools, R., Clark, L., Owen, A.M., & Robbins, T.W. (2002). Defining the neural mechanisms of probabilistic reversal learning using event-related functional magnetic resonance imaging. Journal of Neuroscience, 22(11), 4563–4567.

    Article  PubMed  Google Scholar 

  • Cools, R., & D’Esposito, M. (2011). Inverted U-shaped dopamine actions on human working memory and cognitive control. Biological Psychiatry, 69(12), e113–e125.

    Article  PubMed  PubMed Central  Google Scholar 

  • Cools, R., & Robbins, T.W. (2004). Chemistry of the adaptive mind. Philosophical Transactions of the Royal Society of London. Series A: Mathematical. Physical and Engineering Sciences, 362(1825), 2871–2888.

    Article  Google Scholar 

  • Cornwall, J., & Phillipson, O. (1988). Afferent projections to the parafascicular thalamic nucleus of the rat, as shown by the retrograde transport of wheat germ agglutinin. Brain Research Bulletin, 20(2), 139–150.

    Article  PubMed  Google Scholar 

  • Crossley, M.J., Ashby, F.G., & Maddox, W.T. (2013). Erasing the engram: the unlearning of procedural skills. Journal of Experimental Psychology: General, 142(3), 710–741.

    Article  Google Scholar 

  • Daw, N.D., & O’Doherty, J.P. (2014). Multiple systems for value learning. In P. W. Glimcher, & E. Fehr (Eds.) Neuroeconomics: decision making and the brain, Second edition (pp. 393–410). Amsterdam: Elsevier.

  • Dayan, P., & Abbott, L.F. (2001). Theoretical neuroscience: computational and mathematical modeling of neural systems. Cambridge: MIT Press.

    Google Scholar 

  • Dayan, P., Kakade, S., & Montague, P.R. (2000). Learning and selective attention. Nature Neuroscience, 3(11s), 1218–1223.

    Article  PubMed  Google Scholar 

  • Dayan, P., & Long, T. (1998). Statistical models of conditioning. In Jordan, M. I., Kearns, M. J., & Solla, S. A. (Eds.) Advances in neural information processing systems: Proceedings of the 1997 Conference (pp. 117–123). Cambridge, MA: MIT Press.

  • Dayan, P., & Yu, A.J. (2003). Expected and unexpected uncertainty: ACh and NE in the neocortex. In Becker, S., Thrun, S., & Obermayer, K. (Eds.) Advances in neural information processing systems: Proceedings of the 2002 Conference (pp. 173–180). Cambridge, MA: MIT Press.

  • Deng, P., Zhang, Y., & Xu, Z.C. (2007). Involvement of Ih in dopamine modulation of tonic firing in striatal cholinergic interneurons. Journal of Neuroscience, 27(12), 3148–3156.

    Article  PubMed  Google Scholar 

  • Ding, J.B., Guzman, J.N., Peterson, J.D., Goldberg, J.A., & Surmeier, D.J. (2010). Thalamic gating of corticostriatal signaling by cholinergic interneurons. Neuron, 67(2), 294–307c.

    Article  PubMed  PubMed Central  Google Scholar 

  • Doig, N.M., Magill, P.J., Apicella, P., Bolam, J.P., & Sharott, A. (2014). Cortical and thalamic excitation mediate the multiphasic responses of striatal cholinergic interneurons to motivationally salient stimuli. Journal of Neuroscience, 34(8), 3101–3117.

    Article  PubMed  Google Scholar 

  • Ermentrout, G.B. (1996). Type I membranes, phase resetting curves, and synchrony. Neural Computation, 8 (5), 979–1001.

    Article  PubMed  Google Scholar 

  • Eshel, N., Bukwich, M., Rao, V., Hemmelder, V., Tian, J., & Uchida, N. (2015). Arithmetic and local circuitry underlying dopamine prediction errors. Nature, 525(7568), 243–246.

    Article  PubMed  PubMed Central  Google Scholar 

  • Fabbricatore, A.T., Ghitza, U.E., Prokopenko, V.F., & West, M.O. (2009). Electrophysiological evidence of mediolateral functional dichotomy in the rat accumbens during cocaine self-administration: tonic firing patterns. European Journal of Neuroscience, 30(12), 2387–2400.

    Article  Google Scholar 

  • Faget, L., Osakada, F., Duan, J., Ressler, R., Johnson, A.B., Proudfoot, J.A., & Hnasko, T. S. (2016). Afferent inputs to neurotransmitter-defined cell types in the ventral tegmental area. Cell reports, 15(12), 2796–2808.

    Article  PubMed  Google Scholar 

  • Fanselow, M.S., & Dong, H.W. (2010). Are the dorsal and ventral hippocampus functionally distinct structures? Neuron, 65(1), 7–19.

    Article  PubMed  PubMed Central  Google Scholar 

  • Farashahi, S., Donahue, C.H., Khorsand, P., Seo, H., Lee, D., & Soltani, A. (2017). Metaplasticity as a neural substrate for adaptive learning and choice under uncertainty. Neuron, 94(2), 401–414.

    Article  PubMed  PubMed Central  Google Scholar 

  • Franklin, N.T., & Frank, M.J. (2015). A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning. Elife, 4.

  • Friston, K.J., Shiner, T., FitzGerald, T., Galea, J.M., Adams, R., Brown, H., & Bestmann, S. (2012). Dopamine, affordance and active inference. PLoS Computational Biology, 8(1).

  • Gloor, P. (1997). The temporal lobe and limbic system. New York: Oxford University Press.

    Google Scholar 

  • Grace, A.A. (2010). Dopamine system dysregulation by the ventral subiculum as the common pathophysiological basis for schizophrenia psychosis, psychostimulant abuse, and stress. Neurotoxicity Research, 18(3-4), 367–376.

    Article  PubMed  PubMed Central  Google Scholar 

  • Grace, A.A., & Bunney, B.S. (1983). Intracellular and extracellular electrophysiology of nigral dopaminergic neurons-1. Identification and characterization. Neuroscience, 10(2), 301–315.

    Article  PubMed  Google Scholar 

  • Grace, A.A., Floresco, S.B., Goto, Y., & Lodge, D.J. (2007). Regulation of firing of dopaminergic neurons and control of goal-directed behaviors. Trends in Neurosciences, 30(5), 220–227.

    Article  PubMed  Google Scholar 

  • Haber, S.N. (2016). Corticostriatal circuitry. Dialogues in Clinical Neuroscience, 18(1), 7.

    Article  PubMed  PubMed Central  Google Scholar 

  • Haber, S.N., Fudge, J.L., & McFarland, N.R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. Journal of Neuroscience, 20(6), 2369–2382.

    Article  PubMed  Google Scholar 

  • Harrison, L.M., Duggins, A., & Friston, K.J. (2006). Encoding uncertainty in the hippocampus. Neural Networks, 19(5), 535–546.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hart, A.S., Rutledge, R.B., Glimcher, P.W., & Phillips, P.E. (2014). Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. Journal of Neuroscience, 34(3), 698–704.

    Article  PubMed  Google Scholar 

  • Hazy, T.E., & Frank, M.J. (2010). O’Reilly, R. C Neural mechanisms of acquired phasic dopamine responses in learning. Neuroscience and Biobehavioral Reviews, 34(5), 701–720.

    Article  PubMed  Google Scholar 

  • Hong, S., & Hikosaka, O. (2014). Pedunculopontine tegmental nucleus neurons provide reward, sensorimotor, and alerting signals to midbrain dopamine neurons. Neuroscience, 282, 139–155.

    Article  PubMed  Google Scholar 

  • Hong, S., Jhou, T.C., Smith, M., Saleem, K.S., & Hikosaka, O. (2011). Negative reward signals from the lateral habenula to dopamine neurons are mediated by rostromedial tegmental nucleus in primates. Journal of Neuroscience, 31(32), 11457–11471.

    Article  PubMed  Google Scholar 

  • Horvitz, J.C. (2002). Dopamine gating of glutamatergic sensorimotor and incentive motivational input signals to the striatum. Behavioural Brain Research, 137(1-2), 65–74.

    Article  PubMed  Google Scholar 

  • Houk, J., Adams, J., & Barto, A. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In Davis, J.L., Beiser, D.G., & Houk J.C. (Eds.) Models of information processing in the basal ganglia (pp. 249–270). Cambridge: MIT Press.

  • Huettel, S.A., Song, A.W., & McCarthy, G. (2005). Decisions under uncertainty: probabilistic context influences activation of prefrontal and parietal cortices. Journal of Neuroscience, 25(13), 3304–3311.

    Article  PubMed  Google Scholar 

  • Humphries, M.D., & Prescott, T.J. (2010). The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Progress in Neurobiology, 90(4), 385–417.

    Article  PubMed  Google Scholar 

  • Iigaya, K. (2016). Adaptive learning and decision-making under uncertainty by metaplastic synapses guided by a surprise detection system. Elife, 5, e18073.

    Article  PubMed  PubMed Central  Google Scholar 

  • Insausti, R., Amaral, D., & Cowan, W. (1987). The entorhinal cortex of the monkey: II. Cortical afferents. Journal of Comparative Neurology, 264(3), 356–395.

    Article  Google Scholar 

  • Izhikevich, E.M. (2003). Simple model of spiking neurons. IEEE Transactions on Neural Networks, 14(6), 1569–1572.

    Article  PubMed  Google Scholar 

  • Izhikevich, E.M. (2007). Dynamical systems in neuroscience. Cambridge, CA: MIT Press.

    Google Scholar 

  • Jacobs, J., Kahana, M.J., Ekstrom, A.D., Mollison, M.V., & Fried, I. (2010). A sense of direction in human entorhinal cortex. Proceedings of the National Academy of Sciences, 107(14), 6487–6492.

    Article  Google Scholar 

  • Jhou, T.C., Fields, H.L., Baxter, M.G., Saper, C.B., & Holland, P.C. (2009). The rostromedial tegmental nucleus (RMTg), a GABAergic afferent to midbrain dopamine neurons, encodes aversive stimuli and inhibits motor responses. Neuron, 61(5), 786–800.

    Article  PubMed  PubMed Central  Google Scholar 

  • Jo, S., & Jung, M.W. (2016). Differential coding of uncertain reward in rat insular and orbitofrontal cortex. Scientific Reports, 6, 24085.

    Article  PubMed  PubMed Central  Google Scholar 

  • Joel, D., Niv, Y., & Ruppin, E. (2002). Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks, 15(4-6), 535–547.

    Article  PubMed  Google Scholar 

  • Jones, B.F., & Witter, M.P. (2007). Cingulate cortex projections to the parahippocampal region and hippocampal formation in the rat. Hippocampus, 17(10), 957–976.

    Article  PubMed  Google Scholar 

  • Kawato, M., & Samejima, K. (2007). Efficient reinforcement learning: computational theories, neuroscience and robotics. Current Opinion in Neurobiology, 17(2), 205–212.

    Article  PubMed  Google Scholar 

  • Keiflin, R., Pribut, H.J., Shah, N.B., & Janak, P.H. (2019). Ventral tegmental dopamine neurons participate in reward identity predictions. Current Biology, 29(1), 93–103.

    Article  PubMed  Google Scholar 

  • Kerr, K.M., Agster, K.L., Furtak, S.C., & Burwell, R.D. (2007). Functional neuroanatomy of the parahippocampal region: the lateral and medial entorhinal areas. Hippocampus, 17(9), 697–708.

    Article  PubMed  Google Scholar 

  • Kobayashi, Y., & Okada, K. (2007). Reward prediction error computation in the pedunculopontine tegmental nucleus neurons. Annals of the New York Academy of Sciences, 1104(1), 310–323.

    Article  PubMed  Google Scholar 

  • Kumaran, D., & Maguire, E.A. (2006). An unexpected sequence of events: mismatch detection in the human hippocampus. PLoS Biology, 4(12), e424.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lipski, W.J., & Grace, A.A. (2013). Activation and inhibition of neurons in the hippocampal ventral subiculum by norepinephrine and locus coeruleus stimulation. Neuropsychopharmacology, 38(2), 285.

    Article  PubMed  Google Scholar 

  • Liu, X., Hairston, J., Schrier, M., & Fan, J. (2011). Common and distinct networks underlying reward valence and processing stages: a meta-analysis of functional neuroimaging studies. Neuroscience and Biobehavioral Reviews, 35(5), 1219–1236.

    Article  PubMed  Google Scholar 

  • Lodge, D.J., & Grace, A.A. (2006). The hippocampus modulates dopamine neuron responsivity by regulating the intensity of phasic neuron activation. Neuropsychopharmacology, 31(7), 1356–1361.

    Article  PubMed  Google Scholar 

  • Maia, T.V. (2009). Reinforcement learning, conditioning, and the brain: successes and challenges. Cognitive, Affective, and Behavioral Neuroscience, 9(4), 343–364.

    Article  Google Scholar 

  • Marr, D. (1982). Vision: a computational investigation into the human representation and processing of visual information. New York: Freeman.

    Google Scholar 

  • Mathys, C., Daunizeau, J., Friston, K.J., & Stephan, K.E. (2011). A Bayesian foundation for individual learning under uncertainty. Frontiers in Human Neuroscience, 5, 39.

    Article  PubMed  PubMed Central  Google Scholar 

  • Matsumoto, M., & Hikosaka, O. (2007). Lateral habenula as a source of negative reward signals in dopamine neurons. Nature, 447(7148), 1111–1115.

    Article  PubMed  Google Scholar 

  • Matsumoto, M., & Hikosaka, O. (2009). Representation of negative motivational value in the primate lateral habenula. Nature Neuroscience, 12(1), 77–84.

    Article  PubMed  Google Scholar 

  • Matsumoto, N., Minamimoto, T., Graybiel, A.M., & Kimura, M. (2001). Neurons in the thalamic CM-Pf complex supply striatal neurons with information about behaviorally significant sensory events. Journal of Neurophysiology, 85(2), 960–976.

    Article  PubMed  Google Scholar 

  • Mishkin, M., Malamut, B., & Bachevalier, J. (1984). Memories and habits: two neural systems. In Lynch, G., McGaugh, J. L., & Weinberger, N. M. (Eds.) Neurobiology of human learning and memory (pp. 65–77). New York: Guilford Press.

  • Monosov, I.E. (2017). Anterior cingulate is a source of valence-specific information about value and uncertainty. Nature Communications, 8(1), 134.

    Article  PubMed  PubMed Central  Google Scholar 

  • Montague, P.R., Dayan, P., & Sejnowski, T.J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience, 16(5), 1936–1947.

    Article  PubMed  Google Scholar 

  • Morita, K., Morishima, M., Sakai, K., & Kawaguchi, Y. (2012). Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways. Trends in Neurosciences, 35(8), 457–467.

    Article  PubMed  Google Scholar 

  • Morita, K., Morishima, M., Sakai, K., & Kawaguchi, Y. (2013). Dopaminergic control of motivation and reinforcement learning: a closed-circuit account for reward-oriented behavior. Journal of Neuroscience, 33(20), 8866–8890.

    Article  PubMed  Google Scholar 

  • Niv, Y., Daw, N.D., Joel, D., & Dayan, P. (2007). Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology, 191(3), 507–520.

    Article  PubMed  Google Scholar 

  • Okada, K., & Kobayashi, Y. (2013). Reward prediction-related increases and decreases in tonic neuronal activity of the pedunculopontine tegmental nucleus. Frontiers in Integrative Neuroscience, 7, 36.

    Article  PubMed  PubMed Central  Google Scholar 

  • Neill, M., & Schultz, W. (2010). Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value. Neuron, 68(4), 789–800.

    Article  Google Scholar 

  • O’Reilly, R.C., Frank, M.J., Hazy, T.E., & Watz, B. (2007). PVLV: the primary value and learned value Pavlovian learning algorithm. Behavioral Neuroscience, 121(1), 31–49.

    Article  PubMed  Google Scholar 

  • Payzan-LeNestour, E., & Bossaerts, P. (2011). Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Computational Biology, 7(1), e1001048.

    Article  PubMed  PubMed Central  Google Scholar 

  • Payzan-LeNestour, E., Dunne, S., Bossaerts, P., & O’Doherty, J.P. (2013). The neural representation of unexpected uncertainty during value-based decision making. Neuron, 79(1), 191–201.

    Article  PubMed  PubMed Central  Google Scholar 

  • Pickering, A.D., & Pesola, F. (2014). Modeling dopaminergic and other processes involved in learning from reward prediction error: contributions from an individual differences perspective. Frontiers in Human Neuroscience, 8, 740.

    PubMed  PubMed Central  Google Scholar 

  • Preuschoff, K., & Bossaerts, P. (2007). Adding prediction risk to the theory of reward learning. Annals of the New York Academy of Sciences, 1104(1), 135–146.

    Article  PubMed  Google Scholar 

  • Preuschoff, K., Quartz, S.R., & Bossaerts, P. (2008). Human insula activation reflects risk prediction errors as well as risk. Journal of Neuroscience, 28(11), 2745–2752.

    Article  PubMed  Google Scholar 

  • Quintero, E., Diaz, E., Vargas, J.P., de la Casa, G., & Lopez, J.C. (2011). Ventral subiculum involvement in latent inhibition context specificity. Physiology and Behavior, 102(3-4), 414–420.

    Article  PubMed  Google Scholar 

  • Rall, W. (1967). Distinguishing theoretical synaptic potentials computed for different soma-dendritic distributions of synaptic input. Journal of Neurophysiology, 30(5), 1138–1168.

    Article  PubMed  Google Scholar 

  • Rescorla, R.A., & Wagner, A.R. (1972). A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In Black, A. H., & Prokasy, W. F. (Eds.) Classical conditioning II: current research and theory (pp. 64–99). New York: Appleton-Century-Crofts.

  • Riceberg, J.S., & Shapiro, M.L. (2012). Reward stability determines the contribution of orbitofrontal cortex to adaptive behavior. Journal of Neuroscience, 32(46), 16402–16409.

    Article  PubMed  Google Scholar 

  • Root, D.H., Fabbricatore, A.T., Pawlak, A.P., Barker, D.J., Ma, S., & West, M.O. (2012). Slow phasic and tonic activity of ventral pallidal neurons during cocaine self-administration. Synapse, 66(2), 106–127.

    Article  PubMed  Google Scholar 

  • Rushworth, M.F., & Behrens, T.E. (2008). Choice, uncertainty and value in prefrontal and cingulate cortex. Nature Neuroscience, 11(4), 389–397.

    Article  PubMed  Google Scholar 

  • Rutishauser, U., Mamelak, A.N., & Schuman, E.M. (2006). Single-trial learning of novel stimuli by individual neurons of the human hippocampus-amygdala complex. Neuron, 49(6), 805– 813.

    Article  PubMed  Google Scholar 

  • Sadikot, A., Parent, A., & Francois, C. (1992). Efferent connections of the centromedian and parafascicular thalamic nuclei in the squirrel monkey: a PHA-L study of subcortical projections. Journal of Comparative Neurology, 315(2), 137–159.

    Article  Google Scholar 

  • Salum, C., da Silva, A.R., & Pickering, A. (1999). Striatal dopamine in attentional learning: a computational model. Neurocomputing, 26, 845–854.

    Article  Google Scholar 

  • Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80(1), 1–27.

    Article  PubMed  Google Scholar 

  • Schultz, W., Dayan, P., & Montague, P.R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599.

    Article  PubMed  Google Scholar 

  • Seamans, J.K., & Robbins, T.W. (2010). Dopamine modulation of the prefrontal cortex and cognitive function. In Neve, K. A. (Ed.) The dopamine receptors. 2nd edn. (pp. 373–398). New York: Springer.-.

  • Sesack, S.R., & Grace, A.A. (2010). Cortico-basal ganglia reward network: microcircuitry. Neuropsychopharmacology, 35(1), 27–47.

    Article  PubMed  Google Scholar 

  • Smith, Y., & Kieval, J.Z. (2000). Anatomy of the dopamine system in the basal ganglia. Trends in Neurosciences, 23, S28–S33.

    Article  PubMed  Google Scholar 

  • Soltani, A., & Izquierdo, A. (2019). Adaptive learning under expected and unexpected uncertainty. Nature Reviews Neuroscience, 20(10), 635–644.

    Article  PubMed  PubMed Central  Google Scholar 

  • Strange, B.A., Duggins, A., Penny, W., Dolan, R.J., & Friston, K.J. (2005). Information theory, novelty and hippocampal responses: unpredicted or unpredictable? Neural Networks, 18(3), 225–230.

    Article  PubMed  Google Scholar 

  • Stuber, G.D., Klanker, M., De Ridder, B., Bowers, M.S., Joosten, R.N., Feenstra, M.G., & Bonci, A. (2008). Reward predictive cues enhance excitatory synaptic strength onto midbrain dopamine neurons. Science, 321(5896), 1690–1692.

    Article  PubMed  PubMed Central  Google Scholar 

  • Sutton, R.S. (1992). Adapting bias by gradient descent: an incremental version of delta-bar-delta. In Proceedings of the tenth national conference on artificial intelligence (pp. 171–176). Cambridge: MIT Press.

  • Sutton, R.S., & Barto, A.G. (1998). Reinforcement learning: an introduction Cambridge. MA: MIT Press.

    Google Scholar 

  • Takahashi, Y.K., Langdon, A.J., Niv, Y., & Schoenbaum, G. (2016). Temporal specificity of reward prediction errors signaled by putative dopamine neurons in rat VTA depends on ventral striatum. Neuron, 91(1), 182–193.

    Article  PubMed  PubMed Central  Google Scholar 

  • Takahashi, Y.K., Schoenbaum, G., & Niv, Y. (2008). Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Frontiers in neuroscience, 2, 14.

    Article  Google Scholar 

  • Tan, C.O., & Bullock, D. (2008). A local circuit model of learned striatal and dopamine cell responses under probabilistic schedules of reward. Journal of Neuroscience, 28(40), 10062–10074.

    Article  PubMed  Google Scholar 

  • Taswell, C.A., Costa, V.D., Murray, E.A., & Averbeck, B.B. (2018). Ventral striatum’s role in learning from gains and losses. Proceedings of the National Academy of Sciences, 115(52), E12398–E12406.

    Article  Google Scholar 

  • Tian, J., & Uchida, N. (2015). Habenula lesions reveal that multiple mechanisms underlie dopamine prediction errors. Neuron, 87(6), 1304–1316.

    Article  PubMed  PubMed Central  Google Scholar 

  • Vanni-Mercier, G., Mauguiere, F., Isnard, J., & Dreher, J.-C. (2009). The hippocampus codes the uncertainty of cue-outcome associations: an intracranial electrophysiological study in humans. Journal of Neuroscience, 29(16), 5287–5294.

    Article  PubMed  Google Scholar 

  • Van Rossum, G., & Drake, F.L. (2011). The Python language reference manual. Network Theory Ltd.

  • Vitay, J., & Hamker, F.H. (2014). Timing and expectation of reward: a neuro-computational model of the afferents to the ventral tegmental area. Frontiers in Neurorobotics, 8, 4.

    Article  PubMed  PubMed Central  Google Scholar 

  • Watabe-Uchida, M., Zhu, L., Ogawa, S.K., Vamanrao, A., & Uchida, N. (2012). Whole-brain mapping of direct inputs to midbrain dopamine neurons. Neuron, 74(5), 858–873.

    Article  PubMed  Google Scholar 

  • Willingham, D.B. (1998). A neuropsychological theory of motor skill learning. Psychological Review, 105, 558–584.

    Article  PubMed  Google Scholar 

  • Yu, A.J., & Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron, 46(4), 681–692.

    Article  PubMed  Google Scholar 

Download references

Funding

This research was supported by NIH Grant 2R01MH063760.

Author information

Authors and Affiliations

  1. Interdepartmental Graduate Program in Dynamical Neuroscience, University of California, Santa Barbara, CA, 93106, USA

    Jeffrey B. Inglis

  2. Department of Psychological & Brain Sciences, University of California, Santa Barbara, CA, 93106, USA

    Vivian V. Valentin & F. Gregory Ashby

Authors
  1. Jeffrey B. Inglis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vivian V. Valentin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. F. Gregory Ashby
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to F. Gregory Ashby.

Ethics declarations

Competing Interests

The authors declare no that they no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Inglis, J.B., Valentin, V.V. & Ashby, F.G. Modulation of Dopamine for Adaptive Learning: a Neurocomputational Model. Comput Brain Behav 4, 34–52 (2021). https://doi.org/10.1007/s42113-020-00083-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42113-020-00083-x

Keywords

Search

Navigation