Abstract
There have been many proposals that learning rates in the brain are adaptive, in the sense that they increase or decrease depending on environmental conditions. The majority of these models are abstract and make no attempt to describe the neural circuitry that implements the proposed computations. This article describes a biologically detailed computational model that overcomes this shortcoming. Specifically, we propose a neural circuit that implements adaptive learning rates by modulating the gain on the dopamine response to reward prediction errors, and we model activity within this circuit at the level of spiking neurons. The model generates a dopamine signal that depends on the size of the tonically active dopamine neuron population and the phasic spike rate. The model was tested successfully against results from two single-neuron recording studies and a fast-scan cyclic voltammetry study. We conclude by discussing the general applicability of the model to dopamine-mediated tasks that transcend the experimental phenomena it was initially designed to address.
Similar content being viewed by others
Notes
However, for Figs. 3, 4, 5 (left and center), and 6, the PPTN square wave lasted 1000 ms and the LH square wave lasted a maximum of 1000 ms. This was done to ensure a sufficiently long interval to extract accurate measurements of firing rate and active population size. Figures showing dopamine output used the parameters described in the text.
References
Aminoff, E.M., Kveraga, K., & Bar, M. (2013). The role of the parahippocampal cortex in cognition. Trends in Cognitive Sciences, 17(8), 379–390.
Ashby, F.G. (2018). Computational cognitive neuroscience. In Batchelder, W., Colonius, H., Dzhafarov, E., & Myung, J. (Eds.) New handbook of mathematical psychology, vol. 2 (pp. 223–270). New York. New York: Cambridge University Press.
Ashby, F.G., & Crossley, M.J. (2011). A computational model of how cholinergic interneurons protect striatal-dependent learning. Journal of Cognitive Neuroscience, 23(6), 1549–1566.
Ashby, F.G., & Ennis, J.M. (2006). The role of the basal ganglia in category learning. Psychology of Learning and Motivation, 46, 1–36.
Ashby, F.G., & Helie, S. (2011). A tutorial on computational cognitive neuroscience: modeling the neurodynamics of cognition. Journal of Mathematical Psychology, 55(4), 273–289.
Ashby, F.G., Isen, A.M., & Turken, A. (1999). A neuropsychological theory of positive affect and its influence on cognition. Psychological Review, 106(3), 529–550.
Ashby, F.G., Valentin, V.V., & Turken, A.U. (2002). The effects of positive affect and arousal and working memory and executive attention: neurobiology and computational models. In Moore, S., & Oaksford, M. (Eds.) Emotional cognition: from brain to behaviour (pp. 245–287). Amsterdam: John Benjamins Publishing Company.
Ashby, F.G., & Vucovich, L.E. (2016). The role of feedback contingency in perceptual category learning. Journal of Experimental Psychology: Learning. Memory, and Cognition, 42(11), 1731.
Bayer, H.M., & Glimcher, P.W. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47(1), 129–141.
Bayer, H.M., Lau, B., & Glimcher, P.W. (2007). Statistics of midbrain dopamine neuron spike trains in the awake primate. Journal of Neurophysiology, 98(3), 1428–1439.
Behrens, T.E., Woolrich, M.W., Walton, M.E., & Rushworth, M.F. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10(9), 1214–1221.
Belin, D., & Everitt, B.J. (2008). Cocaine seeking habits depend upon dopamine-dependent serial connectivity linking the ventral with the dorsal striatum. Neuron, 57(3), 432–441.
Berke, J.D. (2018). What does dopamine mean? Nature Neuroscience, 21(6), 787–793.
Bernacchia, A., Seo, H., Lee, D., & Wang, X. -J. (2011). A reservoir of time constants for memory traces in cortical neurons. Nature Neuroscience, 14(3), 366–372.
Berridge, K.C. (2000). Reward learning: reinforcement, incentives, and expectations. In Medin, D. (Ed.) Psychology of learning and motivation, (Vol. 40 pp. 223–278): Elsevier.
Bland, A.R., & Schaefer, A. (2012). Different varieties of uncertainty in human decision-making. Frontiers in Neuroscience, 6, 85.
Bortz, D.M., Gazo, K.L., & Grace, A.A. (2019). The medial septum enhances reversal learning via opposing actions on ventral tegmental area and substantia nigra dopamine neurons. Neuropsychopharmacology, 1–9.
Bortz, D.M., & Grace, A.A. (2018). Medial septum differentially regulates dopamine neuron activity in the rat ventral tegmental area and substantia nigra via distinct pathways. Neuropsychopharmacology, 43, 2093–2100.
Braganza, O., & Beck, H. (2018). The circuit motif as a conceptual tool for multilevel neuroscience. Trends in Neurosciences, 41(3), 128–136.
Bromberg-Martin, E.S., Matsumoto, M., & Hikosaka, O. (2010). Dopamine in motivational control: rewarding, aversive, and alerting. Neuron, 68(5), 815–834.
Brown, J., Bullock, D., & Grossberg, S. (1999). How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. Journal of Neuroscience, 19(23), 10502–10511.
Bush, R.R., & Mosteller, F. (1951). A model for stimulus generalization and discrimination. Psychological Review, 58(6), 413– 423.
Cantwell, G., Riesenhuber, M., Roeder, J.L., & Ashby, F.G. (2017). Perceptual category learning and visual processing: an exercise in computational cognitive neuroscience. Neural Networks, 89, 31–38.
Christopoulos, G.I., Tobler, P.N., Bossaerts, P., Dolan, R.J., & Schultz, W. (2009). Neural correlates of value, risk, and risk aversion contributing to decision making under risk. Journal of Neuroscience, 29 (40), 12574–12583.
Cohen, J.Y., Haesler, S., Vong, L., Lowell, B.B., & Uchida, N. (2012). Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature, 482(7383), 85–88.
Contreras-Vidal, J.L., & Schultz, W. (1999). A predictive reinforcement model of dopamine neurons for learning approach behavior. Journal of Computational Neuroscience, 6(3), 191–214.
Cools, R. (2006). Dopaminergic modulation of cognitive function-implications for L-DOPA treatment in Parkinson’s disease. Neuroscience and Biobehavioral Reviews, 30(1), 1–23.
Cools, R., Clark, L., Owen, A.M., & Robbins, T.W. (2002). Defining the neural mechanisms of probabilistic reversal learning using event-related functional magnetic resonance imaging. Journal of Neuroscience, 22(11), 4563–4567.
Cools, R., & D’Esposito, M. (2011). Inverted U-shaped dopamine actions on human working memory and cognitive control. Biological Psychiatry, 69(12), e113–e125.
Cools, R., & Robbins, T.W. (2004). Chemistry of the adaptive mind. Philosophical Transactions of the Royal Society of London. Series A: Mathematical. Physical and Engineering Sciences, 362(1825), 2871–2888.
Cornwall, J., & Phillipson, O. (1988). Afferent projections to the parafascicular thalamic nucleus of the rat, as shown by the retrograde transport of wheat germ agglutinin. Brain Research Bulletin, 20(2), 139–150.
Crossley, M.J., Ashby, F.G., & Maddox, W.T. (2013). Erasing the engram: the unlearning of procedural skills. Journal of Experimental Psychology: General, 142(3), 710–741.
Daw, N.D., & O’Doherty, J.P. (2014). Multiple systems for value learning. In P. W. Glimcher, & E. Fehr (Eds.) Neuroeconomics: decision making and the brain, Second edition (pp. 393–410). Amsterdam: Elsevier.
Dayan, P., & Abbott, L.F. (2001). Theoretical neuroscience: computational and mathematical modeling of neural systems. Cambridge: MIT Press.
Dayan, P., Kakade, S., & Montague, P.R. (2000). Learning and selective attention. Nature Neuroscience, 3(11s), 1218–1223.
Dayan, P., & Long, T. (1998). Statistical models of conditioning. In Jordan, M. I., Kearns, M. J., & Solla, S. A. (Eds.) Advances in neural information processing systems: Proceedings of the 1997 Conference (pp. 117–123). Cambridge, MA: MIT Press.
Dayan, P., & Yu, A.J. (2003). Expected and unexpected uncertainty: ACh and NE in the neocortex. In Becker, S., Thrun, S., & Obermayer, K. (Eds.) Advances in neural information processing systems: Proceedings of the 2002 Conference (pp. 173–180). Cambridge, MA: MIT Press.
Deng, P., Zhang, Y., & Xu, Z.C. (2007). Involvement of Ih in dopamine modulation of tonic firing in striatal cholinergic interneurons. Journal of Neuroscience, 27(12), 3148–3156.
Ding, J.B., Guzman, J.N., Peterson, J.D., Goldberg, J.A., & Surmeier, D.J. (2010). Thalamic gating of corticostriatal signaling by cholinergic interneurons. Neuron, 67(2), 294–307c.
Doig, N.M., Magill, P.J., Apicella, P., Bolam, J.P., & Sharott, A. (2014). Cortical and thalamic excitation mediate the multiphasic responses of striatal cholinergic interneurons to motivationally salient stimuli. Journal of Neuroscience, 34(8), 3101–3117.
Ermentrout, G.B. (1996). Type I membranes, phase resetting curves, and synchrony. Neural Computation, 8 (5), 979–1001.
Eshel, N., Bukwich, M., Rao, V., Hemmelder, V., Tian, J., & Uchida, N. (2015). Arithmetic and local circuitry underlying dopamine prediction errors. Nature, 525(7568), 243–246.
Fabbricatore, A.T., Ghitza, U.E., Prokopenko, V.F., & West, M.O. (2009). Electrophysiological evidence of mediolateral functional dichotomy in the rat accumbens during cocaine self-administration: tonic firing patterns. European Journal of Neuroscience, 30(12), 2387–2400.
Faget, L., Osakada, F., Duan, J., Ressler, R., Johnson, A.B., Proudfoot, J.A., & Hnasko, T. S. (2016). Afferent inputs to neurotransmitter-defined cell types in the ventral tegmental area. Cell reports, 15(12), 2796–2808.
Fanselow, M.S., & Dong, H.W. (2010). Are the dorsal and ventral hippocampus functionally distinct structures? Neuron, 65(1), 7–19.
Farashahi, S., Donahue, C.H., Khorsand, P., Seo, H., Lee, D., & Soltani, A. (2017). Metaplasticity as a neural substrate for adaptive learning and choice under uncertainty. Neuron, 94(2), 401–414.
Franklin, N.T., & Frank, M.J. (2015). A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning. Elife, 4.
Friston, K.J., Shiner, T., FitzGerald, T., Galea, J.M., Adams, R., Brown, H., & Bestmann, S. (2012). Dopamine, affordance and active inference. PLoS Computational Biology, 8(1).
Gloor, P. (1997). The temporal lobe and limbic system. New York: Oxford University Press.
Grace, A.A. (2010). Dopamine system dysregulation by the ventral subiculum as the common pathophysiological basis for schizophrenia psychosis, psychostimulant abuse, and stress. Neurotoxicity Research, 18(3-4), 367–376.
Grace, A.A., & Bunney, B.S. (1983). Intracellular and extracellular electrophysiology of nigral dopaminergic neurons-1. Identification and characterization. Neuroscience, 10(2), 301–315.
Grace, A.A., Floresco, S.B., Goto, Y., & Lodge, D.J. (2007). Regulation of firing of dopaminergic neurons and control of goal-directed behaviors. Trends in Neurosciences, 30(5), 220–227.
Haber, S.N. (2016). Corticostriatal circuitry. Dialogues in Clinical Neuroscience, 18(1), 7.
Haber, S.N., Fudge, J.L., & McFarland, N.R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. Journal of Neuroscience, 20(6), 2369–2382.
Harrison, L.M., Duggins, A., & Friston, K.J. (2006). Encoding uncertainty in the hippocampus. Neural Networks, 19(5), 535–546.
Hart, A.S., Rutledge, R.B., Glimcher, P.W., & Phillips, P.E. (2014). Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. Journal of Neuroscience, 34(3), 698–704.
Hazy, T.E., & Frank, M.J. (2010). O’Reilly, R. C Neural mechanisms of acquired phasic dopamine responses in learning. Neuroscience and Biobehavioral Reviews, 34(5), 701–720.
Hong, S., & Hikosaka, O. (2014). Pedunculopontine tegmental nucleus neurons provide reward, sensorimotor, and alerting signals to midbrain dopamine neurons. Neuroscience, 282, 139–155.
Hong, S., Jhou, T.C., Smith, M., Saleem, K.S., & Hikosaka, O. (2011). Negative reward signals from the lateral habenula to dopamine neurons are mediated by rostromedial tegmental nucleus in primates. Journal of Neuroscience, 31(32), 11457–11471.
Horvitz, J.C. (2002). Dopamine gating of glutamatergic sensorimotor and incentive motivational input signals to the striatum. Behavioural Brain Research, 137(1-2), 65–74.
Houk, J., Adams, J., & Barto, A. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In Davis, J.L., Beiser, D.G., & Houk J.C. (Eds.) Models of information processing in the basal ganglia (pp. 249–270). Cambridge: MIT Press.
Huettel, S.A., Song, A.W., & McCarthy, G. (2005). Decisions under uncertainty: probabilistic context influences activation of prefrontal and parietal cortices. Journal of Neuroscience, 25(13), 3304–3311.
Humphries, M.D., & Prescott, T.J. (2010). The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Progress in Neurobiology, 90(4), 385–417.
Iigaya, K. (2016). Adaptive learning and decision-making under uncertainty by metaplastic synapses guided by a surprise detection system. Elife, 5, e18073.
Insausti, R., Amaral, D., & Cowan, W. (1987). The entorhinal cortex of the monkey: II. Cortical afferents. Journal of Comparative Neurology, 264(3), 356–395.
Izhikevich, E.M. (2003). Simple model of spiking neurons. IEEE Transactions on Neural Networks, 14(6), 1569–1572.
Izhikevich, E.M. (2007). Dynamical systems in neuroscience. Cambridge, CA: MIT Press.
Jacobs, J., Kahana, M.J., Ekstrom, A.D., Mollison, M.V., & Fried, I. (2010). A sense of direction in human entorhinal cortex. Proceedings of the National Academy of Sciences, 107(14), 6487–6492.
Jhou, T.C., Fields, H.L., Baxter, M.G., Saper, C.B., & Holland, P.C. (2009). The rostromedial tegmental nucleus (RMTg), a GABAergic afferent to midbrain dopamine neurons, encodes aversive stimuli and inhibits motor responses. Neuron, 61(5), 786–800.
Jo, S., & Jung, M.W. (2016). Differential coding of uncertain reward in rat insular and orbitofrontal cortex. Scientific Reports, 6, 24085.
Joel, D., Niv, Y., & Ruppin, E. (2002). Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks, 15(4-6), 535–547.
Jones, B.F., & Witter, M.P. (2007). Cingulate cortex projections to the parahippocampal region and hippocampal formation in the rat. Hippocampus, 17(10), 957–976.
Kawato, M., & Samejima, K. (2007). Efficient reinforcement learning: computational theories, neuroscience and robotics. Current Opinion in Neurobiology, 17(2), 205–212.
Keiflin, R., Pribut, H.J., Shah, N.B., & Janak, P.H. (2019). Ventral tegmental dopamine neurons participate in reward identity predictions. Current Biology, 29(1), 93–103.
Kerr, K.M., Agster, K.L., Furtak, S.C., & Burwell, R.D. (2007). Functional neuroanatomy of the parahippocampal region: the lateral and medial entorhinal areas. Hippocampus, 17(9), 697–708.
Kobayashi, Y., & Okada, K. (2007). Reward prediction error computation in the pedunculopontine tegmental nucleus neurons. Annals of the New York Academy of Sciences, 1104(1), 310–323.
Kumaran, D., & Maguire, E.A. (2006). An unexpected sequence of events: mismatch detection in the human hippocampus. PLoS Biology, 4(12), e424.
Lipski, W.J., & Grace, A.A. (2013). Activation and inhibition of neurons in the hippocampal ventral subiculum by norepinephrine and locus coeruleus stimulation. Neuropsychopharmacology, 38(2), 285.
Liu, X., Hairston, J., Schrier, M., & Fan, J. (2011). Common and distinct networks underlying reward valence and processing stages: a meta-analysis of functional neuroimaging studies. Neuroscience and Biobehavioral Reviews, 35(5), 1219–1236.
Lodge, D.J., & Grace, A.A. (2006). The hippocampus modulates dopamine neuron responsivity by regulating the intensity of phasic neuron activation. Neuropsychopharmacology, 31(7), 1356–1361.
Maia, T.V. (2009). Reinforcement learning, conditioning, and the brain: successes and challenges. Cognitive, Affective, and Behavioral Neuroscience, 9(4), 343–364.
Marr, D. (1982). Vision: a computational investigation into the human representation and processing of visual information. New York: Freeman.
Mathys, C., Daunizeau, J., Friston, K.J., & Stephan, K.E. (2011). A Bayesian foundation for individual learning under uncertainty. Frontiers in Human Neuroscience, 5, 39.
Matsumoto, M., & Hikosaka, O. (2007). Lateral habenula as a source of negative reward signals in dopamine neurons. Nature, 447(7148), 1111–1115.
Matsumoto, M., & Hikosaka, O. (2009). Representation of negative motivational value in the primate lateral habenula. Nature Neuroscience, 12(1), 77–84.
Matsumoto, N., Minamimoto, T., Graybiel, A.M., & Kimura, M. (2001). Neurons in the thalamic CM-Pf complex supply striatal neurons with information about behaviorally significant sensory events. Journal of Neurophysiology, 85(2), 960–976.
Mishkin, M., Malamut, B., & Bachevalier, J. (1984). Memories and habits: two neural systems. In Lynch, G., McGaugh, J. L., & Weinberger, N. M. (Eds.) Neurobiology of human learning and memory (pp. 65–77). New York: Guilford Press.
Monosov, I.E. (2017). Anterior cingulate is a source of valence-specific information about value and uncertainty. Nature Communications, 8(1), 134.
Montague, P.R., Dayan, P., & Sejnowski, T.J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience, 16(5), 1936–1947.
Morita, K., Morishima, M., Sakai, K., & Kawaguchi, Y. (2012). Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways. Trends in Neurosciences, 35(8), 457–467.
Morita, K., Morishima, M., Sakai, K., & Kawaguchi, Y. (2013). Dopaminergic control of motivation and reinforcement learning: a closed-circuit account for reward-oriented behavior. Journal of Neuroscience, 33(20), 8866–8890.
Niv, Y., Daw, N.D., Joel, D., & Dayan, P. (2007). Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology, 191(3), 507–520.
Okada, K., & Kobayashi, Y. (2013). Reward prediction-related increases and decreases in tonic neuronal activity of the pedunculopontine tegmental nucleus. Frontiers in Integrative Neuroscience, 7, 36.
Neill, M., & Schultz, W. (2010). Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value. Neuron, 68(4), 789–800.
O’Reilly, R.C., Frank, M.J., Hazy, T.E., & Watz, B. (2007). PVLV: the primary value and learned value Pavlovian learning algorithm. Behavioral Neuroscience, 121(1), 31–49.
Payzan-LeNestour, E., & Bossaerts, P. (2011). Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Computational Biology, 7(1), e1001048.
Payzan-LeNestour, E., Dunne, S., Bossaerts, P., & O’Doherty, J.P. (2013). The neural representation of unexpected uncertainty during value-based decision making. Neuron, 79(1), 191–201.
Pickering, A.D., & Pesola, F. (2014). Modeling dopaminergic and other processes involved in learning from reward prediction error: contributions from an individual differences perspective. Frontiers in Human Neuroscience, 8, 740.
Preuschoff, K., & Bossaerts, P. (2007). Adding prediction risk to the theory of reward learning. Annals of the New York Academy of Sciences, 1104(1), 135–146.
Preuschoff, K., Quartz, S.R., & Bossaerts, P. (2008). Human insula activation reflects risk prediction errors as well as risk. Journal of Neuroscience, 28(11), 2745–2752.
Quintero, E., Diaz, E., Vargas, J.P., de la Casa, G., & Lopez, J.C. (2011). Ventral subiculum involvement in latent inhibition context specificity. Physiology and Behavior, 102(3-4), 414–420.
Rall, W. (1967). Distinguishing theoretical synaptic potentials computed for different soma-dendritic distributions of synaptic input. Journal of Neurophysiology, 30(5), 1138–1168.
Rescorla, R.A., & Wagner, A.R. (1972). A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In Black, A. H., & Prokasy, W. F. (Eds.) Classical conditioning II: current research and theory (pp. 64–99). New York: Appleton-Century-Crofts.
Riceberg, J.S., & Shapiro, M.L. (2012). Reward stability determines the contribution of orbitofrontal cortex to adaptive behavior. Journal of Neuroscience, 32(46), 16402–16409.
Root, D.H., Fabbricatore, A.T., Pawlak, A.P., Barker, D.J., Ma, S., & West, M.O. (2012). Slow phasic and tonic activity of ventral pallidal neurons during cocaine self-administration. Synapse, 66(2), 106–127.
Rushworth, M.F., & Behrens, T.E. (2008). Choice, uncertainty and value in prefrontal and cingulate cortex. Nature Neuroscience, 11(4), 389–397.
Rutishauser, U., Mamelak, A.N., & Schuman, E.M. (2006). Single-trial learning of novel stimuli by individual neurons of the human hippocampus-amygdala complex. Neuron, 49(6), 805– 813.
Sadikot, A., Parent, A., & Francois, C. (1992). Efferent connections of the centromedian and parafascicular thalamic nuclei in the squirrel monkey: a PHA-L study of subcortical projections. Journal of Comparative Neurology, 315(2), 137–159.
Salum, C., da Silva, A.R., & Pickering, A. (1999). Striatal dopamine in attentional learning: a computational model. Neurocomputing, 26, 845–854.
Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80(1), 1–27.
Schultz, W., Dayan, P., & Montague, P.R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599.
Seamans, J.K., & Robbins, T.W. (2010). Dopamine modulation of the prefrontal cortex and cognitive function. In Neve, K. A. (Ed.) The dopamine receptors. 2nd edn. (pp. 373–398). New York: Springer.-.
Sesack, S.R., & Grace, A.A. (2010). Cortico-basal ganglia reward network: microcircuitry. Neuropsychopharmacology, 35(1), 27–47.
Smith, Y., & Kieval, J.Z. (2000). Anatomy of the dopamine system in the basal ganglia. Trends in Neurosciences, 23, S28–S33.
Soltani, A., & Izquierdo, A. (2019). Adaptive learning under expected and unexpected uncertainty. Nature Reviews Neuroscience, 20(10), 635–644.
Strange, B.A., Duggins, A., Penny, W., Dolan, R.J., & Friston, K.J. (2005). Information theory, novelty and hippocampal responses: unpredicted or unpredictable? Neural Networks, 18(3), 225–230.
Stuber, G.D., Klanker, M., De Ridder, B., Bowers, M.S., Joosten, R.N., Feenstra, M.G., & Bonci, A. (2008). Reward predictive cues enhance excitatory synaptic strength onto midbrain dopamine neurons. Science, 321(5896), 1690–1692.
Sutton, R.S. (1992). Adapting bias by gradient descent: an incremental version of delta-bar-delta. In Proceedings of the tenth national conference on artificial intelligence (pp. 171–176). Cambridge: MIT Press.
Sutton, R.S., & Barto, A.G. (1998). Reinforcement learning: an introduction Cambridge. MA: MIT Press.
Takahashi, Y.K., Langdon, A.J., Niv, Y., & Schoenbaum, G. (2016). Temporal specificity of reward prediction errors signaled by putative dopamine neurons in rat VTA depends on ventral striatum. Neuron, 91(1), 182–193.
Takahashi, Y.K., Schoenbaum, G., & Niv, Y. (2008). Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Frontiers in neuroscience, 2, 14.
Tan, C.O., & Bullock, D. (2008). A local circuit model of learned striatal and dopamine cell responses under probabilistic schedules of reward. Journal of Neuroscience, 28(40), 10062–10074.
Taswell, C.A., Costa, V.D., Murray, E.A., & Averbeck, B.B. (2018). Ventral striatum’s role in learning from gains and losses. Proceedings of the National Academy of Sciences, 115(52), E12398–E12406.
Tian, J., & Uchida, N. (2015). Habenula lesions reveal that multiple mechanisms underlie dopamine prediction errors. Neuron, 87(6), 1304–1316.
Vanni-Mercier, G., Mauguiere, F., Isnard, J., & Dreher, J.-C. (2009). The hippocampus codes the uncertainty of cue-outcome associations: an intracranial electrophysiological study in humans. Journal of Neuroscience, 29(16), 5287–5294.
Van Rossum, G., & Drake, F.L. (2011). The Python language reference manual. Network Theory Ltd.
Vitay, J., & Hamker, F.H. (2014). Timing and expectation of reward: a neuro-computational model of the afferents to the ventral tegmental area. Frontiers in Neurorobotics, 8, 4.
Watabe-Uchida, M., Zhu, L., Ogawa, S.K., Vamanrao, A., & Uchida, N. (2012). Whole-brain mapping of direct inputs to midbrain dopamine neurons. Neuron, 74(5), 858–873.
Willingham, D.B. (1998). A neuropsychological theory of motor skill learning. Psychological Review, 105, 558–584.
Yu, A.J., & Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron, 46(4), 681–692.
Funding
This research was supported by NIH Grant 2R01MH063760.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing Interests
The authors declare no that they no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Inglis, J.B., Valentin, V.V. & Ashby, F.G. Modulation of Dopamine for Adaptive Learning: a Neurocomputational Model. Comput Brain Behav 4, 34–52 (2021). https://doi.org/10.1007/s42113-020-00083-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42113-020-00083-x