Advertisement

The Functions of Dopamine in Operant Conditioned Reflexes

  • V. I. MaiorovEmail author
Article

Dopamine neurons are activated by stimuli of both positive and negative modality according to the magnitude of the “willing effort” required to produce encouragement or avoidance of punishment. Acquisition of an operant conditioned reflex starts when the target of the movement (an external target, e.g., a lever, a location within a maze, or an internal target, e.g., a posture), based on the Pavlovian association mechanism (possibly dopamine-independent), becomes attractive or repulsive and acquires “incentive salience attribution”, the motive force of which is the “dopamine drive.” From the very beginning of execution of the operant conditioned reflex, dopamine is secreted in the window between activation of the conditioned signal and the movement, where it combines “activation of the central motor system of behavior” [Konorski, 1970] and modulation of synaptic plasticity for further learning. The operant movement performed by the animal under the influence of the dopamine drive is reinforced by diminution of the drive.

Keywords

conditioned reflex reinforcement incentive motivation goal-directed action habits 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aitken, T. J., Greenfield, V. Y., and Wassum, K. M., “Nucleus accumbens core dopamine signaling tracks the need-based motivational value of food-paired cues,” J. Neurochem., 136, No. 5, 1026–1036 (2016).Google Scholar
  2. Beier, K. T., Steinberg, E. E., DeLoach, K. E., et al., “Circuit architecture of VTA dopamine neurons revealed by systematic input-output mapping,” Cell, 162, 622–634 (2015).Google Scholar
  3. Belujon, P. and Grace, A., “Regulation of dopamine system responsivity and its adaptive and pathological response to stress,” Proc. R. Soc. B, 282, 2014–2516 (2015).Google Scholar
  4. Berridge, K. C. and Robinson, T. E., “Liking, wanting, and incentive-sensitization theory of addiction,” Am. Psychol., 71, No. 8, 670–679 (2016).Google Scholar
  5. Berridge, K. C. and Robinson, T. E., “What is the role of dopamine in reward: Hedonic impact, reward learning, or incentive salience?” Brain Res. Rev., 28, No. 3, 309–369 (1998).Google Scholar
  6. Bizzi, E., Giszter, S. E., Loeb, E., et al., “Modular organization of motor behavior in the frog’s spinal cord,” Trends Neurosci., 18, No. 10, 442–446 (1995).Google Scholar
  7. Bolles, R. C., “Reinforcement, expectancy, and learning,” Psychol. Rev., 79, No. 5, 394–409 (1972).Google Scholar
  8. Budygin, E. A., Park, J., Bass, C. E., et al., “Aversive stimulus differentially triggers subsecond dopamine release in reward regions,” Neuroscience, 201, 331–337 (2012).Google Scholar
  9. Caggiano, V., Cheung, V. C. K., and Bizzi, E., “An optogenetic demonstration of motor modularity in the mammalian spinal cord,” Sci. Rep., 6, 35185 (2016).Google Scholar
  10. Cain, C. K. and LeDoux, J. E., “Escape from fear: a detailed behavioral analysis of two atypical responses reinforced by CS termination,” J. Exp. Psychol. Anim. Behav. Process., 33, 451–463 (2007).Google Scholar
  11. Chang, C. Y., Gardner, M., Di Tillio, M. G., and Schoenbaum, G., “Optogenetic blockade of dopamine transients prevents learning induced by changes in reward features,” Curr. Biol., 27, 1–7 (2017).Google Scholar
  12. Cole, S. and McNally, G. P., “Opioid receptors mediate direct predictive fear learning: Evidence from one-trial blocking,” Learn. Mem., 14, No. 4, 229–235 (2007).Google Scholar
  13. Collins, A. L., Greenfield, V. Y., Bye, J. K., et al., “Dynamic mesolimbic dopamine signaling during action sequence learning and expectation violation,” Sci. Rep., 6, 20231, 1–15 (2016).Google Scholar
  14. Darvas, M., Fadok, J. P., and Palmiter, R. D., “Requirement of dopamine signaling in the amygdale and striatum for learning and maintenance of a conditioned avoidance response,” Learn. Mem., 18, 136–143 (2011).Google Scholar
  15. De Lavilleon, G., Lacroix, M. M., Rondi-Reig, L., and Benchenane, K., “Explicit memory creation during sleep demonstrates a causal role of place cells in navigation,” Nat. Neurosci., 18, 493–495 (2015).Google Scholar
  16. De Wit, S. and Dickinson, A., “Associative theories of goal-directed behavior: A case for animal-human translational models,” Psychol. Res., 73, No. 4, 463–476 (2009).Google Scholar
  17. Dombrowski, P. A., Maia, T. V., Boschen, S. L., et al., “Evidence that conditioned avoidance responses are reinforced by positive prediction errors signaled by tonic striatal dopamine,” Behav. Brain Res., 241, 112–119 (2013).Google Scholar
  18. Fanselow, M. S. and Wassum, K. M., “The origins and organization of vertebrate Pavlovian conditioning,” Cold Spring Harb. Perspect. Biol., 8, a021717 (2016).Google Scholar
  19. Fernando, A. B. P., Urcelay, G. P., Mar, A. C., et al., “Safety signals as instrumental reinforcers during free-operant avoidance,” Learn. Mem., 21, No. 9, 488–497 (2014).Google Scholar
  20. Flagel, S. B., Akil, H., and Robinson, T. E., “Individual differences in the attribution of incentive salience to reward-related cues: implications for addiction,” Neuropharmacology, 56, Suppl. 1, 139–148 (2009).Google Scholar
  21. Flagel, S. B., Clark, J. J., Robinson, T. E., et al., “A selective role for dopamine in stimulus-reward learning,” Nature, 469, No. 7328, 53–57 (2011).Google Scholar
  22. Fremaux, N. and Gerstner, W., “Neuromodulated spike-timing- dependent plasticity, and theory of three-factor learning rules,” Front. Neural Circ., 9, 85, (2016).Google Scholar
  23. Giszter, S. F., Mussa-Ivaldi, F. A., and Emilio Bizzi, E., “Convergent force fields organized in the frog’s spinal cord,” J. Neurosci., 73, No. 2, 467–491Google Scholar
  24. Glimcher, P. W., “Understanding dopamine and reinforcement learning – the dopamine reward prediction error hypothesis,” Proc. Natl. Acad. Sci. USA, 108, Suppl. 3, 15647–15654 (2011).Google Scholar
  25. Gotz, T. and Janik, V. M., “Repeated elicitation of the acoustic startle reflex leads to sensitisation in subsequent avoidance behaviour and induces fear conditioning,” BMC Neurosci., 12, 1–13 (2011).Google Scholar
  26. Graziano, M. S. A., Taylor, C. S. R., and Moore, T., “Complex movements evoked by microstimulation of precentral cortex,” Neuron, 34, No. 5, 841–851 (2002).Google Scholar
  27. Hart, A. S., Clark, J. J., and Phillips, P. E. M., “Dynamic shaping of dopamine signals during probabilistic Pavlovian conditioning,” Neurobiol. Learn. Mem., 117, 84–92 (2015).Google Scholar
  28. Hart, A. S., Rutledge, R. B., Glimcher, P. W., and Phillips, P. E. M., “Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term,” J. Neurosci., 34, No. 3, 698–704 (2014).Google Scholar
  29. Hart, G., Bradfield, L. A., and Balleine, B. W., “Prefrontal corticostriatal disconnection blocks the acquisition of goal-directed action,” J. Neurosci., 38, No. 5, 1311–1322 (2018).Google Scholar
  30. He, K., Huertas, M., Hong, S. Z., Tie, X. X., et al., “Distinct eligibility traces for LTP and LTD in cortical synapses,” Neuron, 88, 1–11 (2015).Google Scholar
  31. Holland, P. C. and Schiffino, F. L., “Mini-review: Prediction errors, attention and associative learning,” Neurobiol. Learn. Mem, 131, 207–215 (2016).Google Scholar
  32. Holly, E. N. and Miczek, K. A., “Ventral tegmental area dopamine revisited: effects of acute and repeated stress,” Psychopharmacology (Berl.), 233, 163–186 (2016).Google Scholar
  33. Hong, S. and Hikosaka, O., “Dopamine-mediated learning and switching in cortico-striatal circuit explain behavioral changes in reinforcement learning,” Front. Behav. Neurosci., 5, No. 15, 1–17 (2011).Google Scholar
  34. Hull, C. L., Principles of Behavior (1943), cited by Heckhausen (2003).Google Scholar
  35. Ilango, A., Kesner, A. J., Broker, C. J., et al., “Phasic excitation of ventral tegmental dopamine neurons potentiates the initiation of conditioned approach behavior: parametric and reinforcement-schedule analyses,” Front. Behav. Neurosci., 8, 155 (2014).Google Scholar
  36. Khekkhauzen, Kh., “Volitional processes: realization of intentions,” in: Motivation and Activity, Piter, St. Petersburg and Smysl, Moscow (2003), Chpt. 6.Google Scholar
  37. Kim, H. F. and Hikosaka, O., “Parallel basal ganglia circuits for voluntary and automatic behaviour to reach rewards,” Brain, 138, No. 7, 1776–1800 (2015).Google Scholar
  38. Kim, H. F., Ghazizadeh, A., and Hikosaka, O., “Dopamine neurons encoding long-term memory of object value for habitual behavior,” Cell, 163, 1165–1175 (2015).Google Scholar
  39. Kim, K. M., Baratta, M. V., Yang, A., et al., “Optogenetic mimicry of the transient activation of dopamine neurons by natural reward is sufficient for operant reinforcement,” PLoS One, 7, e33612 (2012).Google Scholar
  40. Konorski, J., Integrative Activity of the Brain. An Interdisciplinary Approach, University of Chicago Press (1967).Google Scholar
  41. Konorski, Yu., Integrative Activity of the Brain [Russian translation], Mir, Moscow (1970).Google Scholar
  42. Koob, G. F., “The dopamine anhedonia hypothesis: a pharmacological phrenology,” Behav. Brain Sci., 5, 63–64 (1982).Google Scholar
  43. Krypotos, A.-M., Effting, M., Kindt, M., and Beckers, T., “Avoidance learning: a review of theoretical models and recent developments,” Front. Behav. Neurosci., 9, 189 (2015).Google Scholar
  44. LeDoux, J. and Daw, N. D., “Surviving threats: neural circuit and computational implications of a new taxonomy of defensive behavior,” Nat. Rev. Neurosci. (2018), doi  https://doi.org/10.1038/nrn.22.
  45. Leont’ev, A. N., Lecture in General Psychology. Lecture 50, the Problem of Will, Smysl, Moscow (2005).Google Scholar
  46. Li, S. S. Y. and McNally, G. P., “The conditions that promote fear learning: Prediction error and Pavlovian fear conditioning,” Neurobiol. Learn. Mem, 108, 14–21 (2014).Google Scholar
  47. Lisman, J., Grace, A. A., and Duzel, E., “A neoHebbian framework for episodic memory; role of dopamine-dependent late LTP,” Trends Neurosci., 34, No. 10, 536–547 (2011).Google Scholar
  48. Maia, T. V., “Two-factor theory, the actor-critic model and conditioned avoidance,” Learn. Behav., 38, No. 1, 50–67 (2010).Google Scholar
  49. Maier, S. F. and Seligman, M. E. P., “Learned helplessness at fifty: Insights from neuroscience,” Psychol. Rev., 123, No. 4, 349–367 (2016).Google Scholar
  50. Maiorov, V. I. and Serkov, A. N., “Neuron activity in the ventral tegmental area of the midbrain on fi rst execution of a conditioned active avoidance reflex,” Zh. Vyssh. Nerv. Deyat., 66, No. 6, 725–729 (2016).Google Scholar
  51. Maiorov, V. I., “A model neuronal mechanism of operant conditioned ‘escape from motivation’ reflex in the motor cortex of the cat,” in: Problems in Neurocybernetics: Proc. 14th Int. Conf. (2005), No. 1, pp. 51–55.Google Scholar
  52. Maiorov, V. I., “The role of dopaminergic modulation in the organization of operant conditioned reflexes,” Ros. Fiziol. Zh., 90, No. 8, 381–382 (2004).Google Scholar
  53. Mileykovskiy, B. and Morales, M., “Duration of inhibition of ventral tegmental area dopamine neurons encodes a level of conditioned fear,” J. Neurosci., 31, No. 20, 7471–7476 (2011).Google Scholar
  54. Morgane, P. J., “Medial forebrain bundle and “feeding centers” of the hypothalamus,” J. Comp. Neurol., 117, No. 1, 1–25 (1961).Google Scholar
  55. Moscarello, J. M. and LeDoux, J. E., “Active avoidance learning requires prefrontal suppression of amygdala-mediated defensive reactions,” J. Neurosci., 33, No. 9, 3815–3823 (2013).Google Scholar
  56. Moutoussis, M., Bentall, R. P., Williams, J., and Dayan, P., “A temporal difference account of avoidance learning,” Network, 19, No. 2, 137–160 (2008).Google Scholar
  57. Mowrer, O. H., Learning Theory and Behavior, Wiley, New York (1960), cited by Bolles (1972)Google Scholar
  58. Nakano, T., Yoshimoto, J., and Doya, K., “A model-based prediction of the calcium responses in the striatal synaptic spines depending on the timing of cortical and dopaminergic inputs and post-synaptic spikes,” Front. Comput. Neurosci., 7, Art. 119 (2013).Google Scholar
  59. Nasser, H. M., Calu, D. J., Schoenbaum, G., and Sharpe, M. J., “The Dopamine prediction error: Contributions to associative models of reward learning,” Front. Psychol., 8, No. 244, 1–17 (2017).Google Scholar
  60. Nieh, E. H., Vander Weeler, C. M., Matthews, G. A., et al., “Inhibitory input from the lateral hypothalamus to the ventral tegmental area disinhibits dopamine neurons and promotes behavioral activation,” Neuron, 90, No. 6, 1286–1298 (2016).Google Scholar
  61. Packard, M. G., “Exhumed from thought: Basal ganglia and response learning in the plus-maze,” Behav. Brain Res., 199, No. 1, 24–31 (2009).Google Scholar
  62. Park, J., Bucher, E. S., Budygin, E. A., and Wightman, R. M., “Norepinephrine and dopamine transmission in 2 limbic regions differentially respond to acute noxious stimulation,” Pain, 156, No. 2, 318–327 (2015).Google Scholar
  63. Pascoli, V., Terrier, J., Hiver, A., and Luscher, C., “Sufficiency of mesolimbic dopamine neuron stimulation for the progression to addiction,” Neuron, 88, 1–13 (2015).Google Scholar
  64. Pavlov, I. P., “The feeding center,” in: Twenty Years of Experience in the Study of Higher Nervous Activity (behavior) in Animals, Nauka, Moscow (1973b), pp. 103–110.Google Scholar
  65. Pavlov, I. P., “The physiological mechanism of apparently voluntary movements,” in: Twenty Years of Experience in the Study of Higher Nervous Activity (behavior) in Animals, Nauka, Moscow (1973a), pp. 481–485.Google Scholar
  66. Pawlak, V., Wickens, J. R., Kirkwood, A., and Kerr, J. D., “Timing is not everything: neuromodulation opens the STDP gate,” Front. Synaptic Neurosci., 2, Art. 146 (2010).Google Scholar
  67. Polter, A. M. and Kauer, J. A., “Stress and VTA synapses: Implications for addiction and depression,” Eur. J. Neurosci., 39, No. 7, 1179–1188 (2014).Google Scholar
  68. Redgrave, P., Gurney, K., and Reynolds, J., “What is reinforced by phasic dopamine signals?” Brain Res. Rev., 58, 322–339 (2008).Google Scholar
  69. Rescorla, R. A., “Pavlovian conditioning. It’s not what you think it is,” Am. Psychol., 43, No. 3, 151–160 (1988).Google Scholar
  70. Robinson, M. J. F. and Berridge, K. C., “Instant transformation of learned repulsion into motivational ‘wanting,” Curr. Biol., 23, 282–289 (2013).Google Scholar
  71. Robinson, M. J. F., Fischer, A. M., Ahuja, A., et al., “Roles of ‘wanting’ and ‘liking’ in motivating behavior: gambling, food, and drug addictions. Behavioral neuroscience of motivation,” Curr. Top. Behav. Neurosci., 27, 105–136 (2016).Google Scholar
  72. Robinson, S., Sandstrom, S. M., Denenberg, V. H., and Palmiter, R. D., “Distinguishing whether dopamine regulates liking, wanting, and/or learning about rewards,” Behav. Neurosci., 119, No. 1, 5–15 (2005).Google Scholar
  73. Roesch, M. R., Esber, G. R., Li, J., et al., “Surprise! Neural correlates of Pearce- Hall and Rescorla–Wagner coexist within the brain,” Eur. J. Neurosci., 35, No. 7, 1190–1200 (2012).Google Scholar
  74. Rossi, M. A., Sukharnikova, T., Hayrapetyan, V. Y., et al., “Operant self-stimulation of dopamine neurons in the substantia nigra,” PLoS One, 8, No. 6, e65799 (2013).Google Scholar
  75. Sadacca, B. F., Jones, J., and Schoenbaum, G., “Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework,” eLife, 5, e13665, 1–13 (2016).Google Scholar
  76. Salamone, J. D., Yohn, S. E., Lopez-Cruz, L., et al., “Activational and effort-related aspects of motivation: neural mechanisms and implications for psychopathology,” Brain, 139, No. 5, 1325–1347 (2016).Google Scholar
  77. Schultz, W., Stauffer, W. R., and Lak, A., “The phasic dopamine signal maturing: from reward via behavioural activation to formal economic utility,” Curr. Opin. Neurobiol., 43, 139–148 (2017).Google Scholar
  78. Sesack, S. R. and Carr, D. B., “Selective prefrontal cortex inputs to dopamine cells: implications for schizophrenia,” Physiol. Behav., 77, 513–517 (2002).Google Scholar
  79. Shahaf, G. and Marom, S., “Learning in networks of cortical neurons,” J. Neurosci., 21, No. 22, 8782–8788 (2001).Google Scholar
  80. Sharpe, M. J., Batchelor, H. M., and Schoenbaum, G., “Preconditioned cues have no value,” eLife, 6, e28362, 1–9 (2017).Google Scholar
  81. Sharpe, M. J., Chang, C. Y., Liu, M. A., et al., “Dopamine transients are sufficient and necessary for acquisition of model-based associations,” Nat. Neurosci., 20, No. 5, 735–742 (2017b).Google Scholar
  82. Siegel, S., “Pavlovian conditioning and drug overdose: when tolerance fails,” Addict. Res. Theory, 9, No. 5, 503–513 (2001).Google Scholar
  83. Siegel, S., Baptista, M. A. S., Kim, J. A., et al., “Pavlovian psychopharmacology: The associative basis of tolerance,” Exp. Clin. Psychopharmacology, 8, No. 3, 276–293 (2000).Google Scholar
  84. Simpson, E. H. and Balsam, P. D., “The behavioral neuroscience of motivation: An overview of concepts, measures, and translational applications,” Curr. Top. Behav. Neurosci., 27, 1–12 (2016).Google Scholar
  85. Sinapayen, L., Masumori, A., and Ikegami, T., “Learning by stimulation avoidance: A principle to control spiking neural networks dynamics,” PLoS One, 12, No. 2, e0170388 (2017).Google Scholar
  86. Smith, A., Li, M., Becker, S., and Kapur, S., “Linking animal models of psychosis to computational models of dopamine function,” Neuropsychopharmacology, 32, No. 1, 54–66 (2007).Google Scholar
  87. Smith, K. S. and Graybiel, A. M., “Habit formation,” Dialogues Clin. Neurosci., 18, No. 1, 33–43 (2016).Google Scholar
  88. Takeuchi, T., Duszkiewicz, A. J., Sonneborn, A., et al., “Locus coeruleus and dopaminergic consolidation of everyday memory,” Nature, 537, 357–362 (2016).Google Scholar
  89. Tereshchenko, L. V. and Latanov, A. V., “Impairments to visuomotor functions on development of MPTP-induced Parkinson-like syndrome in monkeys,” Zh. Vyssh. Nerv. Deyat., 68, No. 4 (2018).Google Scholar
  90. Thomson, A. A. and Martinet, A. V., A Practical English Grammar, Oxford Univ. Press (1986).Google Scholar
  91. Tsai, H. C., Zhang, F., Adamantidis, A., et al., “Phasic fi ring in dopaminergic neurons is suffi cient for behavioral conditioning,” Science, 324, 1080–1084 (2009).Google Scholar
  92. Tye, K. M., Mirzabekov, J. J., Warden, M. R., et al., “Dopamine neurons modulate neural encoding and expression of depression- related behaviour,” Nature, 493, No. 7433, 537–541 (2013).Google Scholar
  93. Viola, H., Ballarini, F., Martinez, M. C., and Moncada, D., “The tagging and capture hypothesis from synapse to memory,” Prog. Mol. Biol. Transl. Sci., 122, 391–423 (2014).Google Scholar
  94. Waelti, P., Dickinson, A., and Schultz, W., “Dopamine responses comply with basic assumptions of formal learning theory,” Nature, 412, No. 6842, 43–48 (2001).Google Scholar
  95. Wagatsuma, A., Okuyama, T., Sun, C., et al., “Locus coeruleus input to hippocampal CA3 drives single-trial learning of a novel context,” Proc. Natl. Acad. Sci. USA, 115, No. 2, E310–E316 (2018).Google Scholar
  96. Watabe-Uchida, M., Eshel, N., and Uchida, N., “Neural circuitry of reward prediction error,” Annu. Rev. Neurosci., 40, 373–394 (2017).Google Scholar
  97. Wenzel, J. M., Rausher, N. A., Cheer, J. F., and Oleson, E. B., “A role for phasic dopamine release within the nucleus accumbens in encoding aversion: a review of the neurochemical literature,” ACS Chem. Neurosci., 6, 16–26 (2015).Google Scholar
  98. Wise, R. A. and Koob, G. F., “The development and maintenance of drug addiction,” Neuropsychopharmacology, 39, 254–262 (2014).Google Scholar
  99. Wise, R. A., “Dopamine, learning and motivation,” Nat. Rev. Neurosci., 5, No. 6, 483–494 (2004).Google Scholar
  100. Wise, R. A., “Dual roles of dopamine in food and drug seeking: the drive-reward paradox,” Biol. Psychiatry, 73, No. 9, 819–826 (2013).Google Scholar
  101. Yagishita, S., Hayashi-Takagi, A., Ellis-Davies, G. S. R., et al., “A critical time window for dopamine actions on the structural plasticity of dendritic spines,” Science, 345, No. 6204, 1616–1620 (2014).Google Scholar
  102. Yin, H. H. and Knowlton, B. J., “The role of the basal ganglia in habit formation,” Nat. Rev. Neurosci., 7, No. 6, 464–476 (2006).Google Scholar
  103. Zellner, M. R. and Ranaldi, R., “How conditioned stimuli acquire the ability to activate VTA dopamine cells: A proposed neurobiological component of reward-related learning,” Neurosci. Biobehav. Rev., 34, No. 5, 769–780 (2010).Google Scholar
  104. Zweifel, L. S., Fadok, J. P., Argilli, E., et al., “Activation of dopamine neurons is critical for aversive conditioning and prevention of generalized anxiety,” Nat. Neurosci., 14, 620–626 (2011).Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Higher Nervous ActivityLomonosov Moscow State UniversityMoscowRussia

Personalised recommendations