Advertisement

pp 1-30 | Cite as

The Neurobiology of Impulsive Decision-Making and Reinforcement Learning in Nonhuman Animals

  • Stephanie M. GromanEmail author
Chapter
  • 27 Downloads
Part of the Current Topics in Behavioral Neurosciences book series

Abstract

Impulsive decisions are those that favor immediate over delayed rewards, involve the acceptance of undue risk or uncertainty, or fail to adapt to environmental changes. Pathological levels of impulsive decision-making have been observed in individuals with mental illness, but there may be substantial heterogeneity in the processes that drive impulsive choices. Understanding this behavioral heterogeneity may be critical for understanding associated diverseness in the neural mechanisms that give rise to impulsivity. The application of reinforcement learning algorithms in the deconstruction of impulsive decision-making phenotypes can help bridge the gap between biology and behavior and provide insights into the biobehavioral heterogeneity of impulsive choice. This chapter will review the literature on the neurobiological mechanisms of impulsive decision-making in nonhuman animals; specifically, the role of the amine neuromodulatory systems (dopamine, serotonin, norepinephrine, and acetylcholine) in impulsive decision-making and reinforcement learning processes is discussed. Ultimately, the integration of reinforcement learning algorithms with sophisticated behavioral and neuroscience techniques may be critical for advancing the understanding of the neurochemical basis of impulsive decision-making.

Keywords

Acetylcholine Decision making Dopamine Impulsivity Norepinephrine Reinforcement learning Serotonin 

Notes

Acknowledgments

This work was supported by National Institutes of Health grants R21 MH120615, R21 MH120799, R01 DA041480, and R01 DA043443 and a Young Investigator Award from the Brain & Behavior Research Foundation.

The author thanks Alexander J. Keip and Neema Moin Afshar for their insightful comments and critiques of the manuscript.

References

  1. Abela AR, Chudasama Y (2014) Noradrenergic α2A-receptor stimulation in the ventral hippocampus reduces impulsive decision-making. Psychopharmacology 231:521–531Google Scholar
  2. Addicott MA, Pearson JM, Wilson J, Platt ML, McClernon FJ (2013) Smoking and the bandit: a preliminary study of smoker and nonsmoker differences in exploratory behavior measured with a multiarmed bandit task. Exp Clin Psychopharmacol 21:66–73Google Scholar
  3. Aharonovich E, Hasin DS, Brooks AC, Liu X, Bisaga A, Nunes EV (2006) Cognitive deficits predict low treatment retention in cocaine dependent patients. Drug Alcohol Depend 81:313–322Google Scholar
  4. Ahn W-Y, Rass O, Fridberg DJ, Bishara AJ, Forsyth JK, Breier A, Busemeyer JR, Hetrick WP, Bolbecker AR, O’Donnell BF (2011) Temporal discounting of rewards in patients with bipolar disorder and schizophrenia. J Abnorm Psychol 120:911–921Google Scholar
  5. Alsiö J, Nilsson SRO, Gastambide F, Wang RAH, Dam SA, Mar AC, Tricklebank M, Robbins TW (2015) The role of 5-HT2C receptors in touchscreen visual reversal learning in the rat: a cross-site study. Psychopharmacology 232:4017–4031Google Scholar
  6. Aston-Jones G, Cohen JD (2005) Adaptive gain and the role of the locus coeruleus-norepinephrine system in optimal performance. J Comp Neurol 493:99–110Google Scholar
  7. Aston-Jones G, Rajkowski J, Kubiak P (1997) Conditioned responses of monkey locus coeruleus neurons anticipate acquisition of discriminative behavior in a vigilance task. Neuroscience 80:697–715Google Scholar
  8. Baarendse PJJ, Vanderschuren LJMJ (2012) Dissociable effects of monoamine reuptake inhibitors on distinct forms of impulsive behavior in rats. Psychopharmacology 219:313–326Google Scholar
  9. Baarendse PJJ, Winstanley CA, Vanderschuren LJMJ (2013) Simultaneous blockade of dopamine and noradrenaline reuptake promotes disadvantageous decision making in a rat gambling task. Psychopharmacology 225:719–731Google Scholar
  10. Bari A, Robbins TW (2013) Inhibition and impulsivity: behavioral and neural basis of response control. Prog Neurobiol 108:44–79Google Scholar
  11. Bari A, Theobald DE, Caprioli D, Mar AC, Aidoo-Micah A, Dalley JW, Robbins TW (2010) Serotonin modulates sensitivity to reward and negative feedback in a probabilistic reversal learning task in rats. Neuropsychopharmacology 35:1290–1301Google Scholar
  12. Barkley RA (1997) Behavioral inhibition, sustained attention, and executive functions: constructing a unifying theory of ADHD. Psychol Bull 121:65–94Google Scholar
  13. Barrus MM, Winstanley CA (2016) Dopamine D3 receptors modulate the ability of win-paired cues to increase risky choice in a rat gambling task. J Neurosci 36:785–794Google Scholar
  14. Bizot JC, Thiébot MH, Le Bihan C, Soubrié P, Simon P (1988) Effects of imipramine-like drugs and serotonin uptake blockers on delay of reward in rats. Possible implication in the behavioral mechanism of action of antidepressants. J Pharmacol Exp Ther 246:1144–1151Google Scholar
  15. Bizot J, Le Bihan C, Puech AJ, Hamon M, Thiébot M (1999) Serotonin and tolerance to delay of reward in rats. Psychopharmacology 146:400–412Google Scholar
  16. Bjork JM, Hommer DW, Grant SJ, Danube C (2004) Impulsivity in abstinent alcohol-dependent patients: relation to control subjects and type 1–/type 2–like traits. Alcohol 34:133–150Google Scholar
  17. Blaes SL, Orsini CA, Mitchell MR, Spurrell MS, Betzhold SM, Vera K, Bizon JL, Setlow B (2018) Monoaminergic modulation of decision-making under risk of punishment in a rat model. Behav Pharmacol 29:745–761Google Scholar
  18. Boulougouris V, Robbins TW (2010) Enhancement of spatial reversal learning by 5-HT2C receptor antagonism is neuroanatomically specific. J Neurosci 30:930–938Google Scholar
  19. Boulougouris V, Glennon JC, Robbins TW (2008) Dissociable effects of selective 5-HT2A and 5-HT2C receptor antagonists on serial spatial reversal learning in rats. Neuropsychopharmacology 33:2007–2019Google Scholar
  20. Boulougouris V, Castane A, Robbins TW (2009) Dopamine D2/D3 receptor agonist quinpirole impairs spatial reversal learning in rats: investigation of D3 receptor involvement in persistent behavior. Psychopharmacology 202:611–620Google Scholar
  21. Bouret S, Sara SJ (2004) Reward expectation, orientation of attention and locus coeruleus-medial frontal cortex interplay during learning. Eur J Neurosci 20:791–802Google Scholar
  22. Bradfield LA, Bertran-Gonzalez J, Chieng B, Balleine BW (2013) The thalamostriatal pathway and cholinergic control of goal-directed action: interlacing new with existing learning in the striatum. Neuron 79:153–166Google Scholar
  23. Brigman JL, Mathur P, Harvey-White J, Izquierdo A, Saksida LM, Bussey TJ, Fox S, Deneris E, Murphy DL, Holmes A (2010) Pharmacological or genetic inactivation of the serotonin transporter improves reversal learning in mice. Cereb Cortex 20:1955–1963Google Scholar
  24. Bymaster F, Katner JS, Nelson DL, Hemrick-Luecke SK, Threlkeld PG, Heiligenstein JH, Morin SM, Gehlert DR, Perry KW (2002) Atomoxetine increases extracellular levels of norepinephrine and dopamine in prefrontal cortex of rat a potential mechanism for efficacy in attention deficit/hyperactivity disorder. Neuropsychopharmacology 27:699–711Google Scholar
  25. Cannon DM, Ichise M, Rollis D, Klaver JM, Gandhi SK, Charney DS, Manji HK, Drevets WC (2007) Elevated serotonin transporter binding in major depressive disorder assessed using positron emission tomography and [11C]DASB; comparison with bipolar disorder. Biol Psychiatry 62:870–877Google Scholar
  26. Cardinal RN, Pennicott DR, Sugathapala CL, Robbins TW, Everitt BJ (2001) Impulsive choice induced in rats by lesions of the nucleus accumbens core. Science 292:2499–2501Google Scholar
  27. Chang CY, Gardner MPH, Conroy JC, Whitaker LR, Schoenbaum G (2018) Brief, but not prolonged, pauses in the firing of midbrain dopamine neurons are sufficient to produce a conditioned inhibitor. J Neurosci 38:8822–8830Google Scholar
  28. Chopin P, Colpaert FC, Marien M (1999) Effects of alpha-2 adrenoceptor agonists and antagonists on circling behavior in rats with unilateral 6-hydroxydopamine lesions of the nigrostriatal pathway. J Pharmacol Exp Ther 288:798–804Google Scholar
  29. Clark L, Robbins TW, Ersche KD, Sahakian BJ (2006) Reflection impulsivity in current and former substance users. Biol Psychiatry 60:515–522Google Scholar
  30. Clarke HF, Dalley JW, Crofts HS, Robbins TW, Roberts AC (2004) Cognitive inflexibility after prefrontal serotonin depletion. Science 304:878–880Google Scholar
  31. Clarke HF, Hill GJ, Robbins TW, Roberts AC (2011) Dopamine, but not serotonin, regulates reversal learning in the marmoset caudate nucleus. J Neurosci 31:4290–4297Google Scholar
  32. Cools R, D’Esposito M (2011) Inverted-U–shaped dopamine actions on human working memory and cognitive control. Biol Psychiatry 69:e113–e125Google Scholar
  33. Costa VD, Tran VL, Turchi J, Averbeck BB (2015) Reversal learning and dopamine: a Bayesian perspective. J Neurosci 35:2407–2416Google Scholar
  34. Costa VD, Dal Monte O, Lucas DR, Murray EA, Averbeck BB (2016) Amygdala and ventral striatum make distinct contributions to reinforcement learning. Neuron 92:505–517Google Scholar
  35. Courtney KE, Arellano R, Barkley-Levenson E, Gálvan A, Poldrack RA, MacKillop J, David Jentsch J, Ray LA (2012) The relationship between measures of impulsivity and alcohol misuse: an integrative structural equation modeling approach. Alcohol Clin Exp Res 36:923–931Google Scholar
  36. Craig AR, Maxfield AD, Stein JS, Renda CR, Madden GJ (2014) Do the adjusting-delay and increasing-delay tasks measure the same construct: delay discounting? Behav Pharmacol 25:306–315Google Scholar
  37. Crean JP, de Wit H, Richards JB (2000) Reward discounting as a measure of impulsive behavior in a psychiatric outpatient population. Exp Clin Psychopharmacol 8:155–162Google Scholar
  38. Crockett MJ, Clark L, Lieberman MD, Tabibnia G, Robbins TW (2010) Impulsive choice and altruistic punishment are correlated and increase in tandem with serotonin depletion. Emotion 10:855–862Google Scholar
  39. Dalley JW, Robbins TW (2017) Fractionating impulsivity: neuropsychiatric implications. Nat Rev Neurosci 18:158–171Google Scholar
  40. Dautan D, Huerta-Ocampo I, Witten IB, Deisseroth K, Bolam JP, Gerdjikov T, Mena-Segovia J (2014) A major external source of cholinergic innervation of the striatum and nucleus accumbens originates in the brainstem. J Neurosci 34:4509–4518Google Scholar
  41. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans’ choices and striatal prediction errors. Neuron 69:1204–1215Google Scholar
  42. Dayan P, Daw ND (2008) Decision theory, reinforcement learning, and the brain. Cogn Affect Behav Neurosci 8:429–453Google Scholar
  43. Deserno L, Wilbertz T, Reiter A, Horstmann A, Neumann J, Villringer A, Heinze H-J, Schlagenhauf F (2015) Lateral prefrontal model-based signatures are reduced in healthy individuals with high trait impulsivity. Transl Psychiatry 5:e659Google Scholar
  44. Dom G, D’haene P, Hulstijn W, Sabbe B (2006) Impulsivity in abstinent early- and late-onset alcoholics: differences in self-report measures and a discounting task. Addiction 101:50–59Google Scholar
  45. Doya K (2002) Metalearning and neuromodulation. Neural Netw 15:495–506Google Scholar
  46. Doya K, Samejima K, Katagiri K, Kawato M (2002) Multiple model-based reinforcement learning. Neural Comput 14:1347–1369Google Scholar
  47. Eagle DM, Baunez C, Hutcheson DM, Lehmann O, Shah AP, Robbins TW (2008) Stop-signal reaction-time task performance: role of prefrontal cortex and subthalamic nucleus. Cereb Cortex 18:178–188Google Scholar
  48. Enticott PG, Ogloff JRP, Bradshaw JL (2006) Associations between laboratory measures of executive inhibitory control and self-reported impulsivity. Personal Individ Differ 41:285–294Google Scholar
  49. Evenden JL (1999) Varieties of impulsivity. Psychopharmacology 146:348–361Google Scholar
  50. Fernie G, Cole JC, Goudie AJ, Field M (2010) Risk-taking but not response inhibition or delay discounting predict alcohol consumption in social drinkers. Drug Alcohol Depend 112:54–61Google Scholar
  51. Fillmore MT, Rush CR (2006) Polydrug abusers display impaired discrimination-reversal learning in a model of behavioural control. J Psychopharmacol 20:24–32Google Scholar
  52. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299:1898–1902Google Scholar
  53. Fishbein DH, Eldreth DL, Hyde C, Matochik JA, London ED, Contoreggi C, Kurian V, Kimes AS, Breeden A, Grant S (2005) Risky decision making and the anterior cingulate cortex in abstinent drug abusers and nonusers. Cogn Brain Res 23:119–136Google Scholar
  54. Gläscher J, Daw N, Dayan P, O’Doherty JP (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66:585–595Google Scholar
  55. Gold PE (2003) Acetylcholine modulation of neural systems involved in learning and memory. Neurobiol Learn Mem 80:194–210Google Scholar
  56. Groman SM, Lee B, London ED, Mandelkern MA, James AS, Feiler K, Rivera R, Dahlbom M, Sossi V, Vandervoort E, Jentsch JD (2011) Dorsal striatal D2-like receptor availability covaries with sensitivity to positive reinforcement during discrimination learning. J Neurosci 31:7291–7299Google Scholar
  57. Groman SM, James AS, Seu E, Crawford MA, Harpster SN, Jentsch JD (2013) Monoamine levels within the orbitofrontal cortex and putamen interact to predict reversal learning performance. Biol Psychiatry 73:756–762Google Scholar
  58. Groman SM, James AS, Seu E, Tran S, Clark TA, Harpster SN, Crawford M, Burtner JL, Feiler K, Roth RH, Elsworth JD, London ED, Jentsch JD (2014) In the blink of an eye: relating positive-feedback sensitivity to striatal dopamine D2-like receptors through blink rate. J Neurosci 34:14443–14454Google Scholar
  59. Groman SM, Smith NJ, Petrullli JR, Massi B, Chen L, Ropchan J, Huang Y, Lee D, Morris ED, Taylor JR (2016) Dopamine D3 receptor availability is associated with inflexible decision making. J Neurosci 36:6732–6741Google Scholar
  60. Groman SM, Rich KM, Smith NJ, Lee D, Taylor JR (2018) Chronic exposure to methamphetamine disrupts reinforcement-based decision making in rats. Neuropsychopharmacology 43:770–780Google Scholar
  61. Groman SM, Keistler C, Keip AJ, Hammarlund E, DiLeone RJ, Pittenger C, Lee D, Taylor JR (2019a) Orbitofrontal circuits control multiple reinforcement-learning processes. Neuron 103:734–746.e3Google Scholar
  62. Groman SM, Massi B, Mathias SR, Curry DW, Lee D, Taylor JR (2019b) Neurochemical and behavioral dissections of decision-making in a rodent multistage task. J Neurosci 39:295–306Google Scholar
  63. Groman SM, Massi B, Mathias SR, Lee D, Taylor JR (2019c) Model-free and model-based influences in addiction-related Behaviors. Biol Psychiatry 85:936–945Google Scholar
  64. Hasselmo ME, Bower JM (1993) Acetylcholine and memory. Trends Neurosci 16:218–222Google Scholar
  65. Hasz BM, Redish AD (2018) Deliberation and procedural automation on a two-step task for rats. Front Integr Neurosci 12:30Google Scholar
  66. Hauser TU, Iannaccone R, Ball J, Mathys C, Brandeis D, Walitza S, Brem S (2014) Role of the medial prefrontal cortex in impaired decision making in juvenile attention-deficit/hyperactivity disorder. JAMA Psychiat 71:1165Google Scholar
  67. Herold C (2010) NMDA and D2-like receptors modulate cognitive flexibility in a color discrimination reversal task in pigeons. Behav Neurosci 124:381–390Google Scholar
  68. Hollerman JR, Schultz W (1998) Dopamine neurons report an error in the temporal prediction of reward during learning. Nat Neurosci 1:304–309Google Scholar
  69. Ihalainen JA, Tanila H (2002) In vivo regulation of dopamine and noradrenaline release by α2A-adrenoceptors in the mouse prefrontal cortex. Eur J Neurosci 15:1789–1794Google Scholar
  70. Iigaya K, Fonseca MS, Murakami M, Mainen ZF, Dayan P (2018) An effect of serotonergic stimulation on learning rates for rewards apparent after long intertrial intervals. Nat Commun 9:2477Google Scholar
  71. Ishii H, Ohara S, Tobler PN, Tsutsui K-I, Iijima T (2015) Dopaminergic and serotonergic modulation of anterior insular and orbitofrontal cortex function in risky decision making. Neurosci Res 92:53–61Google Scholar
  72. Itami S, Uno H (2002) Orbitofrontal cortex dysfunction in attention-deficit hyperactivity disorder revealed by reversal and extinction tasks. Neuroreport 13:2453–2457Google Scholar
  73. Jentsch JD, Ashenhurst JR, Cervantes MC, Groman SM, James AS, Pennington ZT (2014) Dissecting impulsivity and its relationships to drug addictions. Ann N Y Acad Sci 1327:1–26Google Scholar
  74. Jing M et al (2018) A genetically encoded fluorescent acetylcholine indicator for in vitro and in vivo studies. Nat Biotechnol 36:726–737Google Scholar
  75. Jones BE, Moore RY (1977) Ascending projections of the locus coeruleus in the rat. II. Autoradiographic study. Brain Res 127:23–53Google Scholar
  76. Jupp B, Dalley JW (2014) Convergent pharmacological mechanisms in impulsivity and addiction: insights from rodent models. Br J Pharmacol 171:4729–4766Google Scholar
  77. Keiflin R, Pribut HJ, Shah NB, Janak PH (2019) Ventral tegmental dopamine neurons participate in reward identity predictions. Curr Biol 29(1):93–103. e3 [Epub ahead of print]Google Scholar
  78. Kim S, Bobeica I, Gamo NJ, Arnsten AFT, Lee D (2012) Effects of α-2A adrenergic receptor agonist on time and risk preference in primates. Psychopharmacology 219:363–375Google Scholar
  79. Kirby KN, Maraković NN (1996) Delay-discounting probabilistic rewards: rates decrease as amounts increase. Psychon Bull Rev 3:100–104Google Scholar
  80. Kjome KL, Lane SD, Schmitz JM, Green C, Ma L, Prasla I, Swann AC, Moeller FG (2010) Relationship between impulsivity and decision making in cocaine dependence. Psychiatry Res 178:299–304Google Scholar
  81. Klanker M, Sandberg T, Joosten R, Willuhn I, Feenstra M, Denys D (2015) Phasic dopamine release induced by positive feedback predicts individual differences in reversal learning. Neurobiol Learn Mem 125:135–145Google Scholar
  82. Kobayashi S, Schultz W (2008) Influence of reward delays on responses of dopamine neurons. J Neurosci 28:7837–7846Google Scholar
  83. Kolokotroni KZ, Rodgers RJ, Harrison AA (2011) Acute nicotine increases both impulsive choice and behavioural disinhibition in rats. Psychopharmacology 217:455–473Google Scholar
  84. Koot S, Zoratto F, Cassano T, Colangeli R, Laviola G, van den Bos R, Adriani W (2012) Compromised decision-making and increased gambling proneness following dietary serotonin depletion in rats. Neuropharmacology 62:1640–1650Google Scholar
  85. Laughlin RE, Grant TL, Williams RW, Jentsch JD (2011) Genetic dissection of behavioral flexibility: reversal learning in mice. Biol Psychiatry 69:1109–1116Google Scholar
  86. Lee D (2013) Decision making: from neuroscience to psychiatry. Neuron 78:233–248Google Scholar
  87. Lee B, Groman S, London ED, Jentsch JD (2007) Dopamine D2/D3 receptors play a specific role in the reversal of a learned visual discrimination in monkeys. Neuropsychopharmacology 32:2125–2134Google Scholar
  88. Lee D, Seo H, Jung MW (2012) Neural basis of reinforcement learning and decision making. Annu Rev Neurosci 35:287–308Google Scholar
  89. Lim MSM, Jocham G, Hunt LT, Behrens TEJ, Rogers RD (2015) Impulsivity and predictive control are associated with suboptimal action-selection and action-value learning in regular gamblers. Int Gambl Stud 15:489–505Google Scholar
  90. Logan GD (1994) On the ability to inhibit thought and action: a users’ signal paradigm. - PsycNET. In: Dagenbach D, Carr TH (eds) Inhibitory processes in attention, memory, and language. Academic Press, San DiegoGoogle Scholar
  91. Long AB, Kuhn CM, Platt ML (2009) Serotonin shapes risky decision making in monkeys. Soc Cogn Affect Neurosci 4:346–356Google Scholar
  92. Loos M, Pattij T, Janssen MCW, Counotte DS, Schoffelmeer ANM, Smit AB, Spijker S, Van Gaalen MM (2010) Dopamine receptor D1/D5 gene expression in the medial prefrontal cortex predicts impulsive choice in rats. Cereb Cortex 20:1064–1070Google Scholar
  93. Macoveanu J, Rowe JB, Hornboll B, Elliott R, Paulson OB, Knudsen GM, Siebner HR (2013) Serotonin 2A receptors contribute to the regulation of risk-averse decisions. NeuroImage 83:35–44Google Scholar
  94. Marvin JS, Borghuis BG, Tian L, Cichon J, Harnett MT, Akerboom J, Gordus A, Renninger SL, Chen T-W, Bargmann CI, Orger MB, Schreiter ER, Demb JB, Gan W-B, Hires SA, Looger LL (2013) An optimized fluorescent probe for visualizing glutamate neurotransmission. Nat Methods 10:162–170Google Scholar
  95. McEnaney KW, Butter CM (1969) Perseveration of responding and nonresponding in monkeys with orbital frontal ablations. J Comp Physiol Psychol 68:558–561Google Scholar
  96. McIntyre CK, Pal SN, Marriott LK, Gold PE (2002) Competition between memory systems: acetylcholine release in the hippocampus correlates negatively with good performance on an amygdala-dependent task. J Neurosci 22:1171–1176Google Scholar
  97. Mendez IA, Gilbert RJ, Bizon JL, Setlow B (2012) Effects of acute administration of nicotinic and muscarinic cholinergic agonists and antagonists on performance in different cost–benefit decision making tasks in rats. Psychopharmacology 224:489–499Google Scholar
  98. Mendez IA, Damborsky JC, Winzer-Serhan UH, Bizon JL, Setlow B (2013) α4β2∗ and α7 nicotinic acetylcholine receptor binding predicts choice preference in two cost benefit decision-making tasks. Neuroscience 230:121–131Google Scholar
  99. Miller KJ, Botvinick MM, Brody CD (2017) Dorsal hippocampus contributes to model-based planning. Nat Neurosci 20:1269–1276Google Scholar
  100. Mitchell MR, Weiss VG, Beas BS, Morgan D, Bizon JL, Setlow B (2014) Adolescent risk taking, cocaine self-administration, and striatal dopamine signaling. Neuropsychopharmacology 39:955–962Google Scholar
  101. Miyazaki KW, Miyazaki K, Doya K (2011) Activation of the central serotonergic system in response to delayed but not omitted rewards. Eur J Neurosci 33:153–160Google Scholar
  102. Miyazaki K, Miyazaki KW, Doya K (2012a) The role of serotonin in the regulation of patience and impulsivity. Mol Neurobiol 45:213–224Google Scholar
  103. Miyazaki KW, Miyazaki K, Doya K (2012b) Activation of dorsal raphe serotonin neurons is necessary for waiting for delayed rewards. J Neurosci 32:10451–10457Google Scholar
  104. Miyazaki KW, Miyazaki K, Tanaka KF, Yamanaka A, Takahashi A, Tabuchi S, Doya K (2014) Optogenetic activation of dorsal raphe serotonin neurons enhances patience for future rewards. Curr Biol 24:2033–2040Google Scholar
  105. Moeller FG, Barratt ES, Dougherty DM, Schmitz JM, Swann AC (2001) Psychiatric aspects of impulsivity. Am J Psychiatry 158:1783–1793Google Scholar
  106. Montague PR, Dolan RJ, Friston KJ, Dayan P (2012) Computational psychiatry. Trends Cogn Sci 16:72–80Google Scholar
  107. Montes DR, Stopper CM, Floresco SB (2015) Noradrenergic modulation of risk/reward decision making. Psychopharmacology 232:2681–2696Google Scholar
  108. Morris LS, Baek K, Kundu P, Harrison NA, Frank MJ, Voon V (2016) Biases in the explore–exploit tradeoff in addictions: the role of avoidance of uncertainty. Neuropsychopharmacology 41:940–948Google Scholar
  109. Moschak TM, Carelli RM (2017) Impulsive rats exhibit blunted dopamine release dynamics during a delay discounting task independent of cocaine history. eNeuro 4(2):ENEURO.0119-17.2017Google Scholar
  110. Najt P, Perez J, Sanches M, Peluso MAM, Glahn D, Soares JC (2007) Impulsivity and bipolar disorder. Eur Neuropsychopharmacol 17:313–320Google Scholar
  111. Nishikawa M, Diksic M, Sakai Y, Kumano H, Charney D, Palacios-Boix J, Negrete J, Gill K (2009) Alterations in brain serotonin synthesis in male alcoholics measured using positron emission tomography. Alcohol Clin Exp Res 33:233–239Google Scholar
  112. Nishitomi K, Yano K, Kobayashi M, Jino K, Kano T, Horiguchi N, Shinohara S, Hasegawa M (2018) Systemic administration of guanfacine improves food-motivated impulsive choice behavior primarily via direct stimulation of postsynaptic α2A-adrenergic receptors in rats. Behav Brain Res 345:21–29Google Scholar
  113. O’Neill M, Brown VJ (2007) The effect of striatal dopamine depletion and the adenosine A2A antagonist KW-6002 on reversal learning in rats. Neurobiol Learn Mem 88:75–81Google Scholar
  114. Onoda K, Abe S, Yamaguchi S (2010) Feedback-related negativity is correlated with unplanned impulsivity. Neuroreport 21:736–739Google Scholar
  115. Paton JJ, Belova MA, Morrison SE, Salzman CD (2006) The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 439:865–870Google Scholar
  116. Peters J, Büchel C (2009) Overlapping and distinct neural systems code for subjective value during intertemporal and risky decision making. J Neurosci 29:15727–15734Google Scholar
  117. Ragozzino ME, Choi D (2004) Dynamic changes in acetylcholine output in the medial striatum during place reversal learning. Learn Mem 11:70–77Google Scholar
  118. Ragozzino ME, Mohler EG, Prior M, Palencia CA, Rozman S (2009) Acetylcholine activity in selective striatal regions supports behavioral flexibility. Neurobiol Learn Mem 91:13–22Google Scholar
  119. Reiter AMF, Heinze H-J, Schlagenhauf F, Deserno L (2017) Impaired flexible reward-based decision-making in binge eating disorder: evidence from computational modeling and functional neuroimaging. Neuropsychopharmacology 42:628–637Google Scholar
  120. Reynolds B, de Wit H, Richards JB (2002) Delay of gratification and delay discounting in rats. Behav Process 59:157–168Google Scholar
  121. Reynolds B, Ortengren A, Richards JB, de Wit H (2006) Dimensions of impulsive behavior: personality and behavioral measures. Personal Individ Differ 40:305–315Google Scholar
  122. Richards JB, Mitchell SH, de Wit H, Seiden LS (1997) Determination of discount functions in rats with an adjusting-amount procedure. J Exp Anal Behav 67:353–366Google Scholar
  123. Ridley RM, Baker HF, Drewett B, Johnson JA (1985) Effects of ibotenic acid lesions of the basal forebrain on serial reversal learning in marmosets. Psychopharmacology 86:438–443Google Scholar
  124. Robbins TW (2002) The 5-choice serial reaction time task: behavioural pharmacology and functional neurochemistry. Psychopharmacology 163:362–380Google Scholar
  125. Robinson ESJ, Eagle DM, Mar AC, Bari A, Banerjee G, Jiang X, Dalley JW, Robbins TW (2008) Similar effects of the selective noradrenaline reuptake inhibitor atomoxetine on three distinct forms of impulsivity in the rat. Neuropsychopharmacology 33:1028–1037Google Scholar
  126. Robinson ESJ, Eagle DM, Economidou D, Theobald DEH, Mar AC, Murphy ER, Robbins TW, Dalley JW (2009) Behavioural characterisation of high impulsivity on the 5-choice serial reaction time task: specific deficits in ‘waiting’ versus ‘stopping.’. Behav Brain Res 196:310–316Google Scholar
  127. Rogers RD, Moeller FG, Swann AC, Clark L (2010) Recent research on impulsivity in individuals with drug use and mental health disorders: implications for alcoholism. Alcohol Clin Exp Res 34:1319–1333Google Scholar
  128. Sadacca BF, Wikenheiser AM, Schoenbaum G (2017) Toward a theoretical role for tonic norepinephrine in the orbitofrontal cortex in facilitating flexible learning. Neuroscience 345:124–129Google Scholar
  129. Saddoris MP, Sugam JA, Stuber GD, Witten IB, Deisseroth K, Carelli RM (2015) Mesolimbic dopamine dynamically tracks, and is causally linked to, discrete aspects of value-based decision making. Biol Psychiatry 77:903–911Google Scholar
  130. Schippers MC, Schetters D, De Vries TJ, Pattij T (2016) Differential effects of the pharmacological stressor yohimbine on impulsive decision making and response inhibition. Psychopharmacology 233:2775–2785Google Scholar
  131. Schweighofer N, Tanaka SC, Doya K (2007) Serotonin and the evaluation of future rewards: theory, experiments, and possible neural mechanisms. Ann N Y Acad Sci 1104:289–300Google Scholar
  132. Schweighofer N, Bertin M, Shishida K, Okamoto Y, Tanaka SC, Yamawaki S, Doya K (2008) Low-serotonin levels increase delayed reward discounting in humans. J Neurosci 28:4528–4532Google Scholar
  133. Seiden LS, Andresen J, MacPhail RC (1979) Methylphenidate and d-amphetamine: effects and interactions with alphamethyltyrosine and tetrabenazine on DRL performance in rats. Pharmacol Biochem Behav 10:577–584Google Scholar
  134. Selden N, Gitelman DR, Salamon-Murayama N, Parrish TB, Mesulam MM (1998) Trajectories of cholinergic pathways within the cerebral hemispheres of the human brain. Brain 121:2249–2257Google Scholar
  135. Seo M, Lee E, Averbeck BB (2012) Action selection and action value in frontal-striatal circuits. Neuron 74:947–960Google Scholar
  136. Seu E, David Jentsch J (2009) Effect of acute and repeated treatment with desipramine or methylphenidate on serial reversal learning in rats. Neuropharmacology 57:665–672Google Scholar
  137. Seu E, Lang A, Rivera RJ, Jentsch JD (2009) Inhibition of the norepinephrine transporter improves behavioral flexibility in rats and monkeys. Psychopharmacology 202:505–519Google Scholar
  138. Sharpe MJ, Chang CY, Liu MA, Batchelor HM, Mueller LE, Jones JL, Niv Y, Schoenbaum G (2017) Dopamine transients are sufficient and necessary for acquisition of model-based associations. Nat Neurosci 20:735–742Google Scholar
  139. Shirey JK, Brady AE, Jones PJ, Davis AA, Bridges TM, Kennedy JP, Jadhav SB, Menon UN, Xiang Z, Watson ML, Christian EP, Doherty JJ, Quirk MC, Snyder DH, Lah JJ, Levey AI, Nicolle MM, Lindsley CW, Conn PJ (2009) A selective allosteric potentiator of the M1 muscarinic acetylcholine receptor increases activity of medial prefrontal cortical neurons and restores impairments in reversal learning. J Neurosci 29:14271–14286Google Scholar
  140. Sigurdardottir HL, Kranz GS, Rami-Mark C, James GM, Vanicek T, Gryglewski G, Kautzky A, Hienert M, Traub-Weidinger T, Mitterhauser M, Wadsak W, Hacker M, Rujescu D, Kasper S, Lanzenberger R (2016) Effects of norepinephrine transporter gene variants on NET binding in ADHD and healthy controls investigated by PET. Hum Brain Mapp 37:884–895Google Scholar
  141. Sigurdardottir HL, Kranz GS, Rami-Mark C, James GM, Vanicek T, Gryglewski G et al (2019) Association of norepinephrine transporter methylation with in vivo NET expression and hyperactivity–impulsivity symptoms in ADHD measured with PET. Mol Psychiatry:1–10 [Epub ahead of print]Google Scholar
  142. Silveira MM, Malcolm E, Shoaib M, Winstanley CA (2015) Scopolamine and amphetamine produce similar decision-making deficits on a rat gambling task via independent pathways. Behav Brain Res 281:86–95Google Scholar
  143. Simon NW, Gilbert RJ, Mayse JD, Bizon JL, Setlow B (2009) Balancing risk and reward: a rat model of risky decision making. Neuropsychopharmacology 34:2208–2217Google Scholar
  144. Simon NW, Montgomery KS, Beas BS, Mitchell MR, LaSarge CL, Mendez IA, Bañuelos C, Vokes CM, Taylor AB, Haberman RP, Bizon JL, Setlow B (2011) Dopaminergic modulation of risky decision-making. J Neurosci 31:17460–17470Google Scholar
  145. St Onge JR, Floresco SB (2009) Dopaminergic modulation of risk-based decision making. Neuropsychopharmacology 34:681–697Google Scholar
  146. Stanger C, Ryan SR, Fu H, Landes RD, Jones BA, Bickel WK, Budney AJ (2012) Delay discounting predicts adolescent substance abuse treatment outcome. Exp Clin Psychopharmacol 20:205–212Google Scholar
  147. Steere JC, Arnsten AFT (1997) The α-2A noradrenergic receptor agonist guanfacine improves visual object discrimination reversal performance in aged rhesus monkeys. Behav Neurosci 111:883–891Google Scholar
  148. Stopper CM, Khayambashi S, Floresco SB (2013) Receptor-specific modulation of risk-based decision making by nucleus accumbens dopamine. Neuropsychopharmacology 38:715–728Google Scholar
  149. Sun F, Zeng J, Kreitzer AC, Cui G, Correspondence YL (2018) A genetically encoded fluorescent sensor enables rapid and specific detection of dopamine in flies, fish, and mice in brief the development of GPCR-activation-based-DA (GRAB DA) sensors enables measurements of dopamine dynamics in freely behaving animals with high spatiotemporal precision. Cell 174:481–496Google Scholar
  150. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, CambridgeGoogle Scholar
  151. Swann AC, Dougherty DM, Pazzaglia PJ, Pham M, Moeller FG (2004) Impulsivity: a link between bipolar disorder and substance abuse. Bipolar Disord 6:204–212Google Scholar
  152. Tanaka SC, Shishida K, Schweighofer N, Okamoto Y, Yamawaki S, Doya K (2009) Serotonin affects Association of Aversive outcomes to past actions. J Neurosci 29:15669–15674Google Scholar
  153. Tanno T, Maguire DR, Henson C, France CP (2014) Effects of amphetamine and methylphenidate on delay discounting in rats: interactions with order of delay presentation. Psychopharmacology 231:85–95Google Scholar
  154. Tedford SE, Persons AL, Napier TC (2015) Dopaminergic lesions of the dorsolateral striatum in rats increase delay discounting in an impulsive choice task Finkelstein DI, ed. PLoS One 10:e0122063Google Scholar
  155. Tzavos A, Jih J, Ragozzino ME (2004) Differential effects of M1 muscarinic receptor blockade and nicotinic receptor blockade in the dorsomedial striatum on response reversal learning. Behav Brain Res 154:245–253Google Scholar
  156. van der Meulen JAJ, Joosten RNJMA, de Bruin JPC, Feenstra MGP (2007) Dopamine and noradrenaline efflux in the medial prefrontal cortex during serial reversals and extinction of instrumental goal-directed behavior. Cereb Cortex 17:1444–1453Google Scholar
  157. Vanes LD, van Holst RJ, Jansen JM, van den Brink W, Oosterlaan J, Goudriaan AE (2014) Contingency learning in alcohol dependence and pathological gambling: learning and unlearning reward contingencies. Alcohol Clin Exp Res 38:1602–1610Google Scholar
  158. Wang X-J, Krystal JH (2014) Computational psychiatry. Neuron 84:638–654Google Scholar
  159. Weiland BJ, Heitzeg MM, Zald D, Cummiford C, Love T, Zucker RA, Zubieta J-K (2014) Relationship between impulsivity, prefrontal anticipatory activation, and striatal dopamine release during rewarded task performance. Psychiatry Res Neuroimaging 223:244–252Google Scholar
  160. Whiteside SP, Lynam DR (2001) The five factor model and impulsivity: using a structural model of personality to understand impulsivity. Personal Individ Differ 30:669–689Google Scholar
  161. Wilson VB, Mitchell SH, Musser ED, Schmitt CF, Nigg JT (2011) Delay discounting of reward in ADHD: application in young children. J Child Psychol Psychiatry 52:256–264Google Scholar
  162. Wilson RC, Takahashi YK, Schoenbaum G, Niv Y (2014) Orbitofrontal cortex as a cognitive map of task space. Neuron 81:267–279Google Scholar
  163. Winstanley CA, Dalley JW, Theobald DEH, Robbins TW (2003) Global 5-HT depletion attenuates the ability of amphetamine to decrease impulsive choice on a delay-discounting task in rats. Psychopharmacology 170:320–331Google Scholar
  164. Winstanley CA, Dalley JW, Theobald DE, Robbins TW (2004) Fractionating impulsivity: contrasting effects of central 5-HT depletion on different measures of impulsive behavior. Neuropsychopharmacology 29:1331–1343Google Scholar
  165. Woolf NJ, Butcher LL (1981) Cholinergic neurons in the caudate-putamen complex proper are intrinsically organized: a combined evans blue and acetylcholinesterase analysis. Brain Res Bull 7:487–507Google Scholar
  166. Yates JR, Bardo MT (2017) Effects of intra-accumbal administration of dopamine and ionotropic glutamate receptor drugs on delay discounting performance in rats. Behav Neurosci 131:392–405Google Scholar
  167. Young JW, Meves JM, Tarantino IS, Caldwell S, Geyer MA (2011) Delayed procedural learning in α7-nicotinic acetylcholine receptor knockout mice. Genes Brain Behav 10:720–733Google Scholar
  168. Zaichenko MI, Bazhenova DA, Grigor’yan GA, Merzhanova GK (2017) Does impulsivity influence the operation of long-term and working memory in rats? Neurosci Behav Physiol 47:427–434Google Scholar
  169. Zalocusky KA, Ramakrishnan C, Lerner TN, Davidson TJ, Knutson B, Deisseroth K (2016) Nucleus accumbens D2R cells signal prior outcomes and control risky decision-making. Nature 531:642–646Google Scholar
  170. Zeeb FD, Robbins TW, Winstanley CA (2009) Serotonergic and dopaminergic modulation of gambling behavior as assessed using a novel rat gambling task. Neuropsychopharmacology 34:2329–2343Google Scholar
  171. Zhang X-L, Wang G-B, Zhao L-Y, Sun L-L, Wang J, Wu P, Lu L, Shi J (2012) Clonidine improved laboratory-measured decision-making performance in abstinent heroin addicts Zhang XY, ed. PLoS One 7:e29084Google Scholar
  172. Zhukovsky P, Puaud M, Jupp B, Sala-Bayo J, Alsiö J, Xia J, Searle L, Morris Z, Sabir A, Giuliano C, Everitt BJ, Belin D, Robbins TW, Dalley JW (2019) Withdrawal from escalated cocaine self-administration impairs reversal learning by disrupting the effects of negative feedback on reward exploitation: a behavioral and computational analysis. Neuropsychopharmacology 44:2163–2173Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of PsychiatryYale UniversityNew HavenUSA

Personalised recommendations