Abstract
Impulsive decisions are those that favor immediate over delayed rewards, involve the acceptance of undue risk or uncertainty, or fail to adapt to environmental changes. Pathological levels of impulsive decision-making have been observed in individuals with mental illness, but there may be substantial heterogeneity in the processes that drive impulsive choices. Understanding this behavioral heterogeneity may be critical for understanding associated diverseness in the neural mechanisms that give rise to impulsivity. The application of reinforcement learning algorithms in the deconstruction of impulsive decision-making phenotypes can help bridge the gap between biology and behavior and provide insights into the biobehavioral heterogeneity of impulsive choice. This chapter will review the literature on the neurobiological mechanisms of impulsive decision-making in nonhuman animals; specifically, the role of the amine neuromodulatory systems (dopamine, serotonin, norepinephrine, and acetylcholine) in impulsive decision-making and reinforcement learning processes is discussed. Ultimately, the integration of reinforcement learning algorithms with sophisticated behavioral and neuroscience techniques may be critical for advancing the understanding of the neurochemical basis of impulsive decision-making.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abela AR, Chudasama Y (2014) Noradrenergic α2A-receptor stimulation in the ventral hippocampus reduces impulsive decision-making. Psychopharmacology 231:521–531
Addicott MA, Pearson JM, Wilson J, Platt ML, McClernon FJ (2013) Smoking and the bandit: a preliminary study of smoker and nonsmoker differences in exploratory behavior measured with a multiarmed bandit task. Exp Clin Psychopharmacol 21:66–73
Aharonovich E, Hasin DS, Brooks AC, Liu X, Bisaga A, Nunes EV (2006) Cognitive deficits predict low treatment retention in cocaine dependent patients. Drug Alcohol Depend 81:313–322
Ahn W-Y, Rass O, Fridberg DJ, Bishara AJ, Forsyth JK, Breier A, Busemeyer JR, Hetrick WP, Bolbecker AR, O’Donnell BF (2011) Temporal discounting of rewards in patients with bipolar disorder and schizophrenia. J Abnorm Psychol 120:911–921
Alsiö J, Nilsson SRO, Gastambide F, Wang RAH, Dam SA, Mar AC, Tricklebank M, Robbins TW (2015) The role of 5-HT2C receptors in touchscreen visual reversal learning in the rat: a cross-site study. Psychopharmacology 232:4017–4031
Aston-Jones G, Cohen JD (2005) Adaptive gain and the role of the locus coeruleus-norepinephrine system in optimal performance. J Comp Neurol 493:99–110
Aston-Jones G, Rajkowski J, Kubiak P (1997) Conditioned responses of monkey locus coeruleus neurons anticipate acquisition of discriminative behavior in a vigilance task. Neuroscience 80:697–715
Baarendse PJJ, Vanderschuren LJMJ (2012) Dissociable effects of monoamine reuptake inhibitors on distinct forms of impulsive behavior in rats. Psychopharmacology 219:313–326
Baarendse PJJ, Winstanley CA, Vanderschuren LJMJ (2013) Simultaneous blockade of dopamine and noradrenaline reuptake promotes disadvantageous decision making in a rat gambling task. Psychopharmacology 225:719–731
Bari A, Robbins TW (2013) Inhibition and impulsivity: behavioral and neural basis of response control. Prog Neurobiol 108:44–79
Bari A, Theobald DE, Caprioli D, Mar AC, Aidoo-Micah A, Dalley JW, Robbins TW (2010) Serotonin modulates sensitivity to reward and negative feedback in a probabilistic reversal learning task in rats. Neuropsychopharmacology 35:1290–1301
Barkley RA (1997) Behavioral inhibition, sustained attention, and executive functions: constructing a unifying theory of ADHD. Psychol Bull 121:65–94
Barrus MM, Winstanley CA (2016) Dopamine D3 receptors modulate the ability of win-paired cues to increase risky choice in a rat gambling task. J Neurosci 36:785–794
Bizot JC, Thiébot MH, Le Bihan C, Soubrié P, Simon P (1988) Effects of imipramine-like drugs and serotonin uptake blockers on delay of reward in rats. Possible implication in the behavioral mechanism of action of antidepressants. J Pharmacol Exp Ther 246:1144–1151
Bizot J, Le Bihan C, Puech AJ, Hamon M, Thiébot M (1999) Serotonin and tolerance to delay of reward in rats. Psychopharmacology 146:400–412
Bjork JM, Hommer DW, Grant SJ, Danube C (2004) Impulsivity in abstinent alcohol-dependent patients: relation to control subjects and type 1–/type 2–like traits. Alcohol 34:133–150
Blaes SL, Orsini CA, Mitchell MR, Spurrell MS, Betzhold SM, Vera K, Bizon JL, Setlow B (2018) Monoaminergic modulation of decision-making under risk of punishment in a rat model. Behav Pharmacol 29:745–761
Boulougouris V, Robbins TW (2010) Enhancement of spatial reversal learning by 5-HT2C receptor antagonism is neuroanatomically specific. J Neurosci 30:930–938
Boulougouris V, Glennon JC, Robbins TW (2008) Dissociable effects of selective 5-HT2A and 5-HT2C receptor antagonists on serial spatial reversal learning in rats. Neuropsychopharmacology 33:2007–2019
Boulougouris V, Castane A, Robbins TW (2009) Dopamine D2/D3 receptor agonist quinpirole impairs spatial reversal learning in rats: investigation of D3 receptor involvement in persistent behavior. Psychopharmacology 202:611–620
Bouret S, Sara SJ (2004) Reward expectation, orientation of attention and locus coeruleus-medial frontal cortex interplay during learning. Eur J Neurosci 20:791–802
Bradfield LA, Bertran-Gonzalez J, Chieng B, Balleine BW (2013) The thalamostriatal pathway and cholinergic control of goal-directed action: interlacing new with existing learning in the striatum. Neuron 79:153–166
Brigman JL, Mathur P, Harvey-White J, Izquierdo A, Saksida LM, Bussey TJ, Fox S, Deneris E, Murphy DL, Holmes A (2010) Pharmacological or genetic inactivation of the serotonin transporter improves reversal learning in mice. Cereb Cortex 20:1955–1963
Bymaster F, Katner JS, Nelson DL, Hemrick-Luecke SK, Threlkeld PG, Heiligenstein JH, Morin SM, Gehlert DR, Perry KW (2002) Atomoxetine increases extracellular levels of norepinephrine and dopamine in prefrontal cortex of rat a potential mechanism for efficacy in attention deficit/hyperactivity disorder. Neuropsychopharmacology 27:699–711
Cannon DM, Ichise M, Rollis D, Klaver JM, Gandhi SK, Charney DS, Manji HK, Drevets WC (2007) Elevated serotonin transporter binding in major depressive disorder assessed using positron emission tomography and [11C]DASB; comparison with bipolar disorder. Biol Psychiatry 62:870–877
Cardinal RN, Pennicott DR, Sugathapala CL, Robbins TW, Everitt BJ (2001) Impulsive choice induced in rats by lesions of the nucleus accumbens core. Science 292:2499–2501
Chang CY, Gardner MPH, Conroy JC, Whitaker LR, Schoenbaum G (2018) Brief, but not prolonged, pauses in the firing of midbrain dopamine neurons are sufficient to produce a conditioned inhibitor. J Neurosci 38:8822–8830
Chopin P, Colpaert FC, Marien M (1999) Effects of alpha-2 adrenoceptor agonists and antagonists on circling behavior in rats with unilateral 6-hydroxydopamine lesions of the nigrostriatal pathway. J Pharmacol Exp Ther 288:798–804
Clark L, Robbins TW, Ersche KD, Sahakian BJ (2006) Reflection impulsivity in current and former substance users. Biol Psychiatry 60:515–522
Clarke HF, Dalley JW, Crofts HS, Robbins TW, Roberts AC (2004) Cognitive inflexibility after prefrontal serotonin depletion. Science 304:878–880
Clarke HF, Hill GJ, Robbins TW, Roberts AC (2011) Dopamine, but not serotonin, regulates reversal learning in the marmoset caudate nucleus. J Neurosci 31:4290–4297
Cools R, D’Esposito M (2011) Inverted-U–shaped dopamine actions on human working memory and cognitive control. Biol Psychiatry 69:e113–e125
Costa VD, Tran VL, Turchi J, Averbeck BB (2015) Reversal learning and dopamine: a Bayesian perspective. J Neurosci 35:2407–2416
Costa VD, Dal Monte O, Lucas DR, Murray EA, Averbeck BB (2016) Amygdala and ventral striatum make distinct contributions to reinforcement learning. Neuron 92:505–517
Courtney KE, Arellano R, Barkley-Levenson E, Gálvan A, Poldrack RA, MacKillop J, David Jentsch J, Ray LA (2012) The relationship between measures of impulsivity and alcohol misuse: an integrative structural equation modeling approach. Alcohol Clin Exp Res 36:923–931
Craig AR, Maxfield AD, Stein JS, Renda CR, Madden GJ (2014) Do the adjusting-delay and increasing-delay tasks measure the same construct: delay discounting? Behav Pharmacol 25:306–315
Crean JP, de Wit H, Richards JB (2000) Reward discounting as a measure of impulsive behavior in a psychiatric outpatient population. Exp Clin Psychopharmacol 8:155–162
Crockett MJ, Clark L, Lieberman MD, Tabibnia G, Robbins TW (2010) Impulsive choice and altruistic punishment are correlated and increase in tandem with serotonin depletion. Emotion 10:855–862
Dalley JW, Robbins TW (2017) Fractionating impulsivity: neuropsychiatric implications. Nat Rev Neurosci 18:158–171
Dautan D, Huerta-Ocampo I, Witten IB, Deisseroth K, Bolam JP, Gerdjikov T, Mena-Segovia J (2014) A major external source of cholinergic innervation of the striatum and nucleus accumbens originates in the brainstem. J Neurosci 34:4509–4518
Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans’ choices and striatal prediction errors. Neuron 69:1204–1215
Dayan P, Daw ND (2008) Decision theory, reinforcement learning, and the brain. Cogn Affect Behav Neurosci 8:429–453
Deserno L, Wilbertz T, Reiter A, Horstmann A, Neumann J, Villringer A, Heinze H-J, Schlagenhauf F (2015) Lateral prefrontal model-based signatures are reduced in healthy individuals with high trait impulsivity. Transl Psychiatry 5:e659
Dom G, D’haene P, Hulstijn W, Sabbe B (2006) Impulsivity in abstinent early- and late-onset alcoholics: differences in self-report measures and a discounting task. Addiction 101:50–59
Doya K (2002) Metalearning and neuromodulation. Neural Netw 15:495–506
Doya K, Samejima K, Katagiri K, Kawato M (2002) Multiple model-based reinforcement learning. Neural Comput 14:1347–1369
Eagle DM, Baunez C, Hutcheson DM, Lehmann O, Shah AP, Robbins TW (2008) Stop-signal reaction-time task performance: role of prefrontal cortex and subthalamic nucleus. Cereb Cortex 18:178–188
Enticott PG, Ogloff JRP, Bradshaw JL (2006) Associations between laboratory measures of executive inhibitory control and self-reported impulsivity. Personal Individ Differ 41:285–294
Evenden JL (1999) Varieties of impulsivity. Psychopharmacology 146:348–361
Fernie G, Cole JC, Goudie AJ, Field M (2010) Risk-taking but not response inhibition or delay discounting predict alcohol consumption in social drinkers. Drug Alcohol Depend 112:54–61
Fillmore MT, Rush CR (2006) Polydrug abusers display impaired discrimination-reversal learning in a model of behavioural control. J Psychopharmacol 20:24–32
Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299:1898–1902
Fishbein DH, Eldreth DL, Hyde C, Matochik JA, London ED, Contoreggi C, Kurian V, Kimes AS, Breeden A, Grant S (2005) Risky decision making and the anterior cingulate cortex in abstinent drug abusers and nonusers. Cogn Brain Res 23:119–136
Gläscher J, Daw N, Dayan P, O’Doherty JP (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66:585–595
Gold PE (2003) Acetylcholine modulation of neural systems involved in learning and memory. Neurobiol Learn Mem 80:194–210
Groman SM, Lee B, London ED, Mandelkern MA, James AS, Feiler K, Rivera R, Dahlbom M, Sossi V, Vandervoort E, Jentsch JD (2011) Dorsal striatal D2-like receptor availability covaries with sensitivity to positive reinforcement during discrimination learning. J Neurosci 31:7291–7299
Groman SM, James AS, Seu E, Crawford MA, Harpster SN, Jentsch JD (2013) Monoamine levels within the orbitofrontal cortex and putamen interact to predict reversal learning performance. Biol Psychiatry 73:756–762
Groman SM, James AS, Seu E, Tran S, Clark TA, Harpster SN, Crawford M, Burtner JL, Feiler K, Roth RH, Elsworth JD, London ED, Jentsch JD (2014) In the blink of an eye: relating positive-feedback sensitivity to striatal dopamine D2-like receptors through blink rate. J Neurosci 34:14443–14454
Groman SM, Smith NJ, Petrullli JR, Massi B, Chen L, Ropchan J, Huang Y, Lee D, Morris ED, Taylor JR (2016) Dopamine D3 receptor availability is associated with inflexible decision making. J Neurosci 36:6732–6741
Groman SM, Rich KM, Smith NJ, Lee D, Taylor JR (2018) Chronic exposure to methamphetamine disrupts reinforcement-based decision making in rats. Neuropsychopharmacology 43:770–780
Groman SM, Keistler C, Keip AJ, Hammarlund E, DiLeone RJ, Pittenger C, Lee D, Taylor JR (2019a) Orbitofrontal circuits control multiple reinforcement-learning processes. Neuron 103:734–746.e3
Groman SM, Massi B, Mathias SR, Curry DW, Lee D, Taylor JR (2019b) Neurochemical and behavioral dissections of decision-making in a rodent multistage task. J Neurosci 39:295–306
Groman SM, Massi B, Mathias SR, Lee D, Taylor JR (2019c) Model-free and model-based influences in addiction-related Behaviors. Biol Psychiatry 85:936–945
Hasselmo ME, Bower JM (1993) Acetylcholine and memory. Trends Neurosci 16:218–222
Hasz BM, Redish AD (2018) Deliberation and procedural automation on a two-step task for rats. Front Integr Neurosci 12:30
Hauser TU, Iannaccone R, Ball J, Mathys C, Brandeis D, Walitza S, Brem S (2014) Role of the medial prefrontal cortex in impaired decision making in juvenile attention-deficit/hyperactivity disorder. JAMA Psychiat 71:1165
Herold C (2010) NMDA and D2-like receptors modulate cognitive flexibility in a color discrimination reversal task in pigeons. Behav Neurosci 124:381–390
Hollerman JR, Schultz W (1998) Dopamine neurons report an error in the temporal prediction of reward during learning. Nat Neurosci 1:304–309
Ihalainen JA, Tanila H (2002) In vivo regulation of dopamine and noradrenaline release by α2A-adrenoceptors in the mouse prefrontal cortex. Eur J Neurosci 15:1789–1794
Iigaya K, Fonseca MS, Murakami M, Mainen ZF, Dayan P (2018) An effect of serotonergic stimulation on learning rates for rewards apparent after long intertrial intervals. Nat Commun 9:2477
Ishii H, Ohara S, Tobler PN, Tsutsui K-I, Iijima T (2015) Dopaminergic and serotonergic modulation of anterior insular and orbitofrontal cortex function in risky decision making. Neurosci Res 92:53–61
Itami S, Uno H (2002) Orbitofrontal cortex dysfunction in attention-deficit hyperactivity disorder revealed by reversal and extinction tasks. Neuroreport 13:2453–2457
Jentsch JD, Ashenhurst JR, Cervantes MC, Groman SM, James AS, Pennington ZT (2014) Dissecting impulsivity and its relationships to drug addictions. Ann N Y Acad Sci 1327:1–26
Jing M et al (2018) A genetically encoded fluorescent acetylcholine indicator for in vitro and in vivo studies. Nat Biotechnol 36:726–737
Jones BE, Moore RY (1977) Ascending projections of the locus coeruleus in the rat. II. Autoradiographic study. Brain Res 127:23–53
Jupp B, Dalley JW (2014) Convergent pharmacological mechanisms in impulsivity and addiction: insights from rodent models. Br J Pharmacol 171:4729–4766
Keiflin R, Pribut HJ, Shah NB, Janak PH (2019) Ventral tegmental dopamine neurons participate in reward identity predictions. Curr Biol 29(1):93–103. e3 [Epub ahead of print]
Kim S, Bobeica I, Gamo NJ, Arnsten AFT, Lee D (2012) Effects of α-2A adrenergic receptor agonist on time and risk preference in primates. Psychopharmacology 219:363–375
Kirby KN, Maraković NN (1996) Delay-discounting probabilistic rewards: rates decrease as amounts increase. Psychon Bull Rev 3:100–104
Kjome KL, Lane SD, Schmitz JM, Green C, Ma L, Prasla I, Swann AC, Moeller FG (2010) Relationship between impulsivity and decision making in cocaine dependence. Psychiatry Res 178:299–304
Klanker M, Sandberg T, Joosten R, Willuhn I, Feenstra M, Denys D (2015) Phasic dopamine release induced by positive feedback predicts individual differences in reversal learning. Neurobiol Learn Mem 125:135–145
Kobayashi S, Schultz W (2008) Influence of reward delays on responses of dopamine neurons. J Neurosci 28:7837–7846
Kolokotroni KZ, Rodgers RJ, Harrison AA (2011) Acute nicotine increases both impulsive choice and behavioural disinhibition in rats. Psychopharmacology 217:455–473
Koot S, Zoratto F, Cassano T, Colangeli R, Laviola G, van den Bos R, Adriani W (2012) Compromised decision-making and increased gambling proneness following dietary serotonin depletion in rats. Neuropharmacology 62:1640–1650
Laughlin RE, Grant TL, Williams RW, Jentsch JD (2011) Genetic dissection of behavioral flexibility: reversal learning in mice. Biol Psychiatry 69:1109–1116
Lee D (2013) Decision making: from neuroscience to psychiatry. Neuron 78:233–248
Lee B, Groman S, London ED, Jentsch JD (2007) Dopamine D2/D3 receptors play a specific role in the reversal of a learned visual discrimination in monkeys. Neuropsychopharmacology 32:2125–2134
Lee D, Seo H, Jung MW (2012) Neural basis of reinforcement learning and decision making. Annu Rev Neurosci 35:287–308
Lim MSM, Jocham G, Hunt LT, Behrens TEJ, Rogers RD (2015) Impulsivity and predictive control are associated with suboptimal action-selection and action-value learning in regular gamblers. Int Gambl Stud 15:489–505
Logan GD (1994) On the ability to inhibit thought and action: a users’ signal paradigm. - PsycNET. In: Dagenbach D, Carr TH (eds) Inhibitory processes in attention, memory, and language. Academic Press, San Diego
Long AB, Kuhn CM, Platt ML (2009) Serotonin shapes risky decision making in monkeys. Soc Cogn Affect Neurosci 4:346–356
Loos M, Pattij T, Janssen MCW, Counotte DS, Schoffelmeer ANM, Smit AB, Spijker S, Van Gaalen MM (2010) Dopamine receptor D1/D5 gene expression in the medial prefrontal cortex predicts impulsive choice in rats. Cereb Cortex 20:1064–1070
Macoveanu J, Rowe JB, Hornboll B, Elliott R, Paulson OB, Knudsen GM, Siebner HR (2013) Serotonin 2A receptors contribute to the regulation of risk-averse decisions. NeuroImage 83:35–44
Marvin JS, Borghuis BG, Tian L, Cichon J, Harnett MT, Akerboom J, Gordus A, Renninger SL, Chen T-W, Bargmann CI, Orger MB, Schreiter ER, Demb JB, Gan W-B, Hires SA, Looger LL (2013) An optimized fluorescent probe for visualizing glutamate neurotransmission. Nat Methods 10:162–170
McEnaney KW, Butter CM (1969) Perseveration of responding and nonresponding in monkeys with orbital frontal ablations. J Comp Physiol Psychol 68:558–561
McIntyre CK, Pal SN, Marriott LK, Gold PE (2002) Competition between memory systems: acetylcholine release in the hippocampus correlates negatively with good performance on an amygdala-dependent task. J Neurosci 22:1171–1176
Mendez IA, Gilbert RJ, Bizon JL, Setlow B (2012) Effects of acute administration of nicotinic and muscarinic cholinergic agonists and antagonists on performance in different cost–benefit decision making tasks in rats. Psychopharmacology 224:489–499
Mendez IA, Damborsky JC, Winzer-Serhan UH, Bizon JL, Setlow B (2013) α4β2∗ and α7 nicotinic acetylcholine receptor binding predicts choice preference in two cost benefit decision-making tasks. Neuroscience 230:121–131
Miller KJ, Botvinick MM, Brody CD (2017) Dorsal hippocampus contributes to model-based planning. Nat Neurosci 20:1269–1276
Mitchell MR, Weiss VG, Beas BS, Morgan D, Bizon JL, Setlow B (2014) Adolescent risk taking, cocaine self-administration, and striatal dopamine signaling. Neuropsychopharmacology 39:955–962
Miyazaki KW, Miyazaki K, Doya K (2011) Activation of the central serotonergic system in response to delayed but not omitted rewards. Eur J Neurosci 33:153–160
Miyazaki K, Miyazaki KW, Doya K (2012a) The role of serotonin in the regulation of patience and impulsivity. Mol Neurobiol 45:213–224
Miyazaki KW, Miyazaki K, Doya K (2012b) Activation of dorsal raphe serotonin neurons is necessary for waiting for delayed rewards. J Neurosci 32:10451–10457
Miyazaki KW, Miyazaki K, Tanaka KF, Yamanaka A, Takahashi A, Tabuchi S, Doya K (2014) Optogenetic activation of dorsal raphe serotonin neurons enhances patience for future rewards. Curr Biol 24:2033–2040
Moeller FG, Barratt ES, Dougherty DM, Schmitz JM, Swann AC (2001) Psychiatric aspects of impulsivity. Am J Psychiatry 158:1783–1793
Montague PR, Dolan RJ, Friston KJ, Dayan P (2012) Computational psychiatry. Trends Cogn Sci 16:72–80
Montes DR, Stopper CM, Floresco SB (2015) Noradrenergic modulation of risk/reward decision making. Psychopharmacology 232:2681–2696
Morris LS, Baek K, Kundu P, Harrison NA, Frank MJ, Voon V (2016) Biases in the explore–exploit tradeoff in addictions: the role of avoidance of uncertainty. Neuropsychopharmacology 41:940–948
Moschak TM, Carelli RM (2017) Impulsive rats exhibit blunted dopamine release dynamics during a delay discounting task independent of cocaine history. eNeuro 4(2):ENEURO.0119-17.2017
Najt P, Perez J, Sanches M, Peluso MAM, Glahn D, Soares JC (2007) Impulsivity and bipolar disorder. Eur Neuropsychopharmacol 17:313–320
Nishikawa M, Diksic M, Sakai Y, Kumano H, Charney D, Palacios-Boix J, Negrete J, Gill K (2009) Alterations in brain serotonin synthesis in male alcoholics measured using positron emission tomography. Alcohol Clin Exp Res 33:233–239
Nishitomi K, Yano K, Kobayashi M, Jino K, Kano T, Horiguchi N, Shinohara S, Hasegawa M (2018) Systemic administration of guanfacine improves food-motivated impulsive choice behavior primarily via direct stimulation of postsynaptic α2A-adrenergic receptors in rats. Behav Brain Res 345:21–29
O’Neill M, Brown VJ (2007) The effect of striatal dopamine depletion and the adenosine A2A antagonist KW-6002 on reversal learning in rats. Neurobiol Learn Mem 88:75–81
Onoda K, Abe S, Yamaguchi S (2010) Feedback-related negativity is correlated with unplanned impulsivity. Neuroreport 21:736–739
Paton JJ, Belova MA, Morrison SE, Salzman CD (2006) The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 439:865–870
Peters J, Büchel C (2009) Overlapping and distinct neural systems code for subjective value during intertemporal and risky decision making. J Neurosci 29:15727–15734
Ragozzino ME, Choi D (2004) Dynamic changes in acetylcholine output in the medial striatum during place reversal learning. Learn Mem 11:70–77
Ragozzino ME, Mohler EG, Prior M, Palencia CA, Rozman S (2009) Acetylcholine activity in selective striatal regions supports behavioral flexibility. Neurobiol Learn Mem 91:13–22
Reiter AMF, Heinze H-J, Schlagenhauf F, Deserno L (2017) Impaired flexible reward-based decision-making in binge eating disorder: evidence from computational modeling and functional neuroimaging. Neuropsychopharmacology 42:628–637
Reynolds B, de Wit H, Richards JB (2002) Delay of gratification and delay discounting in rats. Behav Process 59:157–168
Reynolds B, Ortengren A, Richards JB, de Wit H (2006) Dimensions of impulsive behavior: personality and behavioral measures. Personal Individ Differ 40:305–315
Richards JB, Mitchell SH, de Wit H, Seiden LS (1997) Determination of discount functions in rats with an adjusting-amount procedure. J Exp Anal Behav 67:353–366
Ridley RM, Baker HF, Drewett B, Johnson JA (1985) Effects of ibotenic acid lesions of the basal forebrain on serial reversal learning in marmosets. Psychopharmacology 86:438–443
Robbins TW (2002) The 5-choice serial reaction time task: behavioural pharmacology and functional neurochemistry. Psychopharmacology 163:362–380
Robinson ESJ, Eagle DM, Mar AC, Bari A, Banerjee G, Jiang X, Dalley JW, Robbins TW (2008) Similar effects of the selective noradrenaline reuptake inhibitor atomoxetine on three distinct forms of impulsivity in the rat. Neuropsychopharmacology 33:1028–1037
Robinson ESJ, Eagle DM, Economidou D, Theobald DEH, Mar AC, Murphy ER, Robbins TW, Dalley JW (2009) Behavioural characterisation of high impulsivity on the 5-choice serial reaction time task: specific deficits in ‘waiting’ versus ‘stopping.’. Behav Brain Res 196:310–316
Rogers RD, Moeller FG, Swann AC, Clark L (2010) Recent research on impulsivity in individuals with drug use and mental health disorders: implications for alcoholism. Alcohol Clin Exp Res 34:1319–1333
Sadacca BF, Wikenheiser AM, Schoenbaum G (2017) Toward a theoretical role for tonic norepinephrine in the orbitofrontal cortex in facilitating flexible learning. Neuroscience 345:124–129
Saddoris MP, Sugam JA, Stuber GD, Witten IB, Deisseroth K, Carelli RM (2015) Mesolimbic dopamine dynamically tracks, and is causally linked to, discrete aspects of value-based decision making. Biol Psychiatry 77:903–911
Schippers MC, Schetters D, De Vries TJ, Pattij T (2016) Differential effects of the pharmacological stressor yohimbine on impulsive decision making and response inhibition. Psychopharmacology 233:2775–2785
Schweighofer N, Tanaka SC, Doya K (2007) Serotonin and the evaluation of future rewards: theory, experiments, and possible neural mechanisms. Ann N Y Acad Sci 1104:289–300
Schweighofer N, Bertin M, Shishida K, Okamoto Y, Tanaka SC, Yamawaki S, Doya K (2008) Low-serotonin levels increase delayed reward discounting in humans. J Neurosci 28:4528–4532
Seiden LS, Andresen J, MacPhail RC (1979) Methylphenidate and d-amphetamine: effects and interactions with alphamethyltyrosine and tetrabenazine on DRL performance in rats. Pharmacol Biochem Behav 10:577–584
Selden N, Gitelman DR, Salamon-Murayama N, Parrish TB, Mesulam MM (1998) Trajectories of cholinergic pathways within the cerebral hemispheres of the human brain. Brain 121:2249–2257
Seo M, Lee E, Averbeck BB (2012) Action selection and action value in frontal-striatal circuits. Neuron 74:947–960
Seu E, David Jentsch J (2009) Effect of acute and repeated treatment with desipramine or methylphenidate on serial reversal learning in rats. Neuropharmacology 57:665–672
Seu E, Lang A, Rivera RJ, Jentsch JD (2009) Inhibition of the norepinephrine transporter improves behavioral flexibility in rats and monkeys. Psychopharmacology 202:505–519
Sharpe MJ, Chang CY, Liu MA, Batchelor HM, Mueller LE, Jones JL, Niv Y, Schoenbaum G (2017) Dopamine transients are sufficient and necessary for acquisition of model-based associations. Nat Neurosci 20:735–742
Shirey JK, Brady AE, Jones PJ, Davis AA, Bridges TM, Kennedy JP, Jadhav SB, Menon UN, Xiang Z, Watson ML, Christian EP, Doherty JJ, Quirk MC, Snyder DH, Lah JJ, Levey AI, Nicolle MM, Lindsley CW, Conn PJ (2009) A selective allosteric potentiator of the M1 muscarinic acetylcholine receptor increases activity of medial prefrontal cortical neurons and restores impairments in reversal learning. J Neurosci 29:14271–14286
Sigurdardottir HL, Kranz GS, Rami-Mark C, James GM, Vanicek T, Gryglewski G, Kautzky A, Hienert M, Traub-Weidinger T, Mitterhauser M, Wadsak W, Hacker M, Rujescu D, Kasper S, Lanzenberger R (2016) Effects of norepinephrine transporter gene variants on NET binding in ADHD and healthy controls investigated by PET. Hum Brain Mapp 37:884–895
Sigurdardottir HL, Kranz GS, Rami-Mark C, James GM, Vanicek T, Gryglewski G et al (2019) Association of norepinephrine transporter methylation with in vivo NET expression and hyperactivity–impulsivity symptoms in ADHD measured with PET. Mol Psychiatry:1–10 [Epub ahead of print]
Silveira MM, Malcolm E, Shoaib M, Winstanley CA (2015) Scopolamine and amphetamine produce similar decision-making deficits on a rat gambling task via independent pathways. Behav Brain Res 281:86–95
Simon NW, Gilbert RJ, Mayse JD, Bizon JL, Setlow B (2009) Balancing risk and reward: a rat model of risky decision making. Neuropsychopharmacology 34:2208–2217
Simon NW, Montgomery KS, Beas BS, Mitchell MR, LaSarge CL, Mendez IA, Bañuelos C, Vokes CM, Taylor AB, Haberman RP, Bizon JL, Setlow B (2011) Dopaminergic modulation of risky decision-making. J Neurosci 31:17460–17470
St Onge JR, Floresco SB (2009) Dopaminergic modulation of risk-based decision making. Neuropsychopharmacology 34:681–697
Stanger C, Ryan SR, Fu H, Landes RD, Jones BA, Bickel WK, Budney AJ (2012) Delay discounting predicts adolescent substance abuse treatment outcome. Exp Clin Psychopharmacol 20:205–212
Steere JC, Arnsten AFT (1997) The α-2A noradrenergic receptor agonist guanfacine improves visual object discrimination reversal performance in aged rhesus monkeys. Behav Neurosci 111:883–891
Stopper CM, Khayambashi S, Floresco SB (2013) Receptor-specific modulation of risk-based decision making by nucleus accumbens dopamine. Neuropsychopharmacology 38:715–728
Sun F, Zeng J, Kreitzer AC, Cui G, Correspondence YL (2018) A genetically encoded fluorescent sensor enables rapid and specific detection of dopamine in flies, fish, and mice in brief the development of GPCR-activation-based-DA (GRAB DA) sensors enables measurements of dopamine dynamics in freely behaving animals with high spatiotemporal precision. Cell 174:481–496
Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge
Swann AC, Dougherty DM, Pazzaglia PJ, Pham M, Moeller FG (2004) Impulsivity: a link between bipolar disorder and substance abuse. Bipolar Disord 6:204–212
Tanaka SC, Shishida K, Schweighofer N, Okamoto Y, Yamawaki S, Doya K (2009) Serotonin affects Association of Aversive outcomes to past actions. J Neurosci 29:15669–15674
Tanno T, Maguire DR, Henson C, France CP (2014) Effects of amphetamine and methylphenidate on delay discounting in rats: interactions with order of delay presentation. Psychopharmacology 231:85–95
Tedford SE, Persons AL, Napier TC (2015) Dopaminergic lesions of the dorsolateral striatum in rats increase delay discounting in an impulsive choice task Finkelstein DI, ed. PLoS One 10:e0122063
Tzavos A, Jih J, Ragozzino ME (2004) Differential effects of M1 muscarinic receptor blockade and nicotinic receptor blockade in the dorsomedial striatum on response reversal learning. Behav Brain Res 154:245–253
van der Meulen JAJ, Joosten RNJMA, de Bruin JPC, Feenstra MGP (2007) Dopamine and noradrenaline efflux in the medial prefrontal cortex during serial reversals and extinction of instrumental goal-directed behavior. Cereb Cortex 17:1444–1453
Vanes LD, van Holst RJ, Jansen JM, van den Brink W, Oosterlaan J, Goudriaan AE (2014) Contingency learning in alcohol dependence and pathological gambling: learning and unlearning reward contingencies. Alcohol Clin Exp Res 38:1602–1610
Wang X-J, Krystal JH (2014) Computational psychiatry. Neuron 84:638–654
Weiland BJ, Heitzeg MM, Zald D, Cummiford C, Love T, Zucker RA, Zubieta J-K (2014) Relationship between impulsivity, prefrontal anticipatory activation, and striatal dopamine release during rewarded task performance. Psychiatry Res Neuroimaging 223:244–252
Whiteside SP, Lynam DR (2001) The five factor model and impulsivity: using a structural model of personality to understand impulsivity. Personal Individ Differ 30:669–689
Wilson VB, Mitchell SH, Musser ED, Schmitt CF, Nigg JT (2011) Delay discounting of reward in ADHD: application in young children. J Child Psychol Psychiatry 52:256–264
Wilson RC, Takahashi YK, Schoenbaum G, Niv Y (2014) Orbitofrontal cortex as a cognitive map of task space. Neuron 81:267–279
Winstanley CA, Dalley JW, Theobald DEH, Robbins TW (2003) Global 5-HT depletion attenuates the ability of amphetamine to decrease impulsive choice on a delay-discounting task in rats. Psychopharmacology 170:320–331
Winstanley CA, Dalley JW, Theobald DE, Robbins TW (2004) Fractionating impulsivity: contrasting effects of central 5-HT depletion on different measures of impulsive behavior. Neuropsychopharmacology 29:1331–1343
Woolf NJ, Butcher LL (1981) Cholinergic neurons in the caudate-putamen complex proper are intrinsically organized: a combined evans blue and acetylcholinesterase analysis. Brain Res Bull 7:487–507
Yates JR, Bardo MT (2017) Effects of intra-accumbal administration of dopamine and ionotropic glutamate receptor drugs on delay discounting performance in rats. Behav Neurosci 131:392–405
Young JW, Meves JM, Tarantino IS, Caldwell S, Geyer MA (2011) Delayed procedural learning in α7-nicotinic acetylcholine receptor knockout mice. Genes Brain Behav 10:720–733
Zaichenko MI, Bazhenova DA, Grigor’yan GA, Merzhanova GK (2017) Does impulsivity influence the operation of long-term and working memory in rats? Neurosci Behav Physiol 47:427–434
Zalocusky KA, Ramakrishnan C, Lerner TN, Davidson TJ, Knutson B, Deisseroth K (2016) Nucleus accumbens D2R cells signal prior outcomes and control risky decision-making. Nature 531:642–646
Zeeb FD, Robbins TW, Winstanley CA (2009) Serotonergic and dopaminergic modulation of gambling behavior as assessed using a novel rat gambling task. Neuropsychopharmacology 34:2329–2343
Zhang X-L, Wang G-B, Zhao L-Y, Sun L-L, Wang J, Wu P, Lu L, Shi J (2012) Clonidine improved laboratory-measured decision-making performance in abstinent heroin addicts Zhang XY, ed. PLoS One 7:e29084
Zhukovsky P, Puaud M, Jupp B, Sala-Bayo J, Alsiö J, Xia J, Searle L, Morris Z, Sabir A, Giuliano C, Everitt BJ, Belin D, Robbins TW, Dalley JW (2019) Withdrawal from escalated cocaine self-administration impairs reversal learning by disrupting the effects of negative feedback on reward exploitation: a behavioral and computational analysis. Neuropsychopharmacology 44:2163–2173
Acknowledgments
This work was supported by National Institutes of Health grants R21 MH120615, R21 MH120799, R01 DA041480, and R01 DA043443 and a Young Investigator Award from the Brain & Behavior Research Foundation.
The author thanks Alexander J. Keip and Neema Moin Afshar for their insightful comments and critiques of the manuscript.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Groman, S.M. (2020). The Neurobiology of Impulsive Decision-Making and Reinforcement Learning in Nonhuman Animals. In: de Wit, H., Jentsch, J.D. (eds) Recent Advances in Research on Impulsivity and Impulsive Behaviors. Current Topics in Behavioral Neurosciences, vol 47. Springer, Cham. https://doi.org/10.1007/7854_2020_127
Download citation
DOI: https://doi.org/10.1007/7854_2020_127
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60510-0
Online ISBN: 978-3-030-60511-7
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)