Modeling Decision-Making Systems in Addiction

Part of the Springer Series in Computational Neuroscience book series (NEUROSCI, volume 10)

Abstract

This chapter describes addiction as a failure of decision-making systems. Existing computational theories of addiction have been based on temporal difference (TD) learning as a quantitative model for decision-making. In these theories, drugs of abuse create a non-compensable TD reward prediction error signal that causes pathological overvaluation of drug-seeking choices. However, the TD model is too simple to account for all aspects of decision-making. For example, TD requires a state-space over which to learn. The process of acquiring a state-space, which involves both situation classification and learning causal relationships between states, presents another set of vulnerabilities to addiction. For example, problem gambling may be partly caused by a misclassification of the situations that lead to wins and losses. Extending TD to include state-space learning also permits quantitative descriptions of how changing representations impacts patterns of intertemporal choice behavior, potentially reducing impulsive choices just by changing cause-effect beliefs. This approach suggests that addicts can learn healthy representations to recover from addiction. All the computational models of addiction published so far are based on learning models that do not attempt to look ahead into the future to calculate optimal decisions. A deeper understanding of how decision-making breaks down in addiction will certainly require addressing the interaction of drugs with model-based look-ahead decision mechanisms, a topic that remains unexplored.

Keywords

Sugar Attenuation Dopamine Nicotine Adenosine 

References

  1. Ainslie G (1974) Impulse control in pigeons. J Exp Anal Behav 21:485 PubMedCrossRefGoogle Scholar
  2. Ainslie G (2001) Breakdown of will. Cambridge University Press, Cambridge Google Scholar
  3. Andersen S, Harrison GW, Lau MI, Rutström EE (2008) Eliciting risk and time preferences. Econometrica 76:583 CrossRefGoogle Scholar
  4. Aragona BJ, Cleaveland NA, Stuber GD, Day JJ, Carelli RM, Wightman RM (2008) Preferential enhancement of dopamine transmission within the nucleus accumbens shell by cocaine is attributable to a direct increase in phasic dopamine release events. J Neurosci 28:8821 PubMedCrossRefGoogle Scholar
  5. Arbisi PA, Billington CJ, Levine AS (1999) The effect of naltrexone on taste detection and recognition threshold. Appetite 32:241 PubMedCrossRefGoogle Scholar
  6. Balleine BW (2001) Incentive processes in instrumental conditioning. In: Handbook of contemporary Learning Theories, p 307 Google Scholar
  7. Balleine BW (2004) Incentive behavior. In: The behavior of the laboratory rat: a handbook with tests, p 436 Google Scholar
  8. Balleine BW, Dickinson A (1998) Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37:407 PubMedCrossRefGoogle Scholar
  9. Balleine BW, Daw ND, O’Doherty JP (2008) Multiple forms of value learning and the function of dopamine. In: Neuroeconomics: decision making and the brain, p 367 Google Scholar
  10. Barnes CA (1979) Memory deficits associated with senscence: A neurophysiological and behavioral study in the rat. J Comp Physiol Psychol 93:74 PubMedCrossRefGoogle Scholar
  11. Barto AG (1994) Adaptive critics and the basal ganglia. In: Models of information processing in the basal ganglia, p 215 Google Scholar
  12. Baum W, Rachlin H (1969) Choice as time allocation. J Exp Anal Behav 12:861 PubMedCrossRefGoogle Scholar
  13. Bayer HM, Glimcher P (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129 PubMedCrossRefGoogle Scholar
  14. Becker GS, Murphy KM (1988) A theory of rational addiction. J Polit Econ 96:675 CrossRefGoogle Scholar
  15. Becker GS, Grossman M, Murphy KM (1994) An empirical analysis of cigarette addiction. Am Econ Rev 84:396 Google Scholar
  16. Bernheim BD, Rangel A (2004) Addiction and cue-triggered decision processes. Am Econ Rev 94:1558 CrossRefGoogle Scholar
  17. Berridge KC (2007) The debate over dopamine’s role in reward: the case for incentive salience. Psychopharmacology 191:391 PubMedCrossRefGoogle Scholar
  18. Berridge CW, Arnsten AF, Foote SL (1993) Noradrenergic modulation of cognitive function: clinical implications of anatomical, electrophysiological and behavioural studies in animal models. Psychol Med 23:557 PubMedCrossRefGoogle Scholar
  19. Bickel WK, Odum AL, Madden GJ (1999) Impulsivity and cigarette smoking: delay discounting in current, never, and ex-smokers. Psychopharmacology (Berlin) 146:447 CrossRefGoogle Scholar
  20. Bouton ME (2002) Context, ambiguity, and unlearning: sources of relapse after behavioral extinction. Biol Psychiatry 52:976 PubMedCrossRefGoogle Scholar
  21. Bouton ME, Swartzentruber D (1989) Slow reacquisition following extinction: context, encoding, and retrieval mechanisms. J Exp Psychol, Anim Behav Processes 15:43 CrossRefGoogle Scholar
  22. Bouton ME, Westbrook RF, Corcoran KA, Maren S (2006) Contextual and temporal modulation of extinction: behavioral and biological mechanisms. Biol Psychiatry 60:352 PubMedCrossRefGoogle Scholar
  23. Breland K, Breland M (1961) The misbehavior of organisms. Am Psychol 16:682 CrossRefGoogle Scholar
  24. Burks SV, Carpenter JP, Goette L, Rustichini A (2009) Cognitive skills affect economic preferences, strategic behavior, and job attachment. Proc Natl Acad Sci 106:7745 PubMedCrossRefGoogle Scholar
  25. Childress AR, Ehrman R, Rohsenow DJ, Robbins SJ, O’Brien CP (1992) Classically conditioned factors in drug dependence. In: Substance abuse: a comprehensive textbook, p 56 Google Scholar
  26. Christensen CJ, Silberberg A, Hursh SR, Roma PG, Riley AL (2008) Demand for cocaine and food over time. Pharmacol Biochem Behav 91:209 PubMedCrossRefGoogle Scholar
  27. Chung SH, Herrnstein RJ (1967) Choice and delay of reinforcement. J Exp Anal Behav 10:67 PubMedCrossRefGoogle Scholar
  28. Corbit LH, Balleine BW (2000) The role of the hippocampus in instrumental conditioning. J Neurosci 20:4233 PubMedGoogle Scholar
  29. Cote D, Caron A, Aubert J, Desrochers V, Ladouceur R (2003) Near wins prolong gambling on a video lottery terminal. J Gambl Stud 19:433 PubMedCrossRefGoogle Scholar
  30. Courville AC (2006) A latent cause theory of classical conditioning. Doctoral dissertation, Carnegie Mellon University Google Scholar
  31. Custer RL (1984) Profile of the pathological gambler. J Clin Psychiatry 45:35 PubMedGoogle Scholar
  32. Daw ND (2003) Reinforcement learning models of the dopamine system and their behavioral implications. Doctoral dissertation, Carnegie Mellon University Google Scholar
  33. Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16:199 PubMedCrossRefGoogle Scholar
  34. Daw ND, Kakade S, Dayan P (2002) Opponent interactions between serotonin and dopamine. Neural Netw 15:603 PubMedCrossRefGoogle Scholar
  35. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8:1704 PubMedCrossRefGoogle Scholar
  36. Daw ND, Courville AC, Touretzky DS (2006) Representation and timing in theories of the dopamine system. Neural Comput 18:1637 PubMedCrossRefGoogle Scholar
  37. Dayan P (2002) Motivated reinforcement learning. Advances in neural information processing systems: proceedings of the 2002 conference Google Scholar
  38. Dayan P, Balleine BW (2002) Reward, motivation, and reinforcement learning. Neuron 36:285 PubMedCrossRefGoogle Scholar
  39. Dayan P, Seymour B (2008) Values and actions in aversion. In: Neuroeconomics: decision making and the brain, p 175 Google Scholar
  40. Dayan P, Niv Y, Seymour B, Daw ND (2006) The misbehavior of value and the discipline of the will. Neural Netw 19:1153 PubMedCrossRefGoogle Scholar
  41. Dezfouli A, Piray P, Keramati MM, Ekhtiari H, Lucas C, Mokri A (2009) A neurocomputational model for cocaine addiction. Neural Comput 21:2869 PubMedCrossRefGoogle Scholar
  42. di Chiara G (1999) Drug addiction as dopamine-dependent associative learning disorder. Eur J Pharmacol 375:13 PubMedCrossRefGoogle Scholar
  43. Dickerson M, O’Connor J (2006) Gambling as an addictive behavior. Cambridge University Press, Cambridge CrossRefGoogle Scholar
  44. Domjan M (1998) The principles of learning and behavior. Brooks/Cole Google Scholar
  45. Doya K (2000) Metalearning, neuromodulation, and emotion. In: Affective minds, p 101 Google Scholar
  46. Elster J (1999) Gambling and addiction. In: Getting hooked: rationality and addiction, p 208 Google Scholar
  47. Estes WK (1943) Discriminative conditioning. I. A discriminative property of conditioned anticipation. J Exp Psychol 32:150 CrossRefGoogle Scholar
  48. Fiorillo CD, Newsome WT, Schultz W (2008) The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 11:966 CrossRefGoogle Scholar
  49. Flagel SB, Watson SJ, Akil H, Robinson TE (2008) Individual differences in the attribution of incentive salience to a reward-related cue: Influence on cocaine sensitization. Behav Brain Res 186:48 PubMedCrossRefGoogle Scholar
  50. Frederick S, Loewenstein G, O’Donoghue T (2002) Time Discounting and time preference: A critical review. J Econ Lit 40:351 CrossRefGoogle Scholar
  51. Fuhs MC, Touretzky DS (2007) Context learning in the rodent hippocampus. Neural Comput 19:3172 CrossRefGoogle Scholar
  52. Gershman SJ, Blei DM, Niv Y (2010) Context, learning, and extinction. Psychol Rev 117:197 PubMedCrossRefGoogle Scholar
  53. Glimcher PW, Camerer C, Fehr E, Poldrack RA (2008) Neuroeconomics: decision making and the brain. Elsevier/Academic Press, London Google Scholar
  54. Goldman MS, Brown SA, Christiansen BA (1987) Expectancy theory: thinking about drinking. In: Psychological theories of drinking and alcoholism, p 181 Google Scholar
  55. Goldstein A (2000) Addiction: from biology to drug policy. Oxford University Press, Oxford Google Scholar
  56. Grossman M, Chaloupka FJ (1998) The demand for cocaine by young adults: a rational addiction approach. J Health Econ 17:427 PubMedCrossRefGoogle Scholar
  57. Gul F, Pesendorfer W (2001) Temptation and self-control. Econometrica 69:1403 CrossRefGoogle Scholar
  58. Gutkin BS, Dehaene S, Changeux JP (2006) A neurocomputational hypothesis for nicotine addiction. Proc Natl Acad Sci USA 103:1106 PubMedCrossRefGoogle Scholar
  59. Henly SE, Ostdiek A, Blackwell E, Knutie S, Dunlap AS, Stephens DW (2008) The discounting-by-interruptions hypothesis: model and experiment. Behav Ecol 19:154 CrossRefGoogle Scholar
  60. Hershberger WA (1986) An approach through the looking-glass. Anim Learn Behav 14:443 CrossRefGoogle Scholar
  61. Heyman GM (2009) Addiction: a disorder of choice. Harvard University Press, Cambridge Google Scholar
  62. Higgins ST, Heil SH, Lussier JP (2004) Clinical implications of reinforcement as a determinant of substance use disorders. Annu Rev Psychol 55:431 PubMedCrossRefGoogle Scholar
  63. Hirsh R (1974) The hippocampus and contextual retrieval of information from memory: A theory. Behav Biol 12:421 PubMedCrossRefGoogle Scholar
  64. Hirsh R, Leber B, Gillman K (1978) Fornix fibers and motivational states as controllers of behavior: A study stimulated by the contextual retrieval theory. Behav Biol 22:463 PubMedCrossRefGoogle Scholar
  65. Hu D, Amsel A (1995) A Simple Test of the Vicarious Trial-and-Error Hypothesis of Hippocampal Function. Proc Natl Acad Sci USA 92:5506 PubMedCrossRefGoogle Scholar
  66. Hu D, Xu X, Gonzalez-Lima F (2006) Vicarious trial-and-error behavior and hippocampal cytochrome oxidase activity during Y-maze discrimination learning in the rat. Int J Neurosci 116:265 PubMedCrossRefGoogle Scholar
  67. Hunt WA (1998) Pharmacology of alcohol. In: Tarter RE, Ammerman RT, Ott PJ (eds) Handbook of substance abuse: Neurobehavioral pharmacology. Plenum, New York, pp 7–22 Google Scholar
  68. Isaacson RL (1974) The limbic system. Plenum, New York CrossRefGoogle Scholar
  69. Isoda M, Hikosaka O (2008) Role for subthalamic nucleus neurons in switching from automatic to controlled eye movement. J Neurosci 28:7209 PubMedCrossRefGoogle Scholar
  70. Jaffe JH, Cascella NG, Kumor KM, Sherer MA (1989) Cocaine-induced cocaine craving. Psychopharmacology (Berlin) 97:59 CrossRefGoogle Scholar
  71. Jaffe A, Gitisetan S, Tarash I, Pham AZ, Jentsch JD (2010) Are nicotine-related cues susceptible to the blocking effect? Society for Neuroscience Abstracts, Program Number 268.4 Google Scholar
  72. Johnson A, Redish AD (2007) Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J Neurosci 27:12176 PubMedCrossRefGoogle Scholar
  73. Jones BT, Corbin W, Fromme K (2001) A review of expectancy theory and alcohol consumption. Addiction 96:57 PubMedCrossRefGoogle Scholar
  74. Kamin LJ (1969) Predictability, surprise, attention, and conditioning. In: Learning in animals, p 279 Google Scholar
  75. Kirby KN, Herrnstein RJ (1995) Preference reversals due to myopic discounting of delayed reward. Psychol Sci 6:83 CrossRefGoogle Scholar
  76. Kruse JM, Overmier JB, Konz WA, Rokke E (1983) Pavlovian conditioned stimulus effects upon instrumental choice behavior are reinforcer specific. Learn Motiv 14:165 CrossRefGoogle Scholar
  77. Kuhar MJ, Ritz MC, Sharkey J (1988) Cocaine receptors on dopamine transporters mediate cocaine-reinforced behavior. In: Mechanisms of cocaine abuse and toxicity, p 14 Google Scholar
  78. Kurth-Nelson Z, Redish AD (2009) Temporal-difference reinforcement learning with distributed representations. PLoS ONE 4:e7362 PubMedCrossRefGoogle Scholar
  79. Kurth-Nelson Z, Redish AD (2010) A reinforcement learning model of precommitment in decision making. Frontiers Behav Neurosci 4:184 Google Scholar
  80. Langer EJ, Roth J (1975) Heads I win, tails it’s chance: The illusion of control as a function of the sequence of outcomes in a purely chance task. J Pers Soc Psychol 32:951 CrossRefGoogle Scholar
  81. Lebron K, Milad MR, Quirk GJ (2004) Delayed recall of fear extinction in rats with lesions of ventral medial prefrontal cortex. Learn Mem 11:544 PubMedCrossRefGoogle Scholar
  82. Lenoir M, Serre F, Cantin L, Ahmed SH (2007) Intense sweetness surpasses cocaine reward. PLoS ONE 2:e698 PubMedCrossRefGoogle Scholar
  83. Levine AS, Billington CJ (2004) Opioids as agents of reward-related feeding: a consideration of the evidence. Physiol Behav 82:57 PubMedCrossRefGoogle Scholar
  84. Liao D, Lin H, Law PY, Loh HH (2005) Mu-opioid receptors modulate the stability of dendritic spines. Proc Natl Acad Sci USA 102:1725 PubMedCrossRefGoogle Scholar
  85. Liu J-, Liu J-, Hammit JK, Chou S- (1999) The price elasticity of opium in Taiwan, 1914–1942. J Health Econ 18:795 PubMedCrossRefGoogle Scholar
  86. Ljungberg T, Apicella P, Schultz W (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol 67:145 PubMedGoogle Scholar
  87. Lovibond PF (1983) Facilitation of instrumental behavior by a Pavlovian appetitive conditioned stimulus. J Exp Psychol Anim Behav Process 9:225 PubMedCrossRefGoogle Scholar
  88. Mackintosh NJ (1974) The psychology of animal learning. Academic Press, San Diego Google Scholar
  89. Madden GJ, Bickel WK (2010) Impulsivity: the behavioral and neurological science of discounting. American Psychological Association, Washington, DC CrossRefGoogle Scholar
  90. Mazur J (1987) An adjusting procedure for studying delayed reinforcement. In: Quantitative analyses of behavior, p 55 Google Scholar
  91. McCaul ME, Petry NM (2003) The role of psychosocial treatments in pharmacotherapy for alcoholism. Am J Addict 12:S41 PubMedCrossRefGoogle Scholar
  92. McFarland K, Kalivas PW (2001) The circuitry mediating cocaine-induced reinstatement of drug-seeking behavior. J Neurosci 21:8655 PubMedGoogle Scholar
  93. Milad MR, Vidal-Gonzalez I, Quirk GJ (2004) Electrical stimulation of medial prefrontal cortex reduces conditioned fear in a temporally specific manner. Behav Neurosci 118:389 PubMedCrossRefGoogle Scholar
  94. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16:1936 PubMedGoogle Scholar
  95. Moos RH, Moos BS (2004) Long-term influence of duration and frequency of participation in alcoholics anonymous on individuals with alcohol use disorders. J Consult Clin Psychol 72:81 PubMedCrossRefGoogle Scholar
  96. Moos RH, Moos BS (2006a) Participation in treatment and Alcoholics Anonymous: a 16-year follow-up of initially untreated individuals. J Clin Psychol 62:735 PubMedCrossRefGoogle Scholar
  97. Moos RH, Moos BS (2006b) Rates and predictors of relapse after natural and treated remission from alcohol use disorders. Addiction 101:212 PubMedCrossRefGoogle Scholar
  98. Muenzinger KF (1938) Vicarious trial and error at a point of choice. I. A general survey of its relation to learning efficiency. J Genet Psychol 53:75 Google Scholar
  99. Nadel L, Willner J (1980) Context and conditioning: A place for space. Physiol Psychol 8:218 Google Scholar
  100. Nestler EJ (1996) Under siege: The brain on opiates. Neuron 16:897 PubMedCrossRefGoogle Scholar
  101. Niv Y, Montague PR (2008) Theoretical and empirical studies of learning. In: Neuroeconomics: decision making and the brain, p 331 Google Scholar
  102. Niv Y, Daw ND, Dayan P (2006) Choice values. Nat Neurosci 9:987 PubMedCrossRefGoogle Scholar
  103. O’Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452 PubMedCrossRefGoogle Scholar
  104. O’Keefe J, Dostrovsky J (1971) The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely moving rat. Brain Res 34:171 PubMedCrossRefGoogle Scholar
  105. O’Keefe J, Nadel L (1978) The hippocampus as a cognitive map. Clarendon, Oxford Google Scholar
  106. Oscar-Berman M, Marinkovic K (2003) Alcoholism and the brain: an overview. Alcohol Res Health 27(2):125–134 PubMedGoogle Scholar
  107. Ostlund SB, Balleine BW (2008) The disunity of Pavlovian and instrumental values. Behav Brain Sci 31:456 CrossRefGoogle Scholar
  108. Packard MG, McGaugh JL (1996) Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol Learn Mem 65:65 PubMedCrossRefGoogle Scholar
  109. Paine TA, Dringenberg HC, Olmstead MC (2003) Effects of chronic cocaine on impulsivity: relation to cortical serotonin mechanisms. Behav Brain Res 147:135 PubMedCrossRefGoogle Scholar
  110. Panlilio LV, Thorndike EB, Schindler CW (2007) Blocking of conditioning to a cocaine-paired stimulus: Testing the hypothesis that cocaine perpetually produces a signal of larger-than-expected reward. Pharmacol Biochem Behav 86:774 PubMedCrossRefGoogle Scholar
  111. Parke J, Griffiths M (2004) Gambling addiction and the evolution of the near miss. Addict Res Theory 12:407 CrossRefGoogle Scholar
  112. Pavlov I (1927) Conditioned reflexes. Oxford Univ Press, Oxford Google Scholar
  113. Phillips PEM, Stuber GD, Heien MLAV, Wightman RM, Carelli RM (2003) Subsecond dopamine release promotes cocaine seeking. Nature 422:614 PubMedCrossRefGoogle Scholar
  114. Porrino LJ, Lyons D, Smith HR, Daunais JB, Nader MA (2004) Cocaine self-administration produces a progressive involvement of limbic, association, and sensorimotor striatal domains. J Neurosci 24:3554 PubMedCrossRefGoogle Scholar
  115. Preuschoff K, Bossaerts P, Quartz SR (2006) Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51:381 PubMedCrossRefGoogle Scholar
  116. Quirk GJ, Garcia R, González-Lima F (2006) Prefrontal mechanisms in extinction of conditioned fear. Biol Psychiatry 60:337 PubMedCrossRefGoogle Scholar
  117. Rachlin H (2000) The science of self-control. Harvard University Press, Cambridge Google Scholar
  118. Rachlin H, Green L (1972) Commitment, choice, and self-control. J Exp Anal Behav 17:15 PubMedCrossRefGoogle Scholar
  119. Redish AD (1999) Beyond the cognitive map: from place cells to episodic memory. MIT Press, Cambridge Google Scholar
  120. Redish AD (2004) Addiction as a computational process gone awry. Science 306:1944 PubMedCrossRefGoogle Scholar
  121. Redish AD (2009) Implications of the multiple-vulnerabilities theory of addiction for craving and relapse. Addiction 104:1940 PubMedCrossRefGoogle Scholar
  122. Redish AD, Johnson A (2007) A computational model of craving and obsession. Ann NY Acad Sci 1104:324 PubMedCrossRefGoogle Scholar
  123. Redish AD, Kurth-Nelson Z (2010) Neural models of temporal discounting. In: Impulsivity: the behavioral and neurological science of discounting, p 123 CrossRefGoogle Scholar
  124. Redish AD, Jensen S, Johnson A, Kurth-Nelson Z (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol Rev 114:784 PubMedCrossRefGoogle Scholar
  125. Redish AD, Jensen S, Johnson A (2008) A unified framework for addiction: vulnerabilities in the decision process. Behav Brain Sci 31:415 PubMedGoogle Scholar
  126. Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In: Classical conditioning II, p 64 Google Scholar
  127. Restle F (1957) Discrimination of cues in mazes: A resolution of the ‘place-vs-response’ question. Psychol Rev 64:217 PubMedCrossRefGoogle Scholar
  128. Reynolds B, Ortengren A, Richards JB, de Wit H (2006) Dimensions of impulsive behavior: personality and behavioral measures. Pers Individ Differ 40:305 CrossRefGoogle Scholar
  129. Ritz MC, Lamb RJ, Goldberg SR, Kuhar MJ (1987) Cocaine receptors on dopamine transporters are related to self-administration of cocaine. Science 237:1219 PubMedCrossRefGoogle Scholar
  130. Robinson TE, Berridge KC (1993) The neural basis of drug craving: An incentive-sensitization theory of addiction. Brains Res Rev 18:247 CrossRefGoogle Scholar
  131. Robinson TE, Berridge KC (2001) Mechanisms of action of addictive stimuli: Incentive-sensitization and addiction. Addiction 96:103 PubMedCrossRefGoogle Scholar
  132. Robinson TE, Berridge KC (2003) Addiction. Annu Rev Psychol 54:25 PubMedCrossRefGoogle Scholar
  133. Robinson TE, Berridge KC (2004) Incentive-sensitization and drug ‘wanting’. Psychopharmacology 171:352 CrossRefGoogle Scholar
  134. Schultz W (2002) Getting formal with dopamine and reward. Neuron 36:241 PubMedCrossRefGoogle Scholar
  135. Schultz W, Dayan P, Montague R (1997) A neural substrate of prediction and reward. Science 275:1593 PubMedCrossRefGoogle Scholar
  136. Schweighofer N, Shishida K, Han CE, Yamawaki S, Doya K (2006) Humans can adopt optimal discounting strategy under real-time constraints. PLoS Comput Biol 2:e152 PubMedCrossRefGoogle Scholar
  137. Schweighofer N, Tanaka SC, Doya K (2007) Serotonin and the evaluation of future rewards. Theory, experiments, and possible neural mechanisms. Ann NY Acad Sci 1104:289 PubMedCrossRefGoogle Scholar
  138. Si J, Barto AG, Powell WB, Wunsch D (2004) Handbook of learning and approximate dynamic programming. Wiley/IEEE Press, New York CrossRefGoogle Scholar
  139. Simon NW, Mendez IA, Setlow B (2007) Cocaine exposure causes long-term increases in impulsive choice. Behav Neurosci 121:543 PubMedCrossRefGoogle Scholar
  140. Smith A, Li M, Becker S, Kapur S (2006) Dopamine, prediction error and associative learning: a model-based account. Network: Comput Neural Syst 17:61 CrossRefGoogle Scholar
  141. Sotres-Bayon F, Cain CK, LeDoux JE (2006) Brain mechanisms of fear extinction: historical perspectives on the contribution of prefrontal cortex. Biol Psychiatry 60:329 PubMedCrossRefGoogle Scholar
  142. Sozou PD (1998) On hyperbolic discounting and uncertain hazard rates. R Soc Lond B 265:2015 CrossRefGoogle Scholar
  143. Stahl SM, Pradko JF, Haight BR, Modell JG, Rockett CB, Learned-Coughlin S (2004) A review of the neuropharmacology of bupropion, a dual norepinephrine and dopamine reuptake inhibitor. Prim Care Companion J Clin Psychiat 6:159 CrossRefGoogle Scholar
  144. Strotz RH (1956) Myopia and inconsistency in dynamic utility maximization. Rev Econ Stud 23:165 Google Scholar
  145. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge Google Scholar
  146. Talmi D, Seymour B, Dayan P, Dolan RJ (2008) Human Pavlovian instrumental transfer. J Neurosci 28:360 PubMedCrossRefGoogle Scholar
  147. Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, Yamawaki S (2004) Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci 7:887 PubMedCrossRefGoogle Scholar
  148. Tanaka SC, Schweighofer N, Asahi S, Shishida K, Okamoto Y, Yamawaki S, Doya K (2007) Serotonin differentially regulates short- and long-term prediction of rewards in the ventral and dorsal striatum. PLoS ONE 2:e1333 PubMedCrossRefGoogle Scholar
  149. Tolman EC (1938) The determiners of behavior at a choice point. Psychol Rev 45:1 CrossRefGoogle Scholar
  150. Tolman EC (1939) Prediction of vicarious trial and error by means of the schematic sowbug. Psychol Rev 46:318 CrossRefGoogle Scholar
  151. Tolman EC (1948) Cognitive maps in rats and men. Psychol Rev 55:189 PubMedCrossRefGoogle Scholar
  152. Tsai HC, Zhang F, Adamantidis A, Stuber GD, Bonci A, de Lecea L, Deisseroth K (2009) Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324:1080 PubMedCrossRefGoogle Scholar
  153. Uslaner JM, Acerbo MJ, Jones SA, Robinson TE (2006) The attribution of incentive salience to a stimulus that signals an intravenous injection of cocaine. Behav Brain Res 169:320 PubMedCrossRefGoogle Scholar
  154. van der Meer MA, Redish AD (2009) Covert expectation-of-reward in rat ventral striatum at decision points. Frontiers Integr Neurosci 3:1 Google Scholar
  155. van der Meer MA, Redish AD (2010) Expectancies in decision making, reinforcement learning, and ventral striatum. Front Neurosci 4:29 Google Scholar
  156. Waelti P, Dickinson A, Schultz W (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412:43 PubMedCrossRefGoogle Scholar
  157. Wagenaar WA (1988) Paradoxes of gambling behavior. Erlbaum, London Google Scholar
  158. Weiner I, Lubow RE, Feldon J (1988) Disruption of latent inhibition by acute administration of low doses of amphetamine. Pharmacol Biochem Behav 30:871 PubMedCrossRefGoogle Scholar
  159. White AM (2003) What happened? Alcohol, memory blackouts, and the brain. Alcohol Res Health 27(2):186–196 PubMedGoogle Scholar
  160. Yin HH, Knowlton B, Balleine BW (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci 19:181 PubMedCrossRefGoogle Scholar
  161. Yu AJ, Dayan P (2005) Uncertainty, neuromodulation, and attention. Neuron 46:681 PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.Department of NeuroscienceUniversity of MinnesotaMinneapolisUSA

Personalised recommendations