Neural encoding of choice during a delayed response task in primate striatum and orbitofrontal cortex

  • Howard C. Cromwell
  • Leon Tremblay
  • Wolfram Schultz
Research Article

Abstract

Reward outcomes are available in many diverse situations and all involve choice. If there are multiple outcomes each rewarding, then decisions regarding relative value lead to choosing one over another. Important factors related to choice context should be encoded and utilized for this form of adaptive choosing. These factors can include the number of alternatives, the pacing of choice behavior and the possibility to reverse one’s choice. An essential step in understanding if the context of choice is encoded is to directly compare choice with a context in which choice is absent. Neural activity in orbitofrontal cortex and striatum encodes potential value parameters related to reward quality and quantity as well as relative preference. We examined how neural activations in these brain regions are sensitive to choice situations and potentially involved in a prediction for the upcoming outcome selection. Neural activity was recorded and compared between a two-choice spatial delayed response task and an imperative ‘one-option’ task. Neural activity was obtained that extended from the instruction cue to the movement similar to previous work utilizing the identical imperative task. Orbitofrontal and striatal neural responses depended upon the decision about the choice of which reward to collect. Moreover, signals to predictive instruction cues that precede choice were selective for the choice situation. These neural responses could reflect chosen value with greater information on relative value of individual options as well as encode choice context itself embedded in the task as a part of the post-decision variable.

Keywords

Caudate Decision-making Incentive contrast Nucleus accumbens Putamen Reward 

References

  1. Apicella P, Ljungberg T, Scarnati E, Schultz W (1991) Responses to reward in monkey dorsal and ventral striatum. Exp Brain Res 85(3):491–500CrossRefPubMedGoogle Scholar
  2. Belguermmi A, Bovet D, Pascal A, Prévot-Julliard A, Jalme M, Rat-Fischer L, Leboucher G (2011) Pigeons discriminate between human feeders. Anim Cogn 14(6):909–914.  https://doi.org/10.1007/s10071-011-0420-7 CrossRefGoogle Scholar
  3. Berridge KC, Cromwell HC (1990) Motivational-sensorimotor interaction controls aphagia and exaggerated treading after striatopallidal lesions. Behav Neurosci 104(5):778–795CrossRefPubMedGoogle Scholar
  4. Berridge KC, Kringelbach ML (2013) Neuroscience of affect: brain mechanisms of pleasure and displeasure. Curr Opin Neurobiol 23(3):294–303CrossRefPubMedPubMedCentralGoogle Scholar
  5. Blanchard TC, Hayden BY (2015) Monkeys are more patient in a foraging task than in a standard intertemporal choice task. PLoS One (United States) 10:e0117057CrossRefGoogle Scholar
  6. Bouton ME, Todd TP, Miles OW, Leon SP, Epstein LH (2013) Within- and between-session variety effects in a food-seeking habituation paradigm. Appetite 66:10–19CrossRefPubMedPubMedCentralGoogle Scholar
  7. Braun S, Hauber W (2011) The dorsomedial striatum mediates flexible choice behavior in spatial tasks. Behav Brain Res (Netherlands) 220:288–293CrossRefGoogle Scholar
  8. Brown VJ, Robbins TW (1991) Simple and choice reaction time performance following unilateral striatal dopamine depletion in the rat. Impaired motor readiness but preserved response preparation. Brain 114(Pt 1B):513–525CrossRefPubMedGoogle Scholar
  9. Cai X, Kim S, Lee D (2011) Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice. Neuron 69:170–182CrossRefPubMedPubMedCentralGoogle Scholar
  10. Cardinal RN, Howes NJ (2005) Effects of lesions of the nucleus accumbens core on choice between small certain rewards and large uncertain rewards in rats. BMC Neurosci 6:37CrossRefPubMedPubMedCentralGoogle Scholar
  11. Catania AC (1963) Concurrent performances: reinforcement interaction and response independence. J Exp Anal Beh 6:253–263CrossRefGoogle Scholar
  12. Catania AC, Savgolden T (1980) Preference for free choice over forced choice in pigeons. J Exp Anal Behav 34(1):77–86.  https://doi.org/10.1901/jeab.1980.34-77 CrossRefPubMedPubMedCentralGoogle Scholar
  13. Cromwell HC, Schultz W (2003) Effects of expectations for different reward magnitudes on neuronal activity in primate striatum. J Neurophysiol 89(5):2823–2838CrossRefPubMedGoogle Scholar
  14. Cromwell HC, Hassani OK, Schultz W (2005) Relative reward processing in primate striatum. Exp Brain Res 162(4):520–525CrossRefPubMedGoogle Scholar
  15. Czoty PW, McCabe C, Nader MA (2005) Effects of the 5-HT(1A) agonist (+/-)-8-hydroxy-2-(di-n-propylamino)tetralin (8-OH-DPAT) on cocaine choice in cynomolgus monkeys. Behav Pharmacol (England) 16:187–191CrossRefGoogle Scholar
  16. Farashahi S, Donahue CH, Khorsand P, Seo H, Lee D, Soltani A (2017) Metaplasticity as a neural substrate for adaptive learning and choice under uncertainty. Neuron 94(2):401–414CrossRefPubMedGoogle Scholar
  17. Fisher WW, Thompson RH, Piazza CC, Crosland K, Gotjen D (1997) On the relative reinforcing effects of choice and differential consequences. J Appl Behav Anal 30(3):423–438.  https://doi.org/10.1901/jaba.1997.30-423 CrossRefPubMedPubMedCentralGoogle Scholar
  18. Freidin E, Kacelnik A (2011) Rational choice, context dependence, and the value of information in European starlings (Sturnus vulgaris). Science 334(6058):1000–1002CrossRefGoogle Scholar
  19. Gourley SL, Olevska A, Zimmermann KS, Ressler KJ, Dileone RJ, Taylor JR (2013) The orbitofrontal cortex regulates outcome-based decision-making via the lateral striatum. Eur J Neurosci (France) 38:2382–2388CrossRefGoogle Scholar
  20. Grabenhorst F, Hernadi I, Schultz W (2016) Primate amygdala neurons evaluate the progress of self-defined economic choice sequences. Elife 5:e18731.  https://doi.org/10.7554/eLife.18731 Google Scholar
  21. Hassani OK, Cromwell HC, Schultz W (2001) Influence of the expectation for different rewards onbehavior-related neuronal activity in primate striatum. J Neurophysiol 85:2477–2489CrossRefPubMedGoogle Scholar
  22. Hernádi I, Grabenhorst F, Schultz W (2015) Planning activity for internally generated reward goals in monkey amygdala neurons. Nat Neurosci (United States) 18:461–469CrossRefGoogle Scholar
  23. Hwang J, Kim S, Lee D (2009) Temporal discounting and inter-temporal choice in rhesus monkeys. Front Behav Neurosci (Switzerland) 3:9Google Scholar
  24. Iyengar SS, Lepper MR (2000) When choice is demotivating: can one desire too much of a good thing? J Pers Soc Psychol 79(6):995–1006CrossRefPubMedGoogle Scholar
  25. Kacelnik A, Bateson M (1996) Risky theories—the effects of variance on foraging decisions. Integr Comp Biol 36:402–434Google Scholar
  26. Kalivas PW, Volkow ND (2005) The neural basis of addiction: a pathology of motivation and choice. Am J Psychiatry (United States) 162:1403–1413CrossRefGoogle Scholar
  27. Keiflin R, Reese RM, Woods CA, Janak PH (2013) The orbitofrontal cortex as part of a hierarchical neural system mediating choice between two good options. J Neurosci (United States) 33:15989–15998CrossRefGoogle Scholar
  28. Kheramin S, Body S, Mobini S, Ho MY, Velazquez-Martinez DN, Bradshaw CM et al (2002) Effects of quinolinic acid-induced lesions of the orbital prefrontal cortex on inter-temporal choice: a quantitative analysis. Psychopharmacology (Berl) 165(1):9–17CrossRefGoogle Scholar
  29. Kheramin S, Body S, Ho M, Velazquez-Martinez DN, Bradshaw CM, Szabadi E et al (2003) Role of the orbital prefrontal cortex in choice between delayed and uncertain reinforcers: a quantitative analysis. Behav Process 64(3):239–250CrossRefGoogle Scholar
  30. Kheramin S, Body S, Herrera FM, Bradshaw CM, Szabadi E, Deakin JF et al (2005) The effect of orbital prefrontal cortex lesions on performance on a progressive ratio schedule: implications for models of inter-temporal choice. Behav Brain Res 156(1):145–152CrossRefPubMedGoogle Scholar
  31. Kim H, Sul JH, Huh N, Lee D, Jung MW (2009) Role of striatum in updating values of chosen actions. J Neurosci (United States) 29:14701–14712CrossRefGoogle Scholar
  32. Klapp S, Abbott J, Coffman K, Greim D, Snider R, Young F (1979) Simple and choice reaction time methods in the study of motor programming. J Mot Behav 11(2):91–101CrossRefPubMedGoogle Scholar
  33. Lak A, Stauffer WR, Schultz W (2016) Dopamine neurons learn relative chosen value from probabilistic rewards. Elife (England).  https://doi.org/10.7554/eLife.18044 Google Scholar
  34. Lau B, Glimcher PW (2008) Value representations in the primate striatum during matching behavior. Neuron 58(3):451–463CrossRefPubMedPubMedCentralGoogle Scholar
  35. Leung BK, Balleine BW (2013) The ventral striato-pallidal pathway mediates the effect of predictive learning on choice between goal-directed actions. J Neurosci (United States) 33:13848–13860CrossRefGoogle Scholar
  36. Louie K, Glimcher PW (2012) Efficient coding and the neural representation of value. Ann N Y Acad Sci 1251:13–32.  https://doi.org/10.1111/j.1749-6632.2012.06496.x CrossRefPubMedGoogle Scholar
  37. Marcos E, Genovesio A (2016) Determining monkey free choice long before the choice is made: the principal role of prefrontal neurons involved in both decision and motor processes. Front Neural Circuits (Switzerland) 10:75Google Scholar
  38. McGraw JJ, Zona LC, Cromwell HC (2017) The effects of ethanol on diverse components of choice in the rat: reward discrimination, preference and relative valuation. Eur J Neurosci (France) 46(3):1837–1849.  https://doi.org/10.1111/ejn.13627 CrossRefGoogle Scholar
  39. Melville CL, Rue HC, Rybiski LR, Weatherly JN (1997) Altering reinforcer variety or intensity changes the within-session decrease in responding. Learn Motiv 28(4):609–621CrossRefGoogle Scholar
  40. Nader MA, Woolverton WL (1992a) Choice between cocaine and food by rhesus monkeys: effects of conditions of food availability. Behav Pharmacol 3(6):635–638CrossRefPubMedGoogle Scholar
  41. Nader MA, Woolverton WL (1992b) Effects of increasing response requirement on choice between cocaine and food in rhesus monkeys. Psychopharmacology (Berl) 108(3):295–300CrossRefGoogle Scholar
  42. O’Neill M, Schultz W (2010) Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value. Neuron 68(4):789–800CrossRefPubMedGoogle Scholar
  43. Padoa-Schioppa C, Assad JA (2008) The representation of economic value in the orbitofrontal cortex is invariant for changes of menu. Nat Neurosci 11(1):95–102CrossRefPubMedGoogle Scholar
  44. Palmer CR, Kristan WB Jr (2011) Contextual modulation of behavioral choice. Curr Opin Neurobiol (England) 21:520–526CrossRefGoogle Scholar
  45. Pastor-Bernier A, Plott CR, Schultz W (2017) Monkeys choose as if maximizing utility compatible with basic principles of revealed preference theory. Proc Natl Acad Sci USA (United States) 114:E1766-E1775Google Scholar
  46. Rainwater A, Sanz E, Palmiter RD, Quintana A (2017) Striatal GPR88 modulates foraging efficiency. J Neurosci (United States) 37:7939–7947CrossRefGoogle Scholar
  47. Ramirez-Lugo L, Penas-Rincon A, Angeles-Duran S, Sotres-Bayon F (2016) Choice behavior guided by learned, but not innate, taste aversion recruits the orbitofrontal cortex. J Neurosci (United States) 36:10574–10583CrossRefGoogle Scholar
  48. Riceberg JS, Shapiro ML (2017) Orbitofrontal cortex signals expected outcomes with predictive codes when stable contingencies promote the integration of reward history. J Neurosci (United States) 37:2010–2021CrossRefGoogle Scholar
  49. Rich EL, Wallis JD (2016) Decoding subjective decisions from orbitofrontal cortex. Nat Neurosci (United States) 19:973–980CrossRefGoogle Scholar
  50. Ricker JM, Kopchock RJ 3rd, Drown RM, Cromwell HC (2016a) Effects of striatal lesions on components of choice: Reward discrimination, preference, and relative valuation. Behav Brain Res (Netherlands) 315:130–140CrossRefGoogle Scholar
  51. Ricker JM, Hatch JD, Powers DD, Cromwell HC (2016b) Fractionating choice: a study on reward discrimination, preference, and relative valuation in the rat (rattus norvegicus). J Comp Psychol (United States) 130:174–186CrossRefGoogle Scholar
  52. Robbins TW (2002) The 5-choice serial reaction time task: behavioural pharmacology and functional neurochemistry. Psychopharmacol (Berl) 163(3–4):362–380CrossRefGoogle Scholar
  53. Rolls BJ, Rowe EA, Rolls ET, Kingston B, Megson A, Gunary R (1981) Variety in a meal enhances food intake in man. Physiol Behav 26(2):215–221CrossRefPubMedGoogle Scholar
  54. Schultz W (2004) Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology. Curr Opin Neurobiol 14(2):139–147CrossRefPubMedGoogle Scholar
  55. Schultz W (2010) Subjective neuronal coding of reward: temporal value discounting and risk. Eur J Neurosci 31(12):2124–2135CrossRefPubMedGoogle Scholar
  56. Schultz W (2011) Potential vulnerabilities of neuronal reward, risk, and decision mechanisms to addictive drugs. Neuron 69(4):603–617CrossRefPubMedGoogle Scholar
  57. Schultz W (2015) Neuronal reward and decision signals: From theories to data. Physiol Rev (United States) 95:853–951Google Scholar
  58. Schultz W (2017) Reward prediction error. Curr Biol 27(10):R369–R371.  https://doi.org/10.1016/j.cub.2017.02.064 CrossRefPubMedGoogle Scholar
  59. Seymour B, McClure SM (2008) Anchors, scales and the relative coding of value in the brain. Curr Opin Neurobiol (England) 18:173–178CrossRefGoogle Scholar
  60. Shizgal P (1997) Neural basis of utility estimation. Curr Opin Neurobiol 7(2):198–208CrossRefPubMedGoogle Scholar
  61. St Onge JR, Floresco SB (2010) Prefrontal cortical contribution to risk-based decision making. Cereb Cortex 20(8):1816–1828CrossRefPubMedGoogle Scholar
  62. Stauffer WR, Lak A, Schultz W (2014) Dopamine reward prediction error responses reflect marginal utility. Curr Biol (England) 24:2491–2500CrossRefGoogle Scholar
  63. Sternberg S, Monsell S, Knoll RL, Wright CE (1978) The latency and duration of rapid movement sequences: comparisons of speech and typing. In: Stelmach GE (ed) Information processing in motor control and learning. Academic Press, New York, pp 117–152CrossRefGoogle Scholar
  64. Strait CE, Sleezer BJ, Hayden BY (2015) Signatures of value comparison in ventral striatum neurons. PLoS Biol (United States) 13:e1002173CrossRefGoogle Scholar
  65. Strait CE, Sleezer BJ, Blanchard TC, Azab H, Castagno MD, Hayden BY (2016) Neuronal selectivity for spatial positions of offers and choices in five reward regions. J Neurophysiol 115(3):1098–1111.  https://doi.org/10.1152/jn.00325.2015 CrossRefPubMedGoogle Scholar
  66. Suzuki S (1999) Selection of forced- and free-choice by monkeys (macaca fascicularis). Percept Mot Skills 88:242–250.  https://doi.org/10.2466/pms.1999.88.1.242 CrossRefGoogle Scholar
  67. Treit D, Spetch ML, Deutsch JA (1983) Variety in the flavor of food enhances eating in the rat: a controlled demonstration. Physiol Behav 30(2):207–211CrossRefPubMedGoogle Scholar
  68. Tremblay L, Schultz W (1999) Relative reward preference in primate orbitofrontal cortex. Nature (England) 398:704–708CrossRefGoogle Scholar
  69. Tsujimoto S, Genovesio A, Wise SP (2009) Monkey orbitofrontal cortex encodes response choices near feedback time. J Neurosci 29(8):2569–2574.  https://doi.org/10.1523/JNEUROSCI.5777-08.2009 CrossRefPubMedPubMedCentralGoogle Scholar
  70. Tsutsui K, Grabenhorst F, Kobayashi S, Schultz W (2016) A dynamic code for economic object valuation in prefrontal cortex neurons. Nat Commun (England) 7:12554CrossRefGoogle Scholar
  71. Tversky A, Shafir E (1992) Choice under conflict: the dynamics of deferred decision. Psychol Sci 3:358–361CrossRefGoogle Scholar
  72. van Wingerden M, van der Meij R, Kalenscher T, Maris E, Pennartz CM (2014) Phase-amplitude coupling in rat orbitofrontal cortex discriminates between correct and incorrect decisions during associative learning. J Neurosci (United States) 34:493–505CrossRefGoogle Scholar
  73. Watanabe M, Cromwell HC, Tremblay L, Hollerman JR, Hikosaka K, Schultz W (2001) Behavioral reactions reflecting differential reward expectations in monkeys. Exp Brain Res 140(4):511–518CrossRefPubMedGoogle Scholar
  74. Webber ES, Chambers NE, Kostek JA, Mankin DE, Cromwell HC (2015) Relative reward effects on operant behavior: Incentive contrast, induction and variety effects. Behav Process (Netherlands) 116:87–99CrossRefGoogle Scholar
  75. Webber ES, Mankin DE, Cromwell HC (2016) Striatal activity and reward relativity: neural signals encoding dynamic outcome valuation. Eneuro 3:ENEURO.0022-16.2016Google Scholar
  76. Williams BA (2002) Behavioral contrast redux. Animal Learning Behavior 30:1–20CrossRefPubMedGoogle Scholar
  77. Yang X, Gao M, Shi J, Ye H, Chen S (2017) Modulating the activity of the DLPFC and OFC has distinct effects on risk and ambiguity decision-making: a tDCS study. Front Psychol 8:1417.  https://doi.org/10.3389/fpsyg.2017.01417 CrossRefPubMedPubMedCentralGoogle Scholar
  78. Zona LC, Fry BR, Cromwell HC (2017) Effects of anandamide administration on components of reward processing during free choice. Pharmacol Biochem Behav 158:14–21CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Psychology, JP Scott Center for Neuroscience, Mind and BehaviorBowling Green State UniversityBowling GreenUSA
  2. 2.Centre de Neuroscience CognitiveCedexFrance
  3. 3.Université Claude-Bernard Lyon 1VilleurbanneFrance
  4. 4.Department of Physiology, Development and NeuroscienceUniversity of CambridgeCambridgeUK

Personalised recommendations