Prediction of Immediate and Future Rewards Differentially Recruits Cortico-Basal Ganglia Loops

Abstract

Evaluation of both immediate and future outcomes of an action is a critical requirement for intelligent behavior. We investigated brain mechanisms for reward prediction at different time scales in an fMRI experiment using a Markov decision task. When subjects learned actions from immediate rewards, significant activity was found in the lateral orbitofrontal cortex and the striatum. When subjects learned to acquire large future rewards despite small immediate losses, the dorsolateral prefrontal cortex, inferior parietal cortex, dorsal raphe nucleus, and cerebellum were also activated. Computational model-based regression analysis using the predicted future rewards and prediction errors estimated from subjects’ performance data revealed graded maps of time scale within the insula and the striatum, where ventroanterior parts were responsible for predicting immediate rewards and dorsoposterior parts for future rewards. These results suggest differential involvement of the cortico-basal ganglia loops in reward prediction at different time scales.

The original article first appeared in Nature Neuroscience 7(8):887–893, 2004. A newly written addendum has been added to this book chapter.

1 Addendum: Recent Developments This addendum has been newly written by Saori C. Tanaka for this book chapter (partly taken from the doctoral dissertation “Functional model of serotonin in human reward system based on reinforcement learning theory” by Saori C. Tanaka, 2006).

This addendum has been newly written by Saori C. Tanaka for this book chapter (partly taken from the doctoral dissertation “Functional model of serotonin in human reward system based on reinforcement learning theory” by Saori C. Tanaka, 2006).

This addendum has been newly written by Saori C. Tanaka for this book chapter (partly taken from the doctoral dissertation “Functional model of serotonin in human reward system based on reinforcement learning theory” by Saori C. Tanaka, 2006).

We hypothesized that different cortico-basal ganglia loops are involved in reward prediction at different time scales simultaneously, and one of these time scales is chosen by serotonergic modulation on parallel loops and used in actual action selection. To elucidate the effects of serotonin on the parallel cortico-striatum loop mechanisms, we controlled subjects’ serotonin levels by dietary tryptophan (the precursor of serotonin) manipulation, and measured brain activity at different serotonin levels during choice tasks for both immediate-small reward and delayed-large reward (Experiment 2) (Tanaka et al. 2007). Using a regression analysis of reward prediction signals, we found that while the activity in the ventral part of the striatum correlated strongly with short-term reward prediction at low serotonin levels, those of the dorsal part strongly correlated with long-term reward prediction at high serotonin levels. This result supports the possibility that serotonin controls the time scale of reward prediction by differentially regulating the activity within the striatum.

We found similar graded time-scale maps for reward prediction in the striatum in our previous experiment (Experiment 1) (Tanaka et al. 2004) and later experiment (Experiment 2) (Tanaka et al. 2007). In both maps, the ventral parts are correlated with reward prediction at shorter time scales, indicated by smaller γ values, whereas dorsal parts are correlated with reward prediction at longer time scales (larger γ values). Are both maps graded on the same time scale? That is, is a particular part of the graded map involved in reward prediction at a particular time scale? If so, a question arises as to whether this map is graded in theoretical time or real time. To answer these questions, we verify the graded maps in the striatum that we found in Experiments 1 and 2.

Fig. 22.8

The number of voxels at each z-level that were significantly correlated with reward prediction at each time scale in Experiment 1 and 2 in (a) γ-grading and (b) τ-grading. Colored lines show the median z-coordinate of voxel distribution with each time scale. Although there are gradients of time scales from ventral (low z-level) to dorsal (high z-level) both in Experiments1 and 2, we can see good consistency of time scales between Experiments 1 and 2 only in γ-grading (Note that different color scales are used in γ-grading and τ-grading)

Fig. 22.9

The median z-coordinate of voxel distribution with each time scale. (a) In the γ-grading, we can see good fitting of data in both Experiments 1 (*) and 2 (○) by the same function. (b) In the τ-grading, in contrast, this seems difficult to be explained by the same function

These results indicate that particular parts of the striatum are involved in reward prediction not at absolute time scales but at relative time scales depending on the task. In the real world, we need to solve problems with variable time scales. At some times we choose an action producing a reward after several seconds or minutes, and at other times, we make decisions that reap rewards several years later. In this case, the relative grading of a time scale may be effective because the broader region of the striatum can be engaged to compute reward prediction with a limited number of striatal neurons.